ToME: the Tales of Maj'Eyal
http://forums.te4.org/

[git] Faster serialization
http://forums.te4.org/viewtopic.php?f=45&t=38553
Page 1 of 1

Author:  johnnyzero [ Sat Jul 27, 2013 7:39 pm ]
Post subject:  [git] Faster serialization

T-Engine has basic serialization support implemented in game/loader/pre-init.lua. This serialization is fine for what T-Engine needs, except that it is really slow in the current git* code. So, I've implemented a buffered version of serialize.

After loading a fresh session of ToME, the profile thread flushes CSETs. On my system, the current git code takes about 40 seconds to finish flushing csets, with the profile thread eating all the cpu cycles it can. Using my table.serialize implementation, flushing csets is done in about one second. Note that these times include zipping the resulting serialized data and sending the result to te4.org.

Code:
function table.serialize(src, sub, no_G, base)
   local chunks = {}
   if sub then
      chunks[1] = "{"
   end
   for k, e in pairs(src) do
      local nk = nil
      local nkC = {}
      local tk, te = type(k), type(e)
      if no_G then
         if tk == "table" then
            nkC[#nkC+1] = "["
            nkC[#nkC+1] = table.serialize(k, true)
            nkC[#nkC+1] = "]"
         elseif tk == "string" then
            nkC[#nkC+1] = k
         else
            nkC[#nkC+1] = "["
            nkC[#nkC+1] = tostring(k)
            nkC[#nkC+1] = "]"
         end
      else
         if not sub then
            nkC[#nkC+1] = (base and tostring(base) or "_G")
         end
         nkC[#nkC+1] = "["
         if tk == "table" then
            nkC[#nkC+1] = table.serialize(k, true)
         elseif tk == "string" then
            -- escaped quotes matter
            nkC[#nkC+1] = string.format("%q", k)
         else
            nkC[#nkC+1] = tostring(k)
         end
         nkC[#nkC+1] = "]"
      end

      nk = table.concat(nkC)

      -- These are the types of data we are willing to serialize
      if te == "table" or te == "string" or te == "number" or te == "boolean" then
         chunks[#chunks+1] = nk
         chunks[#chunks+1] = "="
         if te == "table" then
            chunks[#chunks+1] = table.serialize(e, true)
         elseif te == "number" then
            -- float output matters
            chunks[#chunks+1] = string.format("%f", e)
         elseif te == "string" then
            -- escaped quotes matter
            chunks[#chunks+1] = string.format("%q", e)
         else -- te == "boolean"
            chunks[#chunks+1] = tostring(e)
         end
         chunks[#chunks+1] = " "
      end
      
      if sub then
         chunks[#chunks+1] = ", "
      end
   end
   if sub then
      chunks[#chunks+1] = "}"
   end
         
   return table.concat(chunks)
end

Author:  johnnyzero [ Wed Oct 02, 2013 4:21 pm ]
Post subject:  Re: [git] Faster serialization

Update: the patch has been accepted.
Code:
commit 6f43ddef9b0e454c7afc272c20c6227ce7f8f5c8
Author: DarkGod <darkgod@net-core.org>
Date:   Tue Oct 1 18:11:31 2013 +0200

    use faster serialization

Author:  cttw [ Thu Oct 03, 2013 8:12 am ]
Post subject:  Re: [git] Faster serialization

Good job!

Author:  Karagy [ Thu Oct 03, 2013 9:22 pm ]
Post subject:  Re: [git] Faster serialization

Draw attention to Serializing table data into a string
It can proceed cycled tables.

Page 1 of 1 All times are UTC
Powered by phpBB® Forum Software © phpBB Group
https://www.phpbb.com/