[git] Faster serialization
Posted: Sat Jul 27, 2013 7:39 pm
T-Engine has basic serialization support implemented in game/loader/pre-init.lua. This serialization is fine for what T-Engine needs, except that it is really slow in the current git* code. So, I've implemented a buffered version of serialize.
After loading a fresh session of ToME, the profile thread flushes CSETs. On my system, the current git code takes about 40 seconds to finish flushing csets, with the profile thread eating all the cpu cycles it can. Using my table.serialize implementation, flushing csets is done in about one second. Note that these times include zipping the resulting serialized data and sending the result to te4.org.
After loading a fresh session of ToME, the profile thread flushes CSETs. On my system, the current git code takes about 40 seconds to finish flushing csets, with the profile thread eating all the cpu cycles it can. Using my table.serialize implementation, flushing csets is done in about one second. Note that these times include zipping the resulting serialized data and sending the result to te4.org.
- More information on why current git serialization is so slow.
- *current git = commit 559709d / 2013-07-27
- Here's a link to a patch file.
Code: Select all
function table.serialize(src, sub, no_G, base)
local chunks = {}
if sub then
chunks[1] = "{"
end
for k, e in pairs(src) do
local nk = nil
local nkC = {}
local tk, te = type(k), type(e)
if no_G then
if tk == "table" then
nkC[#nkC+1] = "["
nkC[#nkC+1] = table.serialize(k, true)
nkC[#nkC+1] = "]"
elseif tk == "string" then
nkC[#nkC+1] = k
else
nkC[#nkC+1] = "["
nkC[#nkC+1] = tostring(k)
nkC[#nkC+1] = "]"
end
else
if not sub then
nkC[#nkC+1] = (base and tostring(base) or "_G")
end
nkC[#nkC+1] = "["
if tk == "table" then
nkC[#nkC+1] = table.serialize(k, true)
elseif tk == "string" then
-- escaped quotes matter
nkC[#nkC+1] = string.format("%q", k)
else
nkC[#nkC+1] = tostring(k)
end
nkC[#nkC+1] = "]"
end
nk = table.concat(nkC)
-- These are the types of data we are willing to serialize
if te == "table" or te == "string" or te == "number" or te == "boolean" then
chunks[#chunks+1] = nk
chunks[#chunks+1] = "="
if te == "table" then
chunks[#chunks+1] = table.serialize(e, true)
elseif te == "number" then
-- float output matters
chunks[#chunks+1] = string.format("%f", e)
elseif te == "string" then
-- escaped quotes matter
chunks[#chunks+1] = string.format("%q", e)
else -- te == "boolean"
chunks[#chunks+1] = tostring(e)
end
chunks[#chunks+1] = " "
end
if sub then
chunks[#chunks+1] = ", "
end
end
if sub then
chunks[#chunks+1] = "}"
end
return table.concat(chunks)
end