[git] Faster serialization

[johnnyzero]

T-Engine has basic serialization support implemented in game/loader/pre-init.lua. This serialization is fine for what T-Engine needs, except that it is really slow in the current git* code. So, I've implemented a buffered version of serialize.

After loading a fresh session of ToME, the profile thread flushes CSETs. On my system, the current git code takes about 40 seconds to finish flushing csets, with the profile thread eating all the cpu cycles it can. Using my table.serialize implementation, flushing csets is done in about one second. Note that these times include zipping the resulting serialized data and sending the result to te4.org.

• More information on why current git serialization is so slow.
• *current git = commit 559709d / 2013-07-27
• Here's a link to a patch file.

Code: Select all

function table.serialize(src, sub, no_G, base)
	local chunks = {}
	if sub then 
		chunks[1] = "{"
	end
	for k, e in pairs(src) do
		local nk = nil
		local nkC = {}
		local tk, te = type(k), type(e)
		if no_G then
			if tk == "table" then
				nkC[#nkC+1] = "["
				nkC[#nkC+1] = table.serialize(k, true)
				nkC[#nkC+1] = "]"
			elseif tk == "string" then
				nkC[#nkC+1] = k
			else
				nkC[#nkC+1] = "["
				nkC[#nkC+1] = tostring(k)
				nkC[#nkC+1] = "]"
			end
		else
			if not sub then 
				nkC[#nkC+1] = (base and tostring(base) or "_G") 
			end
			nkC[#nkC+1] = "["
			if tk == "table" then 
				nkC[#nkC+1] = table.serialize(k, true)
			elseif tk == "string" then
				-- escaped quotes matter
				nkC[#nkC+1] = string.format("%q", k)
			else
				nkC[#nkC+1] = tostring(k)
			end
			nkC[#nkC+1] = "]"
		end

		nk = table.concat(nkC)

		-- These are the types of data we are willing to serialize
		if te == "table" or te == "string" or te == "number" or te == "boolean" then
			chunks[#chunks+1] = nk
			chunks[#chunks+1] = "="
			if te == "table" then
				chunks[#chunks+1] = table.serialize(e, true)
			elseif te == "number" then
				-- float output matters
				chunks[#chunks+1] = string.format("%f", e)
			elseif te == "string" then
				-- escaped quotes matter
				chunks[#chunks+1] = string.format("%q", e)
			else -- te == "boolean"
				chunks[#chunks+1] = tostring(e)
			end
			chunks[#chunks+1] = " "
		end
		
		if sub then 
			chunks[#chunks+1] = ", "
		end
	end
	if sub then 
		chunks[#chunks+1] = "}"
	end
			
	return table.concat(chunks)
end

[johnnyzero]

Update: the patch has been accepted.

Code: Select all

commit 6f43ddef9b0e454c7afc272c20c6227ce7f8f5c8
Author: DarkGod <darkgod@net-core.org>
Date:   Tue Oct 1 18:11:31 2013 +0200

    use faster serialization

[cttw]

Good job!

[Karagy]

Draw attention to Serializing table data into a string
It can proceed cycled tables.