GCC - compiling with optimizations causes graphical issues

[johnnyzero]

Compiling with certain versions of GCC at optimization levels O2, O3, and Os cause substantial graphical issues. Most noticeable are shaders. When running TOME, the game looks considerably darkened when a full-map FBO shader is used, and other shaders appear completely broken. Other graphical oddities can sometimes be seen, such as talent icons in TOME looking like they've been "scratched". Optimization levels O0, O1, and Og are working as expected.

This issue is highly reminiscent of this issue back in the pre-1.0 betas. I personally saw this issue using msys/mingw32, and was able to sidestep it by cross-compiling using mingw-w64 under Cygwin (4.5.x) and mingw-w64 under Debian (4.6.x).

I now see this issue again when compiling with GCC 4.8.2, both with native GCC and the mingw-w64 cross-compiler. Multiple users have reported such issues with the optimized binaries compiled for 1.1.2+. (issue 1) (issue 2)

Something to note is that none of the user-configurable GCC flags have an effect on these graphical glitches. I can compile with O1 and all of the -fsome_optimization flags from O2 and O3 without seeing any graphical issues.

[johnnyzero]

O1:O2


Difference image (O1 minus O2):

[Suslik]

Mwahaha, fixed. In core_lua.c:

Code: Select all

static int gl_fbo_toscreen(lua_State *L)
{
	//...
	if (lua_isnumber(L, 7))
	{
		r = luaL_checknumber(L, 7);
		g = luaL_checknumber(L, 8);
		b = luaL_checknumber(L, 9);
		a = luaL_checknumber(L, 10);
		GLfloat colors[4*4] = {
			r, g, b, a,
			r, g, b, a,
			r, g, b, a,
			r, g, b, a,
		};
		glColorPointer(4, GL_FLOAT, 0, colors);
	}
	else
	{
		GLfloat colors[4*4] = {
			1, 1, 1, 1,
			1, 1, 1, 1,
			1, 1, 1, 1,
			1, 1, 1, 1,
		};
		glColorPointer(4, GL_FLOAT, 0, colors);
	}
	//...
	glDrawArrays(GL_QUADS, 0, 4);
}

Color pointer is actually used only on glDrawArrays call and since color pointer is relative to if()-block it gets invalidated outside of it. Result was those weird colored quads! Fixed now.

[johnnyzero]

Fix confirmed! Binaries compiled with GCC 4.8.2 and the -O3 flag no longer exhibit the issues in the original post. Tested on both Linux and Windows.

Here's a link to the commit.