Tool for headless OpenGL compilation

[slipher]

The test-compile-glsl program creates an OSMesa (offscreen Mesa) context, compiles the GLSL shaders, and exits. This could be used in a CI script to ensure shaders compile as suggest by @illwieckz. On Debian I was able to get OSMesa by sudo apt install libosmesa6-dev. Enable in CMake by -DBUILD_TEST_COMPILE_GLSL.

GLEW supposedly has a linux-osmesa flavor. I built that and tried to use it, but it just returned null pointers for the functions (in the same way as that crashing bug with 0.51 on Mac). So I had to disable GLEW for this binary. I wonder if we could get rid of GLEW altogether? Just get the extension function pointers and store them in the glconfig2 struct. It doesn't seem like GLEW does anything that difficult, but I'm probably missing something.

For testing against an old version of GLSL, the GLSL compiler directives seem to be the key. #version at the beginning determines what version of the shading language is allowed, and additional #extension directives can enable more functions. Bitwise operators in GLSL<1.3 are controlled with one of these directives. The OpenGL version doesn't seem to be important, so I didn't request a specific version for the context. Anyway the lowest "core profile" is 3.1 so it's not possible to restrict yourself to 2.1.

[illwieckz]

Hmm, I was more thinking about running the engine in an headless X11, so we can do everything an user can do, testing any OpenGL version or extension provided by Mesa llvmpipe software OpenGL driver to catch regressions like the GLSL 1.20 one, to load a nexe game (to catch compatibility breakage) or also run a demo (to catch crashes like the particle one), etc.

Of course this would make the CI much more complex, as may need to download the actual Unvanquished release, but that would be more reusable and less intrusive to the daemon code itself.

[slipher]

Hmm, I was more thinking about running the engine in an headless X11, so we can do everything an user can do, testing any OpenGL version or extension provided by Mesa llvmpipe software OpenGL driver to catch regressions like the GLSL 1.20 one, to load a nexe game (to catch compatibility breakage) or also run a demo (to catch crashes like the particle one), etc.

Catching the particle one could already be done by running the ttyclient, as that crash happens in the cgame. The ttyclient includes pretty much everything except rendering and input code.

This tool may be useful to have a cross-platform way to test the shaders instead of depending on the CI or a special environment. As an alternative, we could try just adding a cvar to force the shading language #version directive, and perhaps a script that sets the version and other cvars to disable extensions, to simulate a minimal supported environment. However, this depends on GL vendors reliably implementing language versioning; I'm not sure if they all do that or if some will be like the Microsoft C++ compiler in not having any support for specific standard versions.

The tool could also be used to test combinations of different extensions being enabled (in case starting the whole renderer is too slow). Not sure if there are any interesting interactions or if an all on/all off test would be good enough.

[illwieckz]

Catching the particle one could already be done by running the ttyclient, as that crash happens in the cgame. The ttyclient includes pretty much everything except rendering and input code.

Oh that's good to know! We may want to run a demo with the tty at CI time.

This tool may be useful to have a cross-platform way to test […]

Yeah, sure.

The tool could also be used to test combinations of different extensions being enabled (in case starting the whole renderer is too slow). Not sure if there are any interesting interactions or if an all on/all off test would be good enough.

There are some cases with old hardware able to compile shaders for low presets but not able to compiler shaders for higher presets…

[illwieckz]

Anyway, I also now have a script to run the game on Linux on an headless X server, I wrote it today and it already helped me to catch and fix one regression from #478 by running a 16-bit desktop, and even catch and revert a compat breaking change in the same PR. I got them at the first run of my script… This is so useful. I don't know where I can store that file, maybe on Dæmon repository because that would make it available to Unvanquished as well and from CI tools for both engine and game.

[slipher]

Besides extensions (and extension-disabling cvars) there are some cvars which control whether certain shaders are compiled at all, including:

r_dynamicLight
r_reflectionMapping
r_bloom
r_liquidMapping
r_ssao
r_FXAA

[VReaperV]

Interestingly enough I found this pr while looking through open ones. Interestingly because I had a similar idea to add shader checking to CI. Instead of starting the whole renderer etc. though, we can use https://github.com/KhronosGroup/glslang. We'd only need to add the correct defines and such to the shaders (also do the shader and uniform post processing for material shaders, but that's easy).

Also to an earlier question about whether it should test any specific combinations: we've had some regressions that only came up when using certain combinations, like #1314 and #1300.

[slipher]

Interestingly enough I found this pr while looking through open ones. Interestingly because I had a similar idea to add shader checking to CI. Instead of starting the whole renderer etc. though, we can use https://github.com/KhronosGroup/glslang. We'd only need to add the correct defines and such to the shaders (also do the shader and uniform post processing for material shaders, but that's easy).

That might be nice. Although there could be a downside of missing errors that only happen with specific vendors. We had a couple of those just this month with one compile not allowing 1.0 / (integer) and another not allowing & between signed an unsigned. Unless the compiler is super strict about standard compliance?

Since making this PR, I have added unit testing where the tests are linked together with the Daemon engine (which may be any of client, ttyclient, server, or dummyapp). So far we don't have any tests that need more than the dummyapp (which is even smaller than server), but the idea is to support tests that depend on any part of the engine code, up to even the full client. If I return to this topic I might want to try making GLSL compilation a part of the test suite. Then the tests could be run with any compiler vendor. Then again, it would be difficult to get most of them running automatically on a server which would deprive the tests of a lot of their value.

[VReaperV]

Interestingly enough I found this pr while looking through open ones. Interestingly because I had a similar idea to add shader checking to CI. Instead of starting the whole renderer etc. though, we can use https://github.com/KhronosGroup/glslang. We'd only need to add the correct defines and such to the shaders (also do the shader and uniform post processing for material shaders, but that's easy).

That might be nice. Although there could be a downside of missing errors that only happen with specific vendors. We had a couple of those just this month with one compile not allowing 1.0 / (integer) and another not allowing & between signed an unsigned. Unless the compiler is super strict about standard compliance?

Since making this PR, I have added unit testing where the tests are linked together with the Daemon engine (which may be any of client, ttyclient, server, or dummyapp). So far we don't have any tests that need more than the dummyapp (which is even smaller than server), but the idea is to support tests that depend on any part of the engine code, up to even the full client. If I return to this topic I might want to try making GLSL compilation a part of the test suite. Then the tests could be run with any compiler vendor. Then again, it would be difficult to get most of them running automatically on a server which would deprive the tests of a lot of their value.

How would we test it for a particular vendor though? We'd need servers running on that particular hardware or something. AMD has some tools for offline cross-compilation, but I haven't been able to make them work.

I'm not sure how strict glslang is, but considering it's made by Khronos and is used to compile into SPIR-V among other things, I'd say it's pretty strict.

[VReaperV]

Well, I suppose for Mac it's less of an issue since they don't have like a million different graphics cards, where each could have some issue. Well, we've had more driver-specific rather than hardware-specific bugs, which probably makes things a bit better.

[slipher]

I'm not sure how strict glslang is, but considering it's made by Khronos and is used to compile into SPIR-V among other things, I'd say it's pretty strict.

Who knows, they might believe in the "liberal in what you accept, strict in what you emit" principle.

How would we test it for a particular vendor though? We'd need servers running on that particular hardware or something.

I don't really expect we'd get CI automated tests going for anything other than Mesa. It would probably be more useful for manually invoking. Like illwieckz could quickly test which features compile on some low-end machine. Or we haven't tested on Mac for 6 months, let's run the test suite to quickly cycle through some option combinations.

We might even combine the two ideas and have tests in the test suite that shell out to glslang to verify the shaders. The advantage of that would be avoiding the hassle of maintaining a separate build target just for GLSL stuff.

[VReaperV]

We might even combine the two ideas and have tests in the test suite that shell out to glslang to verify the shaders. The advantage of that would be avoiding the hassle of maintaining a separate build target just for GLSL stuff.

Sounds good.

[VReaperV]

Unless the compiler is super strict about standard compliance?

It's also a reference compiler:

Glslang is the official reference compiler front end for the OpenGL ES and OpenGL shading languages. It implements a strict interpretation of the specifications for these languages. It is open and free for anyone to use, either from a command line or programmatically. The OpenGL and OpenGL ES working groups are committed to maintaining consistency between the reference compiler and the corresponding shading language specifications.

Purpose
The primary purpose of the reference compiler is to identify shader portability issues. If glslang accepts a shader without errors, then all OpenGL and OpenGL ES implementations claiming to support the shader's language version should also accept the shader without errors. Likewise, if glslang reports an error when compiling a shader, all OpenGL and OpenGL ES implementations for that language version should report errors, unless the glslang errors are caused by differences in implementation-defined limits or extension support (see below).

This wouldn't detect the cases like "We had a couple of those just this month with one compile not allowing 1.0 / (integer) and another not allowing & between signed an unsigned.", because those are driver bugs, but other than that it should work well. It just needs something to set all the cvars/defines. Can just iterate over each value (for bool cvars at least) to make sure they all work, plus all the build macros and post-processing.

[slipher]

SDL_VIDEODRIVER=offscreen exists, so custom code for offscreen rendering is not necessary.