Huge performance drop on r300 with default Thunder and Vega scene

[illwieckz]

I'm trying to investigate a huge performance drop affecting some low-end cards.

I currently suspect:

• a bug in the driver itself.
• something stupid done on our side similar to the way our skybox code was iterating over GPU memory directly:
  Sky performance cost #849 (comment)
• a combination of the twos.

Here is the Thunder map, lowest preset, 640×480 resolution, default spectator scene (1024 -3456 448 -90 0), 3fps:

Here is the Vega map, lowest preset, 640×480 resolution, default spectator scene (1472 -1792 176 -180 0), 5fps:

This configuration is expected to achieve 20~30fps 60fps.

The testbench is a very low-end config, one of my favorites for identifying bottlenecks:

• CPU: AMD Athlon 64 X2 4400+, dual core 2.3GHz.
• RAM: 8Go DDR2 800MHz , 4×2Go Hynix HYMP125U64CP8-S6, dual channel.
• GPU: ATi Radeon X550 HyperMemory, RV370 (r300, tiny ALU), 128Mo 64-bit DDR, PCIe x16 1.0.
• OS: Ubuntu 23.10 mantic, Linux 6.5.0.
• Driver: Mesa 23.2.1 (stock) or Mesa 24.0.7 (self built).

In the Thunder map, I suspect the bottleneck comes from rendering something in that room and/or the outside. It can be the electric arc effect on those pillars, and/or the lightning effect on the top of the outside of the building, as this room as huge windows on the outside. Leaving that room and losing sight with the outside brings back performance, leaving that room without losing sight with the outside keeps the performance low.

In the Vega map, I suspect the bottleneck comes from rendering that globe or something around that globe. From the 4 sides of the room, I recover performance by turning back to that globe.

I identified places were just turning around the view enables/disables the performance loss in both maps.

Thunder, -30 -1497 10 -20 0:

Thunder, -30 -1497 10 112 0:

Vega, 667 -1792 176 0 0:

Vega: 667 -1792 176 -180 0:

As you see, just turning around the head without moving from the place you are enables/disables the cause of the bottleneck.

[illwieckz]

The cause of the bottleneck is very hard to track down.

Orbit fails to show me the calltrace, only identifying a very costly ioctl without any identified caller. Once, Orbit managed to relate the ioctl to some render thread in the Bottom-Up view.

And gprofng also shows me something suspect with openal, but what if the i/o on the sound card is actually slowed down by the whole PCIe bus being on knees because of agressive io for rendering?

But gperftools seems to track it back to the r300 DRM driver through Mesa, more explicitely:

• pprof3725656.0.pdf

That's why I suspect something similar to what we had with the sky: that GPU has a slow PCIe 1.0 x16 bus, and if we process the memory GPU directly in our computation, this can be very bad.

[VReaperV]

As you see, just turning around the head without moving from the place you are enables/disables the cause of the bottleneck.

What if you move backwards from there? If perf drops once the globe is in view, then it is the issue, otherwise it's not.

[illwieckz]

Both the electric arc effect and the globe materials use some tcMod:

textures/thunder_lightning/horizontallightning
{
	qer_editorImage textures/thunder_lightning_src/horizontallightning
	surfaceparm trans
	surfaceparm nolightmap
	surfaceparm nomarks
	surfaceparm nonsolid
	{
		map textures/thunder_lightning_src/horizontallightning
		blendfunc add
		tcmod scroll -8 -0
	}
}

models/mapobjects/vega/globe/globe
{
	surfaceparm nomarks
	surfaceparm nonsolid
	cull none

	{
		map models/mapobjects/vega/globe/globe
		tcmod scroll .05 0
	}
}

In the slow sky thread, SomaZ recommended to also look at tcMod:

Should probably also make sure any tcMods or else also don't read/write to tess.
-- #849 (comment)

[VReaperV]

tcMod uses a matrix uniform.

[illwieckz]

Commenting out the tcmod scroll line in the vega globe shader doesn't bring back the performance.

I also verified that doing stage->bundle[ 0 ].numTexMods = 0 in FinishStages also disables the scrolling but doesn't bring back the performance.

[illwieckz]

viewpos	relation	fps
511 -1792 176 -180 0	globe in the back	40
615 -1792 176 -180 0	globe in the back	30
773 -1792 176 -180 0	globe in the back	17
844 -1792 176 -180 0	globe in the back	14
893 -1792 176 -180 0	inside globe	11
934 -1792 176 -180 0	inside globe	10
997 -1792 176 -180 0	inside globe	10
1006 -1792 176 -180 0	globe in front	10
1074 -1792 176 -180 0	globe in front	10
1143 -1792 176 -180 0	globe in front	9
1288 -1792 176 -180 0	globe in front	6
1382 -1792 176 -180 0	globe in front	6
1621 -1792 176 -180 0	globe in front	4

[illwieckz]

One may also suspect the amount of tris, but vega was running properly in Tremulous.

map	viewpos	performance
thunder	1472 -1792 176 -180 0	slow
thunder	667 -1792 176 0 0	slow
thunder	667 -1792 176 -180 0	fast

map	viewpos	performance
vega	1024 -3456 448 -90 0	slow
vega	-30 -1497 10 -20 0	slow
vega	-30 -1497 10 112 0	fast

[illwieckz]

But I can render the whole plat23 map (0 3568 3194 -90 66) with all tris visible, and keep a smooth 30fps:

[VReaperV]

But I can render the whole plat23 map (0 3568 3194 -90 66) with all tris visible, and keep a smooth 30fps:

For comparing things like that it'd be helpful to know how many triangles are in each case in particular (some value of r_speeds shows that I believe).

[illwieckz]

Same computer, but with onboard nvidia GPU.

• GPU: Nvidia Geforce 6150 LE, NV4C/MCP61 (Curie, ), 128Mo shared DDR2, HyperTransport.

Where the comparison is not fair is that instead of dedicated memory the Nvidia chip uses shared DDR2 memory from the host and instead of PCIe 1.0 x16 bus the Nvidia uses HyperTransport (some onboard bus shared with CPU). This may help with ioctl when something on the GPU is read from/written to by the CPU.

The Nvidia has a larger ALU (but that should not make a difference as long as shader runs) and may accept larger images (or properly downscale them). The ATi one uses ARB_framebuffer_object but the Nvidia only provides the older EXT_framebuffer_object which is a subset (but that should make no difference because we only use the EXT_framebuffer_object part of ARB_framebuffer_object).

Like the ATi one is from the first generation of ATi cards to support OpenGL 2, that Nvidia card is from the first generation of Nvidia cards to support OpenGL 2, and is from the year after then ATi one. Both are low end, in fact the “LE“ string in the product name means “low end”.

That card can sustain solid 20fps on both scenes: (Thunder 1024 -3456 448 -90 0, Vega 1472 -1792 176 -180 0):

[illwieckz]

For comparison on the same computer and that old ATi card, the vega-beta1a map on Tremulous (some GrangerHub Tremulous 1.3 build) on OpenGL 1 renderer gets 130fps (almost 140fps) with lightmaps and 190fps with vertex lighting on same scene:

With the OpenGL 2 renderer it gets 40fps (almost 50fps):

Except for the vertex lighting, this is using the default Tremulous configuration (no “lowest”-like preset).

That other engine is also ridled with bugs, on such hardware the OpenGL1 renderer spawns lots of error logs every frame:

Mesa: error: GL_INVALID_VALUE in glLockArraysEXT(count)
Mesa: error: GL_INVALID_OPERATION in glUnlockArraysEXT(reexit)

and on such hardware the OpenGL2 renderer doesn't work at all with lightmaps, and some fonts are totally broken, and some texture blending is broken:

But the experiments says something: not only it may be possible to render such map on such hardware without any performance drop in that scene, but we may be able to achieve double the performance on such hardware. Edit: In fact we already achieve 60fps in other maps that don't trigger the slow down.

This test was done with the ATi card described in first message, on the computer described in first message.

[illwieckz]

That's not the same map build that is used in Tremulous, but they should not be that different. And the amount of tris should be similar:

[illwieckz]

So, it looks like I uncovered not only one performance problem but two.

First performance problem is that serious performance drop in some places.

Second performance problem is that we may be able to get twice the same performance on some lower-end hardware but don't get it.

I expect such hardware to perform better with the OpenGL1 renderer, as at the time most of the game were doing things the GL1 way and only using shaders for some effects, so the hardware is probably architectured for that in mind. On the contrary we use shaders for everything.

But since the OpenGL 2 renderer of ioquake3 seems to still be twice faster than us when doing things the shader way in that range of hardware, I believe we should also get twice the performance we already have (reaching 40fps on such map).

And I'm very annoyed by how much it is hard to track down this. I rebuilt entirely Orbit and its dependencies to run on that old CPU, I have a own build of Mesa and libdrm (even libpciaccess) built with debug symbols, I installed all the debug packages I could find for the installed packages of my distro, including the one of the libc, and Orbit still fails to track down the slow code backtrace.