A Hat in Time hangs when starting with nir

[orbea]

OS: Slackware64-current
wine-nine-standalone: iXit/wine-nine-standalone@b41e162
mesa: https://github.com/mesa3d/mesa/commit/21c795ab0723cf0595768ea3f2165c10e89fd8c1
wine-staging: 4.10
GPU: RX Vega 56
driconf: 0.9.1

When running the game A Hat in Time with nir and wine-nine-standalone it will hang when starting until it consumes all of my available ram and the entire system halts. I am enabling nir with driconf and have 8 GB of ram available as well as an additional 14 GB between zram and swap.

I am using the GOG version of the game with the first expansion.

https://www.gog.com/game/a_hat_in_time
https://www.gog.com/game/a_hat_in_time_seal_the_deal

I bisected it in mesa to commit https://github.com/mesa3d/mesa/commit/c94b70a1781b8fb65be24b7e3a72462426cd86c4.

With help from #d3d9 I tried a few things:

• I can not reproduce it with NINE_SHADER=no_nir_ps and the game successfully starts, but it uses much more ram than without nir which makes it very slow and unplayable when it starts to hit swap.
• I can reproduce it with NINE_SHADER=no_nir_vs.
• I can not reproduce it with NINE_SHADER=no_nir_ps,no_nir_vs and the game works fine.

I tried commenting the following line in mesa, but it did not make any difference.

https://github.com/mesa3d/mesa/blob/f2dc0f28728af63e1a79756dab06a7035fecb590/src/gallium/auxiliary/nir/tgsi_to_nir.c#L2113

I looked at perf top while the issue was reproducing, but it does not seem to reveal anything.

[orbea]

Wine segfaulting with nir as discussed in issue #350 might be related to this.

[Venemo]

I managed to reproduce this problem with the Steam version of A Hat in Time. There is certainly a problem.

	RAM use	What happens?
Nine (no NIR) mesa 19.0	5 GiB	slow to load, then works OK
Nine (no NIR) mesa 19.1	5 GiB	slow to load, then works OK
Nine + NIR (no PS) mesa 19.1	9 GiB	same
Nine + NIR (PS and VS) mesa 19.1	all	eats all RAM, then crashes with meaningless backtrace
wined3d mesa 19.1	3.5 GiB	works OK

[axeldavy]

I don't know where was the leak reported previously June 2019), but I have found a leak in the live shader cache introduced for radeonsi in November 2019.
With a workaround to disable radeonsi memory shader cache (I think this one is going to end up removed for 32bits builds as it means more address space eaten), I can get the trace running.

Finally the long loading time is removed with a cache of tgsi->nir. It goes from 339s to 41s the second run.

[orbea]

@axeldavy Is there any work being done in mesa for this I could follow?

[axeldavy]

Well it's all in mesa master

[axeldavy]

Actually this part https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5017
is not yet merged

[orbea]

Fixed and I can start A Hat in Time again here with the current mesa master, thanks!

https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4980