Add ReSTIR

[res2k]

Continuing discussion from NVIDIA#477 as upstream project is archived.

[res2k]

@abalfoort, continuing from NVIDIA#477 (comment):

I got around to looking into it, at least a little.
So I can reproduce the effect as well.

My rough hypothesis right now:
Not enough "good" samples.
The darker "smear" appears in areas on the screen that were just "revealed" (ie they were occluded by the weapon the previous frame(s)); initially, the chosen samples did not hit a light, turning out dark. Only over time more desirable samples are collected, improving the situation.

What to do about it? Not sure right now. Probably read a bit up on state of the ReSTIR art and see what options the pros came up with.

[res2k]

Addendum: Seems things get especially odd when sunlight is involved. Hm.

[abalfoort]

Addendum: Seems things get especially odd when sunlight is involved. Hm.

Or at least, more visible.

Good to see that you still are on this project.
If you are interested, I've moved my Q2RTX project (including the mission packs) to my Gitlab account.

[res2k]

@abalfoort, so after spending quite a while trying to understand more of how this is all supposed to work, and prodding the implementation, I think managed to address the issue you were still seeing (still visible smearing outdoors)...

First of all, be aware it's not entirely gone in all cases; but that is in the nature of the thing: when pixels are disoccluded, the denoiser initially has to guess a lot in that area, and that usually tends to result in comparatively darker pixels.
However, I at least tried to make not worse than the "old" RIS light selection.

As I found out, the ReSTIR implementation actually contained a few hacks that made things look alright when using skybox sky lighting (which I mostly used when checking it out previously), but no so much with physical sky lighting (which is comparively bright). So the main aim was to fix those hacks, yet to keep things still looking nicely.

Other things I found out on the journey:

• The "old" RIS light selection, in conjunction with the A-SVGF denoiser, is quite effective for Q2's lighting setup.
• The denoiser actually works quite well with input that changes every frame; ReSTIR's tendency to produce the same samples over multiple frames was actually detrimental to denoising quality, so a puzzle piece was to less aggressively reuse samples.
• Generally, the aggressive sample reuse seemed to result in lighting appearing to only change sluggishly. But again, some extent that's also due to how the denoiser works.

In conclusion, if you have time, I'd appreciate if you could give that latest branch a spin and see how it works for you.

[abalfoort]

@res2k Thanks for the hard work.
I have compiled (Debian 13) and only had a few warnings:

...
/home/arjen/dev/_hobby/q2rtx/src/src/refresh/vkpt/main.c: In function ‘add_dlights’:
/home/arjen/dev/_hobby/q2rtx/src/src/refresh/vkpt/main.c:1868:60: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
 1868 |                         light_entity_ids[(*num_lights)] = *(uint32_t*)&hash;
      |                                                            ^~~~~~~~~~~~~~~~
/home/arjen/dev/_hobby/q2rtx/src/src/refresh/vkpt/main.c: In function ‘instance_model_lights’:
/home/arjen/dev/_hobby/q2rtx/src/src/refresh/vkpt/main.c:1916:73: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
 1916 |                 light_entity_ids[entity_frame_num][num_model_lights] = *(uint32_t*)&hash;
      |                                                                         ^~~~~~~~~~~~~~~~
/home/arjen/dev/_hobby/q2rtx/src/src/refresh/vkpt/main.c: In function ‘update_mlight_prev_to_current’:
/home/arjen/dev/_hobby/q2rtx/src/src/refresh/vkpt/main.c:3007:39: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
 3007 |                 entity_hash_t hash = *(entity_hash_t*)&light_entity_ids[entity_frame_num][i];
      |                                       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...
[ 82%] Building C object src/CMakeFiles/client.dir/refresh/vkpt/transparency.c.o
/home/arjen/dev/_hobby/q2rtx/src/src/refresh/vkpt/transparency.c: In function ‘vkpt_build_cylinder_light’:
/home/arjen/dev/_hobby/q2rtx/src/src/refresh/vkpt/transparency.c:550:60: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
  550 |                         light_entity_ids[(*num_lights)] = *(uint32_t*)&hash;
      |                                                            ^~~~~~~~~~~~~~~~
[ 82%] Building C object src/CMakeFiles/client.dir/refresh/vkpt/uniform_buffer.c.o
...

I have started and played baseq2 a little and immediately saw a big difference. I will try to find some time this week to test this, also with the mission packs.

[res2k]

@res2k Thanks for the hard work. I have compiled (Debian 13) and only had a few warnings:

I'll look into them

I have started and played baseq2 a little and immediately saw a big difference. I will try to find some time this week to test this, also with the mission packs.

Awesome!
Take your time. After all, so did I ;)

[abalfoort]

I finally had some time.

Compiled both master and restir-update. The latter with pt_restir 0 and pt_restir 1 set:

demo	master branch frametime	restir-update branch, pt_restir 0, ms	percentage of master	restir-update branch, pt_restir 1, ms	percentage of master
crusher	23.457	23.183	98.8	23.251	99.1
demo1	24.110	24.289	100.7	23.843	98.9
demo2	24.794	24.897	100.4	25.097	101.2

The smearing is now almost completely gone, even in the mission packs. I see some ghosting but that could be my machine. On the other hand, only Q2RTX does that. Making a screenshot is no use because that only saves a single frame, not the ghosting. I don't know how to fix that.

[res2k]

The smearing is now almost completely gone, even in the mission packs.

Great!

I see some ghosting but that could be my machine. On the other hand, only Q2RTX does that. Making a screenshot is no use because that only saves a single frame, not the ghosting. I don't know how to fix that.

Hm, maybe you could make a short recording using some screen recording utility?

"On the other hand, only Q2RTX does that" - what are your other points of comparison?

[abalfoort]

I was mistaken. I just ran God of War and Shadow of the Tomb Raider and they had that both. So, it's my system.

Probably Linux and the latest proprietary Nvidia drivers (the .run installation files from their site) don't go well together.