Adaptive sampling improvements

[alpistinho]

In the posted example picture, the bright lamp appears noise-free to me in both renders.
However, the wall is clearly more noisy in the render with adaptive sampling.

In general, I agree that a noise metric should be based on the human vision system, but the metric that is currently used does not seem very good at this to me.

By the way, it looks like the current metric only compares old with new, but not neighboring pixels, is that correct?

I guess the metric is not good for this scene. The solution does not make many assumptions about the metric, so if we find a better one I can implement it to see if it's better. I will try looking for a new one, but it is possible that there is no single metric that works for all scenes

The metric is calculated by pixel, but it is filtered by a 9x9 box filter afterwards, so the final results considers 4 pixels in each direction, besides the central one. I am basically following what other people that implemented this technique did

[alpistinho]

So, I've first seen the issue with this scene (100 samples in both case, warmup and test step is 5 with adaptivity):
adaptivity-test1.png

Would it be possible for you to share this scene? I am helpless with modelling things

[B.Y.O.B.]

The metric is calculated by pixel, but it is filtered by a 9x9 box filter afterwards, so the final results considers 4 pixels in each direction, besides the central one. I am basically following what other people that implemented this technique did

I meant that it does not consider differences between neighbors when looking for noise, only difference between old and new passes.
Of course it's obvious that doing so would lead to oversampling in areas with high-frequency detail and on high-contrast edges. But it might be better than the current approach. We could also use a weighted mix between the current approach and contrast between neighbours.
For example, only use neighbour contrast if the old/new check showed a big difference between old and new (i.e. the pixel is probably undersampled):

Code: Select all

pixel_done = false
if (difference_old_new < old_new_threshold)
	pixel_done = true
else
	if (difference_with_neighbours < neighbour_threshold)
		pixel_done = true

I'm probably getting ahead a bit too far. Best would be to first look for relevant papers on this topic and see which one leads to the best results. Also it would be good to have a variety of test scenes to test everything with.

[Dade]

@Dade, can you give me a wiki account so I can update the SDL page when I change it?

I need your email address to be able to create the account

[FarbigeWelt]

I'm probably getting ahead a bit too far. Best would be to first look for relevant papers on this topic and see which one leads to the best results. Also it would be good to have a variety of test scenes to test everything with.

Do you remember my posts looking for an easy algorithm to reduce noise resp. restore image from a noisy picture? I found a one pixel solution but it has to store at least one value per pixel.

My opinion is noise is in both cases good visible for the human in dark as well as in bright parts of a picture. This means a good algorithm is required here.

[B.Y.O.B.]

I have done some experiments:

First, we have to think about how an ideal convergence AOV would look like.
My naive idea is: abs(noisy - noisefree)
So I created a noisy render at 10 spp and a denoised render from 50 spp as "ground truth" (I know, far from perfect, but it does the job for this example).

noisy (luminance), 10 spp

denoised (luminance) of 50 spp render

And this is the difference between the two, where noisy pixels are brighter and noise-free pixels are darker:

abs(noisy - denoised)

Now that we know what we are looking for, the problem becomes: How to obtain a "noisefree" image when we only have a noisy one?
One idea would be to use a powerful denoiser like OIDN. Problem with this is high RAM usage and long computation time.
Again, naive idea: we are looking for high-frequency noise. If we eliminate high-frequency noise, we might get something close enough to a "noisefree" image.
One super simple trick to eliminate high-frequency noise is to blur the image.
So I used a 3x3 gauss filter on the noisy 10 spp image:

noisy image blurred with 3x3 gauss filter

If we now calculate the difference between the noisy and this blurred image, we get:

abs(noisy - noisy_blurred)

Which is not so far from the "ideal convergence AOV"
However, it obviously creates halos on high-contrast edges.

Do you remember my posts looking for an easy algorithm to reduce noise resp. restore image from a noisy picture? I found a one pixel solution but it has to store at least one value per pixel.

Can you link the post?

[B.Y.O.B.]

In the last post I have omitted that it's not a good idea to use the raw convergence AOV as-is: it should be blurred to avoid introducing noise (in non-converged images) when some single pixels are over- and others are undersampled.

Some more of my findings so far:

All adaptive renders below were done with these settings:
batch.haltthreshold = 0.015625 (4/256)
batch.haltthreshold.warmup = 5
batch.haltthreshold.step = 5
batch.haltspp = 50


abs(noisy - noisy_blurred)
This is the idea proposed above, plus a blurring step afterwards to avoid the problem mentioned at the beginning.
Not so great.
A lot of pixels fall below the noise threshold very fast, most of the image is sampled rather uniformly.

abs(blurred - luminance)

abs(blurred - luminance) convergence

abs(blurred - luminance) samplecount

abs(log(noisy) - log(noisy_blurred))
I played around a bit with logarithms and tested out this formula.
It's a bit inspired from the MultiScale Retinex (MSR) algorithm.
In my opinion, it shows some promising results:

abs(log(blurred) - log(luminance))

abs(log(blurred) - log(luminance)) convergence

abs(log(blurred) - log(luminance)) samplecount

A 50 sample render with uniform sampling, for comparison:

Here's the code I use for testing (not a proper patch, just so you have an idea how the concrete implementation looks like): https://gist.github.com/Theverat/e5f82e ... a916f7d021
This snippet is in the file filmconvtest.cpp, where I commented out most of the lines after https://github.com/LuxCoreRender/LuxCor ... t.cpp#L101

[B.Y.O.B.]

Note that am not proposing to replace the current "old vs. new" comparison completely.
Rather, maybe we can mix it with other methods of noise detection to fix its shortcomings.
I am also aware that 2 times gaussian filter is quite heavy on the performance.

[alpistinho]

Thanks for this!

I will have a closer look when I am at the computer.

I am looking for related papers and will list here the ones I find promising:
https://ieeexplore.ieee.org/document/5636310
https://onlinelibrary.wiley.com/doi/ful ... /cgf.12592
https://cseweb.ucsd.edu/~ravir/Overbeck2009AWR.pdf
http://citeseerx.ist.psu.edu/viewdoc/do ... 1&type=pdf

[alpistinho]

About the actual development:

I've finished fixing LuxCoreUI, so there is now an extra AOV called NOISE and the CONVERGENCE is like it was before.

I will work now on the new parameters, so to improve the ergonomics and possibilities, and will have a look about the metric after that.

About the metric:

The scene provisory sent us seems to be a pathological case. If we have a closer look at the scene you've provided me, you can see that not only the bright regions have higher values, but also the shade under the wrinkled spheres. The Food scene also shows some regions with higher sample counts that are not the brightest in the scene. I don't know what exactly makes this new scene different.