Scene translation take forever or crash with big textures

[TAO]

Blender crash with big HDRI or any png texture will take forever to translate and some time crash.
It's the same in render and interactive render. I have seen exact same behavior in 3dsmax and blender.
the error I get is different, sometimes out of memory, sometimes unexpected issues and in other times crash with no error.

[Dade]

You seem to run out of memory and it probably takes forever because you are swapping memory on HD. Check the memory usage during the process. What is the size of the HDR you are trying to use ? How much RAM do you have ? There may be something wrong in the HDR/PNG you are trying to use. I need to have access to an example.

[TAO]

You seem to run out of memory and it probably takes forever because you are swapping memory on HD. Check the memory usage during the process. What is the size of the HDR you are trying to use ? How much RAM do you have ? There may be something wrong in the HDR/PNG you are trying to use. I need to have access to an example.

I do not know why is happening, I have at least 24 GB out of 32GB of ram available on my pc and 6 Gb out of 8GB Of GPU ram.
HDRI is an 8K image from HDRIheaven and the png that i used is 4k simple raindrops.

When I check the visual studio debugger the most time is about the png translation. I'm not sure but I think luxCore converts every image to png and that is a huge process with big images.
When I saving the scene with filesaver, all images converting to png and getting way too large on HDD.

There is an Eifel tower in the scene to but it takes no time to translate.
The Eifel tower (Low poly).

HDRI

https://hdrihaven.com/hdri/?c=skies&h=kloppenheim_06

RainDrops

After a few crashes I manage to render the scene but it's unstable. here is the results.

[Dade]

I need a test scene. If it is HDR/PNG related, a scene with a cube should be enough to show the problem.

LuxCore has rendered thousands of scenes with HDRs and PNGs so I'm quite sure it is something specific of the scene or/and images.

[TAO]

I need a test scene. If it is HDR/PNG related, a scene with a cube should be enough to show the problem.

LuxCore has rendered thousands of scenes with HDRs and PNGs so I'm quite sure it is something specific of the scene or/and images.

Sure.
Step to reproduce the issue.
1- Select LuxCore as the renderer
2- Select any 16K/bigger HDRI as an environment map
3- Try to change the ENV gamma

Issues:
First, changing parameters is taken too much time compared to other renderers with the exact same steps.
Second, After a few try blender/3dsmax/LuxCore getting unstable and most likely crashes.

I think the issue is coming from converting all maps in LuxCore and not just use them. (it somehow no other choice with plugins like blender plugin)

Proof of concept:

Screen-Capture.mp4

(2.56 MiB) Downloaded 172 times

Sample scene:
http://3dfine.com/content/LuxCore-BigMap_Crash.zip

It can be reproduced with any other big HDRI, or Png texture, used on objects material or environment map.

[Martini]

Step to reproduce the issue.
1- Select LuxCore as the renderer
2- Select any 16K/bigger HDRI as an environment map
3- Try to change the ENV gamma

Issues:
First, changing parameters is taken too much time compared to other renderers with the exact same steps.
Second, After a few try blender/3dsmax/LuxCore getting unstable and most likely crashes.

On my system (32 GB RAM) I do not get a crash when following your steps, however I do notice that while re-using the same viewport render session, changing the ENV gamma increases memory usage each time a unique value for gamma is entered (also, changing the gamma is very slow and cannot be done in realtime - I have to enter a value on the keyboard).

I can repeat this process and see the memory usage climbing in Process Explorer. However, if I re-enter a gamma value that I already used, it is both much faster, and also the memory usage does not increase. Thus I can concluded that these 'images' are cached somehow, and each unique gamma-adjusted image is kept in memory.

Once I go back to solid view, the memory used by all maps is freed, and goes back to the same level as before the viewport render started.

I guess your crash happens when your memory is filled? I could not repro a crash, I think Windows just keeps resizing my paging file as it gets slower and slower each time I increase a new gamma.

You can see in this image, I increase the gamma from 1.0, 1.01, 1.02, 1.03, 1.04, 1.05, 1.06, 1.07, 1.08, 1.09, 1.10.
Then there is a small plateau where I go back and repeat each one of these previous values.
Then I increase 1.11, 1.12, 1.13, 1.14, 1.15, 1.16, 1.17, 1.18, 1.19 - things start to really slow down as I hit my physical memory limit and it starts paging - 1.20, 1.21, 1.22, 1.23, 1.24, 1.25, 1.26, 1.27, 1.28, 1.29, 1.30

And finally you can see when I go back to solid view, all the memory is freed.

[TAO]

Did you try 16K or any big hdri, because with the small HDRI is not going to happen, and did the interactive render getting slow when you change the game or other parameters?
I already upload a sample 16k HDRI from hdriheaven, I'm pretty sure that happens on many pc because at my office we test it on at least 5 different PC with a different configuration.
Sometimes just slowing down on parameter changes and sometimes crash.
I need to mention that the HDRI we used is 320MB and in our case environment map cause the page file to get bigger very soon with every change, 1.2 -> 1.8 -> 2.4, and with every change getting bigger and bigger, this scene has just one map, now imaging a scene with 100 maps.

System info that helps the case.

[Martini]

Did you try 16K or any big hdri, because with the small HDRI is not going to happen, and did the interactive render getting slow when you change the game or other parameters?

Yes, I tried with both hansaplatz_16k.hdr (390MB) and spiaggia_di_mondello_16k.hdr (358MB).

And yes, Blender became unresponsive (although it did not "white out" like in your video) until the gamma was processed. After that, though, I was able to navigate around the viewport with the render going with no issues, even modify geometry in edit mode in rendered mode. Only when modifying the gamma did I have slowdowns. Tested with both CPU rendering and GPU rendering.

System info that helps the case.
Screenshot 2021-02-12 093009.jpg

Hey, I have the same CPU as you I have 2 RTX 2080 Tis, not sure if that matters though because I saw the same behaviour in CPU rendering.

[TAO]

Yes, I tried with both hansaplatz_16k.hdr (390MB) and spiaggia_di_mondello_16k.hdr (358MB).
And yes, Blender became unresponsive (although it did not "white out" like in your video) until the gamma was processed. After that, though, I was able to navigate around the viewport with the render going with no issues, even modify geometry in edit mode in rendered mode. Only when modifying the gamma did I have slowdowns. Tested with both CPU rendering and GPU rendering.

Hey, I have the same CPU as you I have 2 RTX 2080 Tis, not sure if that matters though because I saw the same behaviour in CPU rendering.

The white screen is because I click on the blender when it is unresponsive, also the blender will crash after a few changes on the game and then after that try to change the game to a number like 2.0 by typing the game value and hit the enter to set it.
Try that as I explain and let see if that's your case too.
On slower PC changing parameters like gamma will take much more time especially when other PNG maps available on the scene and an unresponsive blender will cause instability eventually.
by the way, Nice PC configuration, respect.

[Martini]

This is my experience. Unfortunately, I had to upload it to Drive as it was too large for the forum

https://drive.google.com/file/d/1ul3igy ... sp=sharing (14.3MB)

The white screen is because I click on the blender when it is unresponsive, also the blender will crash after a few changes on the game and then after that try to change the game to a number like 2.0 by typing the game value and hit the enter to set it.
Try that as I explain and let see if that's your case too.

Click+drag on the slider for me is basically not a viable option - it's not interactive at all as it takes multiple seconds to become responsive again. Also, you can see the memory usage increasing on the left side with every update.

On slower PC changing parameters like gamma will take much more time especially when other PNG maps available on the scene and an unresponsive blender will cause instability eventually.

Agreed. Best not to do that while rendering, I suppose. It would be nice if the memory leak were fixed, though. I suppose it was not designed for this use-case, but rather it behaves as though it were used by multiple (different) scene objects - I suppose it can't tell the difference. Basically every time you change the gamma, it allocated another 1.0GB - 1.5GB of memory.

by the way, Nice PC configuration, respect.

Cheers! I built it for Blender/LuxCore a couple of years ago and it's still a beast! Also, water cooled and of course OC'd