Page 3 of 4

Re: Luxcore and memory

Posted: Fri Aug 10, 2018 4:36 pm
by lacilaci
I will check back on this, thanks for the input. This memory usage is my biggest concerns for now.

Re: Luxcore and memory

Posted: Fri Aug 10, 2018 4:59 pm
by B.Y.O.B.
Dade wrote: Fri Aug 10, 2018 4:33 pm Side note: you can reduce Film memory usage by disabling OpenCL image pipeline (property film.opencl.enable). Indeed, some complex image pipeline will run a lot slower if using the CPU.
By the way, this setting is not exposed in the addon yet, it is always enabled (if you have an OpenCL build).

However, I have done all my tests on this matter so far with a Non-OpenCL build on Linux, so the OpenCL imagepipeline was not enabled.

Re: Luxcore and memory

Posted: Sat Aug 11, 2018 7:12 am
by lacilaci
Alright so here are some results of path vs tilepath ram usage:

scene in blender before rendering:
3.3GB mostly empty scene and cycles at any resolution gives below 4GB RAM usage

luxcore:

cpu+PATH:
256x256: 5.8 GB
denoiser: 5.9 GB

2048x2048: 8 GB
denoiser: 10.3 GB

cpu+TILEPATH:
256x256: 5.8 GB
denoiser: 5.9 GB

2048x2048: 7.2 GB
denoiser: 9.5 GB

OpenCL+PATH(using also CPU):
256x256: 6.3 GB
denoiser: 6.3 GB

2048x2048: 9.8 GB
denoiser: 12.1 GB

OpenCL+TILEPATH(using also CPU):
256x256: 6.3 GB
denoiser: 6.3 GB

2048x2048: 8.9 GB
denoiser: 11.2 GB

OpenCL+PATH(GPU only/GTX1060-6GB):
256x256: 5.2 GB
denoiser: 5.2 GB

2048x2048: 8 GB
denoiser: 10.3 GB

OpenCL+TILEPATH(GPU only/GTX1060-6GB):
256x256: 5.2 GB
denoiser: 5.2 GB - not working, black image as denoised result

2048x2048: 7.8 GB
denoiser: 10 GB - not working, black image as denoised result


So in conclusion, not only luxcore uses too much memory in general, even at low resolution, the memory usage skyrockets at 2K resolution(that's not even hi-res that's for mobile viewing) in a mostly empty scene with no textures and no hdri. Also even at 2K the denoiser performance is terribly slow. The differences between Path and TilePath were in all cases minimal and it's bad vs bad.

If you suspect problem could be elsewhere, like in the scene or luxcore usage itself. It is more or less the same version of the scene I gave Simon few days ago. Since I'm noob with luxcore please do check yourselves, I might be doing something wrong in the end.

Re: Luxcore and memory

Posted: Sat Aug 11, 2018 10:08 am
by Dade
lacilaci wrote: Sat Aug 11, 2018 7:12 am cpu+PATH:
256x256: 5.8 GB
denoiser: 5.9 GB
Is this this denoiser enabled Vs. disabled ?

Re: Luxcore and memory

Posted: Sat Aug 11, 2018 10:14 am
by lacilaci
Dade wrote: Sat Aug 11, 2018 10:08 am
lacilaci wrote: Sat Aug 11, 2018 7:12 am cpu+PATH:
256x256: 5.8 GB
denoiser: 5.9 GB
Is this this denoiser enabled Vs. disabled ?
no it's memory consumption during rendering phase vs denoiser phase...

I assumed disabled denoiser won't affect rendering itself.

Re: Luxcore and memory

Posted: Sat Aug 11, 2018 10:58 am
by Dade
lacilaci wrote: Sat Aug 11, 2018 10:14 am I assumed disabled denoiser won't affect rendering itself.
Collecting and saving the statistics for denoising requires an HUGE amount of data per pixel. If you disable the denoiser, the memory usage will go down a lot.

Re: Luxcore and memory

Posted: Sat Aug 11, 2018 11:02 am
by lacilaci
Dade wrote: Sat Aug 11, 2018 10:58 am
lacilaci wrote: Sat Aug 11, 2018 10:14 am I assumed disabled denoiser won't affect rendering itself.
Collecting and saving the statistics for denoising requires an HUGE amount of data per pixel. If you disable the denoiser, the memory usage will go down a lot.
damn, I did not expect that... I'm gonna do more tests tomorrow morning, but...
Using denoiser is pretty important so I hope that something can be done about that.
Now I'm really curious what will the results be without denoiser.

Re: Luxcore and memory

Posted: Sat Aug 11, 2018 11:30 am
by lacilaci
Alright, so here is the updated test - denoiser means denoiser phase during rendering and denoiser disabled means... disabled.

scene in blender before rendering:
3.3GB mostly empty scene and cycles at any resolution gives below 4GB RAM usage usage

luxcore:

cpu+PATH:
256x256: 5.8 GB
denoiser: 5.9 GB
denoiser disabled: 4.5 GB

2048x2048: 8 GB
denoiser: 10.3 GB
denoiser disabled: 5.5 GB

cpu+TILEPATH:
256x256: 5.8 GB
denoiser: 5.9 GB
denoiser disabled: 4.5 GB

2048x2048: 7.2 GB
denoiser: 9.5 GB
denoiser disabled: 6 GB

OpenCL+PATH(using also CPU):
256x256: 6.3 GB
denoiser: 6.3 GB
denoiser disabled: 6.3 GB

2048x2048: 9.8 GB
denoiser: 12.1 GB
denoiser disabled: 7.4 GB

OpenCL+TILEPATH(using also CPU):
256x256: 6.3 GB
denoiser: 6.3 GB
denoiser disabled: 6.3GB

2048x2048: 8.9 GB
denoiser: 11.2 GB
denoiser disabled: 7.8 GB

OpenCL+PATH(GPU only/GTX1060-6GB):
256x256: 5.2 GB
denoiser: 5.2 GB
denoiser disabled: 5.2 GB

2048x2048: 8 GB
denoiser: 10.3 GB
denoiser disabled: 5.7 GB

OpenCL+TILEPATH(GPU only/GTX1060-6GB):
256x256: 5.2 GB
denoiser: 5.2 GB - not working, black image as denoised result
denoiser disabled: 5.2 GB

2048x2048: 7.8 GB
denoiser: 10 GB - not working, black image as denoised result
denoiser disabled: 6.7GB

I hope I didn't make this somewhat confusing :D

Re: Luxcore and memory

Posted: Sat Aug 11, 2018 6:24 pm
by Dade
lacilaci wrote: Sat Aug 11, 2018 11:30 am cpu+PATH:
256x256: 5.8 GB
denoiser: 5.9 GB
denoiser disabled: 4.5 GB

2048x2048: 8 GB
denoiser: 10.3 GB
denoiser disabled: 5.5 GB
As you see, the denoiser requires to store a lot of data for each pixel (and, no, it is not avoidable, only mitigabile with the use of same already existing parameter). However it is a fixed cost related only to the number of pixel not the complexity of the scene (polygons count, materials, image maps, etc.)

What it can be avoided is the use of a local Film for each thread in order to reduce the increase of memory used with the increase of rendering threads (if you try to do a PATHCPU 2048x2048 rendering with 1 threads and with 10, you will se a HUGE increment in memory usage).

Re: Luxcore and memory

Posted: Sun Aug 12, 2018 6:52 am
by lacilaci
Dade wrote: Sat Aug 11, 2018 6:24 pm
lacilaci wrote: Sat Aug 11, 2018 11:30 am cpu+PATH:
256x256: 5.8 GB
denoiser: 5.9 GB
denoiser disabled: 4.5 GB

2048x2048: 8 GB
denoiser: 10.3 GB
denoiser disabled: 5.5 GB
As you see, the denoiser requires to store a lot of data for each pixel (and, no, it is not avoidable, only mitigabile with the use of same already existing parameter). However it is a fixed cost related only to the number of pixel not the complexity of the scene (polygons count, materials, image maps, etc.)

What it can be avoided is the use of a local Film for each thread in order to reduce the increase of memory used with the increase of rendering threads (if you try to do a PATHCPU 2048x2048 rendering with 1 threads and with 10, you will se a HUGE increment in memory usage).
8192x8192:

2.5GB scene in blender

cpu+path+logpower

denoiser disabled:
1 thread: 9.7 GB
8 thread: 20 GB

cycles: 3.7 GB