Page 1 of 2

Huge RAM usage during kernel compilation

Posted: Tue Aug 06, 2019 2:41 pm
by marcatore
Is it normal that during the kernel compilation the RAM usage doubles?

I have a scene that needs 4Gb (at least this is what task manager says) but during the kernel compilation RAM goes to 8GB.

Anyone else?

Re: Huge RAM usage during kernel compilation

Posted: Tue Aug 06, 2019 2:46 pm
by Dade
marcatore wrote: Tue Aug 06, 2019 2:41 pm Is it normal that during the kernel compilation the RAM usage doubles?

I have a scene that needs 4Gb (at least this is what task manager says) but during the kernel compilation RAM goes to 8GB.

Anyone else?
NVIDIA GPU ? Some OpenCL compiler is both slow and memory angry, however, if it is the compiler, the amount of memory used will not be scene related but code related. So a scene using the same amount of features will still use 4GB even if it is 40GB large (instead of 4GB) or even 4Kb large.

It is a kind of fixed cost related to the features. It should also be not CUDA PTX related so it should happen only the first time, than the code is cached (OpenCL C is compiled to CUDA PTX, aka "assembler", as far as I know).

Re: Huge RAM usage during kernel compilation

Posted: Tue Aug 06, 2019 2:57 pm
by marcatore
NVIDIA here.
Update on RAM usage.
I've killed blender. Closed any other programs.
16Gb of total ram.
Started again blender and the scene.
Press render.
Just before kernel compilation I was less than 5Gb. Kernel compilation started and fastly grow up to 15Gb (ram usage of blender process).
And it's super slow.

I can't understand when you talk about the caching of the kernel compilation.
It seems that it never cache anything cause it redo everytime... full compilation is needed just for one additional feature? Is it not possible to have a kernel compilation for just new features?

Re: Huge RAM usage during kernel compilation

Posted: Tue Aug 06, 2019 3:02 pm
by Dade
marcatore wrote: Tue Aug 06, 2019 2:57 pm I can't understand when you talk about the caching of the kernel compilation.
Just try to run the same rendering 2 times, the second should be very fast.

marcatore wrote: Tue Aug 06, 2019 2:57 pm It seems that it never cache anything cause it redo everytime... full compilation is needed just for one additional feature? Is it not possible to have a kernel compilation for just new features?
I have tried but the driver is so smart to recompile everything anyway :roll:

In general, it is the problem described here: viewtopic.php?f=5&t=1080#p12555
OpenCL should now support all recursive materials/textures without the need to recompile the kernel (i.e. material/texture interpreter);
P.S. AMD compiler doesn't suffer of this problem.

Re: Huge RAM usage during kernel compilation

Posted: Tue Aug 06, 2019 3:04 pm
by marcatore
Understood. Now I'm stuck cause it seems that it doesn't finish the kernel compilation.

I can't understand what's going on.

Could be a material that make things bad?

Re: Huge RAM usage during kernel compilation

Posted: Tue Aug 06, 2019 3:09 pm
by Dade
marcatore wrote: Tue Aug 06, 2019 3:04 pm Could be a material that make things bad?
Recursive materials or textures are usually main source of problems (a material referencing another material like mix, coating, etc, or a texture referencing another texture like add, mix, etc.).

Re: Huge RAM usage during kernel compilation

Posted: Tue Aug 06, 2019 3:18 pm
by marcatore
I have a material with 30 textures mixed by the objectID.

I'll check to disable it... it's the most complex I have in terms of nodes

Re: Huge RAM usage during kernel compilation

Posted: Tue Aug 06, 2019 3:46 pm
by lacilaci
Do you by any chance try to do a border rendering? There seems to be an issue with that.. Or some of the caches (caustic cache?) These are 2 instances where I get some memory leak/blender crash...

Re: Huge RAM usage during kernel compilation

Posted: Tue Aug 06, 2019 4:01 pm
by marcatore
Found it... it was that material.

Re: Huge RAM usage during kernel compilation

Posted: Tue Aug 06, 2019 4:05 pm
by lacilaci
marcatore wrote: Tue Aug 06, 2019 4:01 pm Found it... it was that material.
What material, might be good to know what material causes such memory issuses.