Luxcore and memory

Use this forum for general user support and related questions.
Forum rules
Please upload a testscene that allows developers to reproduce the problem, and attach some images.
User avatar
FarbigeWelt
Donor
Donor
Posts: 1046
Joined: Sun Jul 01, 2018 12:07 pm
Location: Switzerland
Contact:

Re: Luxcore and memory

Post by FarbigeWelt »

To stress memory usage a bit a setup a test scene with 250 trees with particle system. Thanks to B.Y.O.B. I have been able to reduce time creating a tree group. Most tricky thing is the orientation, I think the behaviour of particle system regarding rotation is strange for a long time. One has to rotate an object or a group until one can use it properly for randomzied rotation around veritcal axis of a an upright standing object on the emitter plane.

I don't see much unexpected memory usage nor with opencl path neither with opencl tile path.
But rendering 2xHD resolution blender is not very responsive, means I can wait several seconds until blender reacts to mouse inputs if blender shows any reaction at all. And this even with an overal CPU usage of less than 2%.
The tough blender response is not the case rendering HD resolution only.
Hazel 2.zip
Blender Scence
(3.34 MiB) Downloaded 157 times
250 Hazels OpenCL path tile
250 Hazels OpenCL path tile
For the texture of the leaf I made a simple sketch to avoid copy right issues. ;)
Light and Word designing Creator - www.farbigewelt.ch - aka quantenkristall || #luxcorerender
MacBook Air with M1
User avatar
lacilaci
Donor
Donor
Posts: 1969
Joined: Fri May 04, 2018 5:16 am

Re: Luxcore and memory

Post by lacilaci »

FarbigeWelt wrote: Sun Aug 12, 2018 5:39 pm To stress memory usage a bit a setup a test scene with 250 trees with particle system. Thanks to B.Y.O.B. I have been able to reduce time creating a tree group. Most tricky thing is the orientation, I think the behaviour of particle system regarding rotation is strange for a long time. One has to rotate an object or a group until one can use it properly for randomzied rotation around veritcal axis of a an upright standing object on the emitter plane.

I don't see much unexpected memory usage nor with opencl path neither with opencl tile path.
But rendering 2xHD resolution blender is not very responsive, means I can wait several seconds until blender reacts to mouse inputs if blender shows any reaction at all. And this even with an overal CPU usage of less than 2%.
The tough blender response is not the case rendering HD resolution only.

Hazel 2.zip
250 Hazels OpenCL path tile .jpg

For the texture of the leaf I made a simple sketch to avoid copy right issues. ;)
In memory it's still just one tree not 250 if it's in the particle system.
And as David said it's not scene complexity but multiple films and high resolution+denoiser issue. Although, even in the ideal case of using lowres no denoiser and single core, luxcore still uses quite a bit more ram than cycles.
But I guess it is because cycles is much better inegrated into blender.
User avatar
FarbigeWelt
Donor
Donor
Posts: 1046
Joined: Sun Jul 01, 2018 12:07 pm
Location: Switzerland
Contact:

Re: Luxcore and memory

Post by FarbigeWelt »

[quote=lacilaci post_id=5403 time=1534131748 user_id=211
FarbigeWelt wrote: Sun Aug 12, 2018 5:39 pm To stress memory usage a bit a setup a test scene with 250 trees with particle system.

I don't see much unexpected memory usage nor with opencl path neither with opencl tile path.
a) In memory it's still just one tree not 250 if it's in the particle system.

b) But I guess it is because cycles is much better inegrated into blender.
[/quote]

a) Are you sure? I made a test with 1 M particles. Referenced object: a single rectangle with an image texture glossy material. An image with 50x255. During export process task manager reported 28 GB for blender. It took luxcore about half an hour to build all duplicate before it threw some errors in console and killed blender.

Yes, increasing resolution takes mem. 2HD took about 1.6 GB, 4HD about 5.6 GB. Opening 32 bit float in PS takes 1 GB, after conversion to 16 bit, PS takes 3.6 GB, why ever. Maybe related to undo stack. It still allocated tjis ram after closing the file.

Maybe there should be one result layer for large resolutions and smaller render layers per core like every 1/cores pixel and displaying only render of a core until end conditions are reached or esc has been pressed.

b) Why do you think so?
Light and Word designing Creator - www.farbigewelt.ch - aka quantenkristall || #luxcorerender
MacBook Air with M1
User avatar
B.Y.O.B.
Developer
Developer
Posts: 4146
Joined: Mon Dec 04, 2017 10:08 pm
Location: Germany
Contact:

Re: Luxcore and memory

Post by B.Y.O.B. »

FarbigeWelt wrote: Mon Aug 13, 2018 10:57 am a) Are you sure? I made a test with 1 M particles. Referenced object: a single rectangle with an image texture glossy material. An image with 50x255. During export process task manager reported 28 GB for blender. It took luxcore about half an hour to build all duplicate before it threw some errors in console and killed blender.
lacilaci is right, the point of instances is that the whole mesh is only loaded into memory once.
The instances just need a transformation matrix and some extra info (e.g. material), so they need very little memory.
However they still need some memory (for example, a matrix is 4 * 4 * sizeof(float) = 64 Bytes (a float is 4 Bytes).
So if you instance 100,000 trees, you need the memory for the base mesh plus 100,000 * 64 = 6,400,000 Bytes = 6.4 MB for the instances.
However an instance is more than just the matrix, so you will need a bit more.

You should not test instances with simple meshes: A quad is just two triangles, so about 6 vertices * sizeof(float) = 24 Bytes. With a simple mesh, you save almost nothing by using instancing compared to duplicating the 24 Bytes around.

Anyway, this thread is not about instancing, it's about film memory usage (at least I assume that it is causing most of the issues).
User avatar
FarbigeWelt
Donor
Donor
Posts: 1046
Joined: Sun Jul 01, 2018 12:07 pm
Location: Switzerland
Contact:

Re: Luxcore and memory

Post by FarbigeWelt »

B.Y.O.B. wrote: Mon Aug 13, 2018 11:22 am
Anyway, this thread is not about instancing, it's about film memory usage (at least I assume that it is causing most of the issues).
Thanks for the explanation, which makes sense.

I forgot to ask, what is about light groups? Does memory usage grow with their number?
Light and Word designing Creator - www.farbigewelt.ch - aka quantenkristall || #luxcorerender
MacBook Air with M1
User avatar
B.Y.O.B.
Developer
Developer
Posts: 4146
Joined: Mon Dec 04, 2017 10:08 pm
Location: Germany
Contact:

Re: Luxcore and memory

Post by B.Y.O.B. »

FarbigeWelt wrote: Mon Aug 13, 2018 12:19 pm I forgot to ask, what is about light groups? Does memory usage grow with their number?
Yes, each light group's contribution needs a separate buffer.
I think the memory usage of a light group is width * height * 3 * sizeof(float) Bytes.
User avatar
Dade
Developer
Developer
Posts: 5672
Joined: Mon Dec 04, 2017 8:36 pm
Location: Italy

Re: Luxcore and memory

Post by Dade »

B.Y.O.B. wrote: Mon Aug 13, 2018 12:27 pm
FarbigeWelt wrote: Mon Aug 13, 2018 12:19 pm I forgot to ask, what is about light groups? Does memory usage grow with their number?
Yes, each light group's contribution needs a separate buffer.
I think the memory usage of a light group is width * height * 3 * sizeof(float) Bytes.
It is "4 *" (i.e. the weight) for eye path buffer and "3 *" for light path buffer (used only by BIDIRCPU and LIGHTCPU, in conjunction with the eye path buffer).
Support LuxCoreRender project with salts and bounties
Post Reply