Luxcore and memory

[FarbigeWelt]

To stress memory usage a bit a setup a test scene with 250 trees with particle system. Thanks to B.Y.O.B. I have been able to reduce time creating a tree group. Most tricky thing is the orientation, I think the behaviour of particle system regarding rotation is strange for a long time. One has to rotate an object or a group until one can use it properly for randomzied rotation around veritcal axis of a an upright standing object on the emitter plane.

I don't see much unexpected memory usage nor with opencl path neither with opencl tile path.
But rendering 2xHD resolution blender is not very responsive, means I can wait several seconds until blender reacts to mouse inputs if blender shows any reaction at all. And this even with an overal CPU usage of less than 2%.
The tough blender response is not the case rendering HD resolution only.

Hazel 2.zip

Blender Scence

(3.34 MiB) Downloaded 326 times

250 Hazels OpenCL path tile

For the texture of the leaf I made a simple sketch to avoid copy right issues.

[lacilaci]

To stress memory usage a bit a setup a test scene with 250 trees with particle system. Thanks to B.Y.O.B. I have been able to reduce time creating a tree group. Most tricky thing is the orientation, I think the behaviour of particle system regarding rotation is strange for a long time. One has to rotate an object or a group until one can use it properly for randomzied rotation around veritcal axis of a an upright standing object on the emitter plane.

I don't see much unexpected memory usage nor with opencl path neither with opencl tile path.
But rendering 2xHD resolution blender is not very responsive, means I can wait several seconds until blender reacts to mouse inputs if blender shows any reaction at all. And this even with an overal CPU usage of less than 2%.
The tough blender response is not the case rendering HD resolution only.

Hazel 2.zip
250 Hazels OpenCL path tile .jpg

For the texture of the leaf I made a simple sketch to avoid copy right issues.

In memory it's still just one tree not 250 if it's in the particle system.
And as David said it's not scene complexity but multiple films and high resolution+denoiser issue. Although, even in the ideal case of using lowres no denoiser and single core, luxcore still uses quite a bit more ram than cycles.
But I guess it is because cycles is much better inegrated into blender.

[FarbigeWelt]

[quote=lacilaci post_id=5403 time=1534131748 user_id=211

To stress memory usage a bit a setup a test scene with 250 trees with particle system.

I don't see much unexpected memory usage nor with opencl path neither with opencl tile path.

a) In memory it's still just one tree not 250 if it's in the particle system.

b) But I guess it is because cycles is much better inegrated into blender.
[/quote]

a) Are you sure? I made a test with 1 M particles. Referenced object: a single rectangle with an image texture glossy material. An image with 50x255. During export process task manager reported 28 GB for blender. It took luxcore about half an hour to build all duplicate before it threw some errors in console and killed blender.

Yes, increasing resolution takes mem. 2HD took about 1.6 GB, 4HD about 5.6 GB. Opening 32 bit float in PS takes 1 GB, after conversion to 16 bit, PS takes 3.6 GB, why ever. Maybe related to undo stack. It still allocated tjis ram after closing the file.

Maybe there should be one result layer for large resolutions and smaller render layers per core like every 1/cores pixel and displaying only render of a core until end conditions are reached or esc has been pressed.

b) Why do you think so?

[B.Y.O.B.]

a) Are you sure? I made a test with 1 M particles. Referenced object: a single rectangle with an image texture glossy material. An image with 50x255. During export process task manager reported 28 GB for blender. It took luxcore about half an hour to build all duplicate before it threw some errors in console and killed blender.

lacilaci is right, the point of instances is that the whole mesh is only loaded into memory once.
The instances just need a transformation matrix and some extra info (e.g. material), so they need very little memory.
However they still need some memory (for example, a matrix is 4 * 4 * sizeof(float) = 64 Bytes (a float is 4 Bytes).
So if you instance 100,000 trees, you need the memory for the base mesh plus 100,000 * 64 = 6,400,000 Bytes = 6.4 MB for the instances.
However an instance is more than just the matrix, so you will need a bit more.

You should not test instances with simple meshes: A quad is just two triangles, so about 6 vertices * sizeof(float) = 24 Bytes. With a simple mesh, you save almost nothing by using instancing compared to duplicating the 24 Bytes around.

Anyway, this thread is not about instancing, it's about film memory usage (at least I assume that it is causing most of the issues).

[FarbigeWelt]

Anyway, this thread is not about instancing, it's about film memory usage (at least I assume that it is causing most of the issues).

Thanks for the explanation, which makes sense.

I forgot to ask, what is about light groups? Does memory usage grow with their number?

[B.Y.O.B.]

I forgot to ask, what is about light groups? Does memory usage grow with their number?

Yes, each light group's contribution needs a separate buffer.
I think the memory usage of a light group is width * height * 3 * sizeof(float) Bytes.

[Dade]

I forgot to ask, what is about light groups? Does memory usage grow with their number?

Yes, each light group's contribution needs a separate buffer.
I think the memory usage of a light group is width * height * 3 * sizeof(float) Bytes.

It is "4 *" (i.e. the weight) for eye path buffer and "3 *" for light path buffer (used only by BIDIRCPU and LIGHTCPU, in conjunction with the eye path buffer).