High RAM usage and long startup time with small tiles

Use this forum for general user support and related questions.
Forum rules
Please upload a testscene that allows developers to reproduce the problem, and attach some images.
Post Reply
User avatar
B.Y.O.B.
Developer
Developer
Posts: 4146
Joined: Mon Dec 04, 2017 10:08 pm
Location: Germany
Contact:

High RAM usage and long startup time with small tiles

Post by B.Y.O.B. »

Hi,

not sure if this is some kind of bug or if I'm configuring something wrong, but I observe the following behaviour:
Smaller tile sizes lead to higher RAM usage and longer startup time (time between finished export and start of the render) than large tile sizes.
I'm testing TILEPATHCPU.
See the attached image.
Attachments
scrn_2017-12-20_11-57-15.png
User avatar
Dade
Developer
Developer
Posts: 5672
Joined: Mon Dec 04, 2017 8:36 pm
Location: Italy

Re: High RAM usage and long startup time with small tiles

Post by Dade »

B.Y.O.B. wrote: Wed Dec 20, 2017 11:01 am not sure if this is some kind of bug or if I'm configuring something wrong, but I observe the following behaviour:
Smaller tile sizes lead to higher RAM usage and longer startup time (time between finished export and start of the render) than large tile sizes.
I'm testing TILEPATHCPU.
The list of tiles to render is initialized (according an Hilbert curve) at the start. Now, it the tiles are small and the image resolution is high, there can be a lot of tiles to add to the list. I guess it can become a not trivial task in some case (i.e. slow CPU, very small tiles, very high resolution) but I'm a bit surprised it can become a slow process.

What image resolution are you using with tile size 8 ?

P.S. for each tile 128x128, you have 256 (!) 8x8 tiles so the list initialization process is expected to be 256 times slower (however it should be 256 slower of something very fast so it should be still not noticeable).
Support LuxCoreRender project with salts and bounties
User avatar
B.Y.O.B.
Developer
Developer
Posts: 4146
Joined: Mon Dec 04, 2017 10:08 pm
Location: Germany
Contact:

Re: High RAM usage and long startup time with small tiles

Post by B.Y.O.B. »

Dade wrote: Wed Dec 20, 2017 1:08 pm What image resolution are you using with tile size 8 ?
960x540px (half of full HD).
That's 120*68 = 8160 tiles.

Here's the scene, but I guess it should be reproducable with any scene: https://www.dropbox.com/s/2nb2wkv5fvugi ... s.zip?dl=1
User avatar
Dade
Developer
Developer
Posts: 5672
Joined: Mon Dec 04, 2017 8:36 pm
Location: Italy

Re: High RAM usage and long startup time with small tiles

Post by Dade »

The source of the problem is in having a single small film for each tile (instead of one single big film). So the cost of allocating a tile, at the startup, is not trivial.
Once we had one big film but it was changed to be able to render the same tile multiple times (for instance, when you have only one tile to render, LuxCore still uses all CPU cores available instead of running with a single thread). So it is a trade-off of between memory/startup costs and running speed.

I have alleviated the problem by supporting multi-threading in the initialization. The initialization of your scene, now takes 3.15 seconds:

Code: Select all

[LuxCore][0.726] Configuring 12 CPU render threads
[LuxCore][3.880] Tiles initialization time: 3.15 secs
[LuxCore][4.008] Film OpenCL image pipeline
instead of 21.13 seconds:

Code: Select all

[LuxCore][0.715] Configuring 12 CPU render threads
<The old code miss the "Tiles initialization time" message>
[LuxCore][21.840] Film OpenCL image pipeline
It is now about 6 times as expected on six core + hyper-threading.

Note, the source of the problem is also in using a such small size for the tiles (i.e. 8x8). It has only drawback and no advantages. 32x23 would be a more appropriate size for the rendering.
Support LuxCoreRender project with salts and bounties
User avatar
B.Y.O.B.
Developer
Developer
Posts: 4146
Joined: Mon Dec 04, 2017 10:08 pm
Location: Germany
Contact:

Re: High RAM usage and long startup time with small tiles

Post by B.Y.O.B. »

Dade wrote: Thu Dec 21, 2017 12:10 pm Note, the source of the problem is also in using a such small size for the tiles (i.e. 8x8). It has only drawback and no advantages. 32x23 would be a more appropriate size for the rendering.
You know how it is, as a programmer you always test the boundaries first :D
I'll raise the soft_min in BlendLuxCore so normal users don't stumble on this so easily.
Post Reply