Multi thread parse/DefineMesh

Discussion related to the LuxCore functionality, implementations and API.
Post Reply
User avatar
TAO
Developer
Developer
Posts: 851
Joined: Sun Mar 24, 2019 4:49 pm
Location: France
Contact:

Multi thread parse/DefineMesh

Post by TAO »

Is there any way to do the scene Parse and/or DefineMesh from different threads at the same time?
I tried to use Mutex and Critical Section before this functions call but the performance suffers from all synchronization processes and in most cases, I didn't get any benefit from multi threads, in the best case I achieved 25% of performance.
On the other hand for the extensive process part from 3dsmax, I achieved at least 4 to 8 times better performance. considering those parts did not require too much synchronization and waiting for writing variables.
I tried to compare a few scene translation part performance with BlendLuxCore and because of the nature of the BlendLuxCore that mostly runs by the python language, the 3dsmax version has better performance, for example in my case, translation of almost 174000 objects (most of them are low poly) take around 160 seconds with Core i9 9900k and take no more than 8% of CPU usage.
Of course, the MaxToLux code needs to be optimized and it's not finished yet, as I started to code not so far and more focus on adding features that optimizing the code, and try to make it less complicated at this point, with optimized code may come better performance.
Long story short, my question is:
Is there any other solution available that anyone tried before to multi-thread these parts? or, any suggestions to put me in the right direction?
User avatar
Dade
Developer
Developer
Posts: 5672
Joined: Mon Dec 04, 2017 8:36 pm
Location: Italy

Re: Multi thread parse/DefineMesh

Post by Dade »

TAO wrote: Thu Jan 28, 2021 2:26 am Is there any way to do the scene Parse and/or DefineMesh from different threads at the same time?
No, it is not thread safe.
TAO wrote: Thu Jan 28, 2021 2:26 am Is there any other solution available that anyone tried before to multi-thread these parts? or, any suggestions to put me in the right direction?
Are you defining 174000 "different" objects ? Or are they mostly copies of the same object ?

Can you post a test scene ?

There may be many ways to accelerate the process but I need to see where is the bottleneck. For instance, you could merge the meshes by material in order to reduce the object count, etc.
Support LuxCoreRender project with salts and bounties
User avatar
TAO
Developer
Developer
Posts: 851
Joined: Sun Mar 24, 2019 4:49 pm
Location: France
Contact:

Re: Multi thread parse/DefineMesh

Post by TAO »

Dade wrote: Thu Jan 28, 2021 12:00 pm
TAO wrote: Thu Jan 28, 2021 2:26 am Is there any way to do the scene Parse and/or DefineMesh from different threads at the same time?
No, it is not thread safe.
TAO wrote: Thu Jan 28, 2021 2:26 am Is there any other solution available that anyone tried before to multi-thread these parts? or, any suggestions to put me in the right direction?
Are you defining 174000 "different" objects ? Or are they mostly copies of the same object ?

Can you post a test scene ?

There may be many ways to accelerate the process but I need to see where is the bottleneck. For instance, you could merge the meshes by material in order to reduce the object count, etc.
ost of them are different objects but I also have around 57000 instance objects that half of them are identical instance and the rest are instanced plus adding a few modifiers on top of them (I think that makes them a different object and not instance, not sure), I try to use instancing but I still don't know how to detect these objects are instanced from each other in 3dsmax, especially when I have a different modifier stack on top of a few of them.
One of the issues I faced is I don't know how to decide which objects are instanced and which is not.
The scene is too huge to be uploaded. around 1 GB.
Screenshot 2021-01-28 132349.jpg
But I think I can make a sample from a few objects, so, maybe you can help me to better understand instancing proccess.
User avatar
TAO
Developer
Developer
Posts: 851
Joined: Sun Mar 24, 2019 4:49 pm
Location: France
Contact:

Re: Multi thread parse/DefineMesh

Post by TAO »

Here is a simple scene to start with.
In this scene, all of these objects are instances from the 3dsmax point of view.

Object 2 is an instance from object 1 and they are identical.
Object 3 is a copy of object 2, exactly the same plus on top of that add a UV modifier to change the UV scale.
Object 4 is an instance of object 1 plus mesh modifier on top of it. (reference object)
if object one change all other objects will change, if any modifier change on object 3 and 4 will not affect the other objects.
Screenshot 2021-01-28 141439.jpg
02-instance-test.zip
(325.84 KiB) Downloaded 144 times
Are these objects considering instance at all?
User avatar
Dade
Developer
Developer
Posts: 5672
Joined: Mon Dec 04, 2017 8:36 pm
Location: Italy

Re: Multi thread parse/DefineMesh

Post by Dade »

As I wrote elsewhere, you need to use the DuplicateObject method to create a large number of instances, it is the correct way to create them.

You need also to detect instances to save memory: otherwise you are duplicating the meshes and using an insane amount of memory.

The scene you posted created 4 different meshes while 1 and 2 should share the same mesh (not the others).
Support LuxCoreRender project with salts and bounties
User avatar
TAO
Developer
Developer
Posts: 851
Joined: Sun Mar 24, 2019 4:49 pm
Location: France
Contact:

Re: Multi thread parse/DefineMesh

Post by TAO »

Thank you for the point. as I can see instance objects should be exactly the same and in my scene, there is about 26000 instance object that can speed up the project with instancing.
Still, I have a hard time using it in the correct way.
Post Reply