How to speedup instances operations?

Discussion related to the LuxCore functionality, implementations and API.
bartek_zgo
Posts: 103
Joined: Mon Oct 26, 2020 11:42 am

How to speedup instances operations?

Post by bartek_zgo »

Hi Guys,
let say that I want to create garden full of flowers (5000 flowers). The flower is composed by 2 layers: stem and petals.
So at the beginning I add geometry to the scene. Than I create the first instance:

Code: Select all

std::string propertyPrefix = "scene.objects.stem";
Properties* props = new Properties();
props->Set(Property(propertyPrefix + ".id")(colorId));
props->Set(Property(propertyPrefix + ".shape")("stemMesh"));
props->Set(Property(propertyPrefix + ".material")("stemMaterial"));
props->Set(Property(propertyPrefix + ".transfromation")(transform));
pScene->Parse(*props);
delete props;
Than I create 4999 copies. Initially I was repeating the code above but it was very slow. Thanks to TAO now I use much faster way.

Code: Select all

pScene->DuplicateObject("stem", "stem_", 4999, transform);
Than I repeat all the code for petals.

Now let say that I want to remove half of them.

Code: Select all

std::string instancePrefix_s = "stem_";
std::string instanceName;
for (int n = 2500; n < 5000; n++)
{
	instanceName = instancePrefix_s + std::to_string(n);
	pScene->DeleteObject(instanceName);
}

instancePrefix_s = "petals_";
for (int n = 2500; n < 5000; n++)
{
	instanceName = instancePrefix_s + std::to_string(n);
	pScene->DeleteObject(instanceName);
}

The code is working but it is extremely slow. Removing 2500 instances of stem takes about 8s. Another 8s for petals.
How can I speedup?
User avatar
TAO
Developer
Developer
Posts: 837
Joined: Sun Mar 24, 2019 4:49 pm
Location: France
Contact:

Re: How to speedup instances operations?

Post by TAO »

To speed up even more for instancing I highly recommended you to use any type of unsorted map variable to check if an instance has been already added in the instance lists if it is the case skip the object and go to the next object. it can significantly improve the time of parsing in many cases.
About removing the objects story is a little bit different, Are you doing that at the render time or doing it before starting the render?
Omid Ghotbi (TAO)
Last update information
viewtopic.php?f=5&t=1010&start=590#p30154
artstation https://www.artstation.com/omidghotbi
bartek_zgo
Posts: 103
Joined: Mon Oct 26, 2020 11:42 am

Re: How to speedup instances operations?

Post by bartek_zgo »

I'm doing it at render time. I call BeginSceneEdit(), remove 2500 instances and call EndSceneEdit()
User avatar
TAO
Developer
Developer
Posts: 837
Joined: Sun Mar 24, 2019 4:49 pm
Location: France
Contact:

Re: How to speedup instances operations?

Post by TAO »

Normally I'm doing it like this.
First session.BeginSceneEdit()
then in a try and cache, I will update the scene, delete or parse, and after that in another try and cache, I will end it using session.EndSceneEdit() to ending updating the scene by nature I will stop the session if anything goes wrong in the cache section by calling the session.Stop().
Pausing and resuming the session during the update can help the process because normally all processes will be stopped until you resume it after updating, normally it will free more resources so PC should be able to use them as well.

Code: Select all

session.Pause();
if (changes)
{
    session.BeginSceneEdit();
    try
    {
        //updating the scene.....
    }
    cache
    {
        //error report ........
    }
    
    try
    {
        session.EndSceneEdit();
    }
    cache
    {
        //error report
        session.Stop();
    }
}
if (session.IsInPause())
     session.Resume();
I will test the problem you have to see if I can find any solution for it.
By the way, I noticed in some scenes restarting the session or even parsing the whole session from the beginning is faster than doing something like deleting too many objects.
try what i suggested and see if it can improve your code speed, I will get back to you if I found a good actual working solution for your deleting problem.
Omid Ghotbi (TAO)
Last update information
viewtopic.php?f=5&t=1010&start=590#p30154
artstation https://www.artstation.com/omidghotbi
bartek_zgo
Posts: 103
Joined: Mon Oct 26, 2020 11:42 am

Re: How to speedup instances operations?

Post by bartek_zgo »

I will try to add session.Pause(). Maybe this will help.
One question. Is there a command to remove all instances? I have seen that creating instances with duplicate command is extremely fast. So if there is a fast command to remove all, I could recreate missing 2500 instances.
I can not restart the session from beginning. The whole scene is quinte big, so I wold take too much time. What is more, I do not store history of all modifications done by user during the live session. So starting from 0 and executing all commands from history is not a good idea. I know that there is renderConfig->Export command, but is is extremely slow, or you have another solution in mind?
It wold be nice to have command similar to duplicate. Live Remove(instance_prefix, start_idx, count);
bartek_zgo
Posts: 103
Joined: Mon Oct 26, 2020 11:42 am

Re: How to speedup instances operations?

Post by bartek_zgo »

Another question. What is the fastest way to update position of all 5000 instances?
bartek_zgo
Posts: 103
Joined: Mon Oct 26, 2020 11:42 am

Re: How to speedup instances operations?

Post by bartek_zgo »

Currently, to update position, I repeat the procedure described int original post. I create first object than I duplicate it 4999 times. I use the same names, so luxcore overwrite them. But still, removing instances is a disaster
User avatar
TAO
Developer
Developer
Posts: 837
Joined: Sun Mar 24, 2019 4:49 pm
Location: France
Contact:

Re: How to speedup instances operations?

Post by TAO »

bartek_zgo wrote: Thu Feb 24, 2022 2:08 pm Currently, to update position, I repeat the procedure described int original post. I create first object than I duplicate it 4999 times. I use the same names, so luxcore overwrite them. But still, removing instances is a disaster
in my opinion, retranslating the whole scene can be faster in this situation. I may be wrong but in my experience that is what I found.
Omid Ghotbi (TAO)
Last update information
viewtopic.php?f=5&t=1010&start=590#p30154
artstation https://www.artstation.com/omidghotbi
User avatar
TAO
Developer
Developer
Posts: 837
Joined: Sun Mar 24, 2019 4:49 pm
Location: France
Contact:

Re: How to speedup instances operations?

Post by TAO »

Maybe if dade have enough time he can add an option to delete multiple objects or instance at the same time, Also the option to disable or make an object/ multiple objects hidden in render time.
Omid Ghotbi (TAO)
Last update information
viewtopic.php?f=5&t=1010&start=590#p30154
artstation https://www.artstation.com/omidghotbi
User avatar
TAO
Developer
Developer
Posts: 837
Joined: Sun Mar 24, 2019 4:49 pm
Location: France
Contact:

Re: How to speedup instances operations?

Post by TAO »

I add the code to luxcore, I will request dade to give me access to the luxcore repo so I can push the code on server
If you can complete the luxcore from the source you can do it like this.

add these lines in loxcore.h after DeleteObject definition around line 1079. it should look like something like that.

Code: Select all

	virtual void DeleteObject(const std::string &objName) = 0;

	virtual void DeleteObjects(std::vector<std::string> &names) = 0;

	virtual void DeleteObjectsInstance(const std::string &prefixName, const unsigned int count, const unsigned int start) = 0;
add these lines in scene.h after DeleteObject definition around line 157. it should look like something like that.

Code: Select all

	void DeleteObject(const std::string &objName);
	void DeleteObjects(std::vector<std::string> &names);
	void DeleteObjectsInstance(const std::string &prefixName, const unsigned int count, const unsigned int start);
and finally, add these lines in scene.cpp after DeleteObject definition around line 481. it should look like something like that.

Code: Select all

void Scene::DeleteObject(const string &objName) {
	if (objDefs.IsSceneObjectDefined(objName)) {
		const SceneObject *oldObj = objDefs.GetSceneObject(objName);
		const bool wasLightSource = oldObj->GetMaterial()->IsLightSource();

		// Check if the old object was a light source
		if (wasLightSource) {
			editActions.AddActions(LIGHTS_EDIT | LIGHT_TYPES_EDIT);

			// Delete all old triangle lights
			const ExtMesh *mesh = oldObj->GetExtMesh();
			for (u_int i = 0; i < mesh->GetTotalTriangleCount(); ++i)
				lightDefs.DeleteLightSource(oldObj->GetName() + TRIANGLE_LIGHT_POSTFIX + ToString(i));
		}

		objDefs.DeleteSceneObject(objName);

		editActions.AddAction(GEOMETRY_EDIT);
	}
}

void Scene::DeleteObjects(std::vector<std::string> &names) {
	BOOST_FOREACH(const string  &objName, names) {
		DeleteObject(objName);
	}
}

void Scene::DeleteObjectsInstance(const std::string &prefixName, const unsigned int count, const unsigned int start) {
	for (unsigned int i = start; i < start + count; i++)
	{
		string objectName = prefixName + std::to_string(i);
		DeleteObject(objectName);
	}
}
now you can compile the code and use it like this.
Scene->DeleteObject(objName); // delete one single object as always. objName = object name in String
Scene->DeleteObjects(objNames); // delete a list of object. objectNames = vector of object names in String
Scene->DeleteObjectsInstance(objPrefixName, count); // delete multiple instance objects. objPrefixName = instance prefix name in String, count = number of instance to be deleted, start = is the number of instance to start to delete from it
To make it safer you can check if the start + count is not bigger than existing instance numbers. as the existence of the object will be always checked in the DeleteObject function it should not be a problem.
it is possible to make all functions as it is in duplicating instances in the same name but I prefer different names to avoid confusion.
If you are using Pyluxcore you need to add the functions there too.
you can manipulate the code to fit your needs.
Last edited by TAO on Tue Mar 01, 2022 12:57 pm, edited 4 times in total.
Omid Ghotbi (TAO)
Last update information
viewtopic.php?f=5&t=1010&start=590#p30154
artstation https://www.artstation.com/omidghotbi
Post Reply