How to speedup instances operations?

Discussion related to the LuxCore functionality, implementations and API.
User avatar
TAO
Developer
Developer
Posts: 850
Joined: Sun Mar 24, 2019 4:49 pm
Location: France
Contact:

Re: How to speedup instances operations?

Post by TAO »

Hi "Dade", if this is a correct approach please give me access to the repo so I can update it and also add more features. Thank you. if I need to improve it I will be more than happy to give me a few suggestions.
User avatar
Dade
Developer
Developer
Posts: 5672
Joined: Mon Dec 04, 2017 8:36 pm
Location: Italy

Re: How to speedup instances operations?

Post by Dade »

You can fill a "Pull Request", anyone can: https://docs.github.com/en/pull-request ... ll-request
Support LuxCoreRender project with salts and bounties
daros
Posts: 280
Joined: Thu Dec 12, 2019 3:25 pm
Location: inside human language
Contact:

Re: How to speedup instances operations?

Post by daros »

Wow! That's great! This was the last issue which forced our users to reinitialize the real time session. Thanks a lot!
User avatar
TAO
Developer
Developer
Posts: 850
Joined: Sun Mar 24, 2019 4:49 pm
Location: France
Contact:

Re: How to speedup instances operations?

Post by TAO »

Dade wrote: Fri Feb 25, 2022 9:11 pm You can fill a "Pull Request", anyone can: https://docs.github.com/en/pull-request ... ll-request
Thank you dade, a will create a pull request, just one question. should I work on new code and test it with the master branch always? and do you have any suggestions about the code?
User avatar
Dade
Developer
Developer
Posts: 5672
Joined: Mon Dec 04, 2017 8:36 pm
Location: Italy

Re: How to speedup instances operations?

Post by Dade »

TAO wrote: Sat Feb 26, 2022 7:47 am
Dade wrote: Fri Feb 25, 2022 9:11 pm You can fill a "Pull Request", anyone can: https://docs.github.com/en/pull-request ... ll-request
Thank you dade, a will create a pull request, just one question. should I work on new code and test it with the master branch always? and do you have any suggestions about the code?
Yes, the master branch is always the "next" main version, maintenance/minor versions, if they existing, are on a separate branches.
Support LuxCoreRender project with salts and bounties
bartek_zgo
Posts: 116
Joined: Mon Oct 26, 2020 11:42 am

Re: How to speedup instances operations?

Post by bartek_zgo »

Thanks TAO for your code.
Unfortunately I'm not able to compile. I cloned repository and started from 0. Start command firt_run and than build-64-sse2 LuxCore-sdk 5
Everything was perfect. No compile errors.
So I made changes in scene.cpp and scene.h. Run again build-64-sse2 LuxCore 5. No compile errors.
So I made changes in luxcore.h. And I got following error:

Code: Select all

/mnt/nodeHDD1/LuxCore/LuxCore_latest/LinuxCompile/LuxCore-sdk/src/luxcore/luxcoreimpl.cpp: In constructor ‘luxcore::detail::RenderConfigImpl::RenderConfigImpl(const luxrays::Properties&, luxcore::detail::SceneImpl*)’:
/mnt/nodeHDD1/LuxCore/LuxCore_latest/LinuxCompile/LuxCore-sdk/src/luxcore/luxcoreimpl.cpp:1225:44: error: invalid new-expression of abstract class type ‘luxcore::detail::SceneImpl’
   scene = new SceneImpl(renderConfig->scene);
                                            ^
In file included from /mnt/nodeHDD1/LuxCore/LuxCore_latest/LinuxCompile/LuxCore-sdk/src/luxcore/luxcoreimpl.cpp:39:
/mnt/nodeHDD1/LuxCore/LuxCore_latest/LinuxCompile/LuxCore-sdk/include/luxcore/luxcoreimpl.h:145:7: note:   because the following virtual functions are pure within ‘luxcore::detail::SceneImpl’:
 class SceneImpl : public Scene {
       ^~~~~~~~~
In file included from /mnt/nodeHDD1/LuxCore/LuxCore_latest/LinuxCompile/LuxCore-sdk/src/luxcore/luxcoreimpl.cpp:38:
/mnt/nodeHDD1/LuxCore/LuxCore_latest/LinuxCompile/LuxCore-sdk/include/luxcore/luxcore.h:1094:15: note:  ‘virtual void luxcore::Scene::DeleteObjects(std::vector<std::__cxx11::basic_string<char> >&)’
  virtual void DeleteObjects(std::vector<std::string> &objNames) = 0;
               ^~~~~~~~~~~~~
/mnt/nodeHDD1/LuxCore/LuxCore_latest/LinuxCompile/LuxCore-sdk/include/luxcore/luxcore.h:1101:15: note:  ‘virtual void luxcore::Scene::DeleteObjectsInstance(const string&, unsigned int, unsigned int)’
  virtual void DeleteObjectsInstance(const std::string &prefixName, const unsigned int count, const unsigned int start) = 0;
               ^~~~~~~~~~~~~~~~~~~~~

User avatar
TAO
Developer
Developer
Posts: 850
Joined: Sun Mar 24, 2019 4:49 pm
Location: France
Contact:

Re: How to speedup instances operations?

Post by TAO »

you need to add it in luxcoreimpl .
in luxcoreimpl.h around the line 206 add delete function deceleration, it should look like this:

Code: Select all

	void DeleteObject(const std::string &objName);
	void DeleteObjects(std::vector<std::string> &objNames);
	void DeleteObjectsInstance(const std::string &prefixName, const unsigned int count, const unsigned int start);
	void DeleteLight(const std::string &lightName);
	void DeleteLights(std::vector<std::string> &lightNames);
in luxcoreimpl.cpp around the line 1060 add functions as before you can add scenePropertiesCache.Clear(); in it too. it should look like this:

Code: Select all

void SceneImpl::DeleteObject(const string &objName) {
	API_BEGIN("{}", ToArgString(objName));

	// Invalidate the scene properties cache
	scenePropertiesCache.Clear();

	scene->DeleteObject(objName);

	API_END();
}

void SceneImpl::DeleteObjects(std::vector<std::string> &objNames) {

	// Invalidate the scene properties cache
	scenePropertiesCache.Clear();

	BOOST_FOREACH(const string  &objName, objNames) {
		scene->DeleteObject(objName);
	}
}

void SceneImpl::DeleteObjectsInstance(const std::string &prefixName, const unsigned int count, const unsigned int start) {

	// Invalidate the scene properties cache
	scenePropertiesCache.Clear();

	// Deleting the instances from the start amount till the counting number of instances
	for (unsigned int i = 0; i < start + count; i++)
	{
		string objectName = prefixName + std::to_string(i);
		scene->DeleteObject(objectName);
	}
}

void SceneImpl::DeleteLight(const string &lightName) {
	API_BEGIN("{}", ToArgString(lightName));

	// Invalidate the scene properties cache
	scenePropertiesCache.Clear();

	scene->DeleteLight(lightName);

	API_END();
}

void SceneImpl::DeleteLights(std::vector<std::string> &lightNames) {

	// Invalidate the scene properties cache
	scenePropertiesCache.Clear();

	// Separate the objects and send them to delete
	BOOST_FOREACH(const string  &lightName, lightNames) {
		scene->DeleteLight(lightName);
	}
}
avoid repeating the existing functions. I add the deleting multiple lights too if you do not need it to remove it or use the next message to add it to the source.
it should compile with no issue.
Last edited by TAO on Tue Mar 01, 2022 4:36 pm, edited 1 time in total.
User avatar
TAO
Developer
Developer
Posts: 850
Joined: Sun Mar 24, 2019 4:49 pm
Location: France
Contact:

Re: How to speedup instances operations?

Post by TAO »

in the last scene, i added deleting multiple lights too, so you need to remove it or added it as blew to other parts of the code.
in scene.h line 161 after DeleteLight deceleration add:

Code: Select all

	void DeleteLights(std::vector<std::string> &lightNames);
in luxcore.h line 1103 after DeleteLight deceleration add:

Code: Select all

	/*!
	 * \brief Deletes multiple lights from the scene.
	 *
	 * \param lightNames is the vector array name of the object to delete. Note: to delete
	 * area lights, use DeleteObject().
	 */
	virtual void DeleteLights(std::vector<std::string> &lightNames) = 0;
and finally in scene.cpp line 526 after DeleteLight deceleration add:

Code: Select all

void Scene::DeleteLights(std::vector<std::string> &lightNames) {
	// Separate the objects and send them to delete
	BOOST_FOREACH(const string  &lightName, lightNames) {
		DeleteLight(lightName);
	}
}
bartek_zgo
Posts: 116
Joined: Mon Oct 26, 2020 11:42 am

Re: How to speedup instances operations?

Post by bartek_zgo »

Hi. I have compiled the code. It is working but there is no time improvement.
I have analyzed the code and in my opinion the problem is in NamedObjectVector::DeleteObj(const std::string &name);
This code is dramatically slow. You should add DeleteObj(objectNamesArray) or something like that.
Take a look at this code. First you search index by name: GetIndex(name); (just this is not a happy solution when repeated many times).
But after this, you rebuild some index array! So it is done, each time you remove an instance. So if I remove 1000 instances, you rebuild the index array 1000 times!
First quite easy fix would be to pass a vector of instances names to remove, and rebuild indexes only once after removing all instances!
This is an easy way, that should speed up. But to do it well, it should be done in more complex way, that I'm not sure is it possible or not, because I don't know the code very well.
We want to remove some range of instances based on index. So I think GetIndex(name) could be called only once. As I suppose next instance should just have next index. I hope that if for example instance "petal_74" has index 138, "petal_75" has index 139. I hope so, but I don't know the data structure.
User avatar
TAO
Developer
Developer
Posts: 850
Joined: Sun Mar 24, 2019 4:49 pm
Location: France
Contact:

Re: How to speedup instances operations?

Post by TAO »

bartek_zgo wrote: Wed Mar 02, 2022 9:14 am Hi. I have compiled the code. It is working but there is no time improvement.
I have analyzed the code and in my opinion the problem is in NamedObjectVector::DeleteObj(const std::string &name);
This code is dramatically slow. You should add DeleteObj(objectNamesArray) or something like that.
Take a look at this code. First you search index by name: GetIndex(name); (just this is not a happy solution when repeated many times).
But after this, you rebuild some index array! So it is done, each time you remove an instance. So if I remove 1000 instances, you rebuild the index array 1000 times!
First quite easy fix would be to pass a vector of instances names to remove, and rebuild indexes only once after removing all instances!
This is an easy way, that should speed up. But to do it well, it should be done in more complex way, that I'm not sure is it possible or not, because I don't know the code very well.
We want to remove some range of instances based on index. So I think GetIndex(name) could be called only once. As I suppose next instance should just have next index. I hope that if for example instance "petal_74" has index 138, "petal_75" has index 139. I hope so, but I don't know the data structure.
i will check the code to see if there is any way to make the delete function faster as well.
in mean while the function corresponding to delete objects is in namedobjectvectore.cpp

Code: Select all

void NamedObjectVector::DeleteObj(const std::string &name) {
	const u_int index = GetIndex(name);
	objs.erase(objs.begin() + index);

	// I have to rebuild the indices from scratch
	name2index.clear();
	index2obj.clear();

	for (u_int i = 0; i < objs.size(); ++i) {
		NamedObject *obj = objs[i];

		name2index.insert(Name2IndexType::value_type(obj->GetName(), i));
		index2obj.insert(Index2ObjType::value_type(i, obj));	
	}
}
if you have enough time, as you mention in last post you can do what you have in mind in it.
Post Reply