memory cleanup

Discussion related to the LuxCore functionality, implementations and API.
Post Reply
bartek_zgo
Posts: 116
Joined: Mon Oct 26, 2020 11:42 am

memory cleanup

Post by bartek_zgo »

Hi guys. I want to ask how should I clean after render. When render ends I destroy session, renderConfig and scene but still a lot of memory is used. I can see that after calling each delete the memory usage is reduced but still a lot of memory is used. When I create scene I set: scene->SetDeleteMeshData(True) do remove points and vertex created by AllocVerticesBuffer and AllocTrianglesBuffer.

I add a lot of textures using properties:
scene.textures.83042_265_kd.file = "/mnt/nodeHDD1/cache/Texture/595/17922.jpg"

If I create Properties should I dispose them after calling scene->Parse(properties); ?

In my test with with a big scene
after the render the process use 12GB or RAM.

Code: Select all

delete session 
RAM USAGE 11.5GB

Code: Select all

delete renderConfig 
RAM USAGE 11.5GB

Code: Select all

delete scene
RAM USAGE 8.8GB
User avatar
Dade
Developer
Developer
Posts: 5672
Joined: Mon Dec 04, 2017 8:36 pm
Location: Italy

Re: memory cleanup

Post by Dade »

bartek_zgo wrote: Mon Nov 15, 2021 2:24 pm Hi guys. I want to ask how should I clean after render. When render ends I destroy session, renderConfig and scene but still a lot of memory is used. I can see that after calling each delete the memory usage is reduced but still a lot of memory is used. When I create scene I set: scene->SetDeleteMeshData(True) do remove points and vertex created by AllocVerticesBuffer and AllocTrianglesBuffer.
If the scene is delete or not when you delete the RenderConfig depends on how the scene has been created (i.e. if the owner is you or LuxCore). you can use RenderConfig::DeleteSceneOnExit() to force a specific behavior: https://github.com/LuxCoreRender/LuxCor ... re.h#L1263
bartek_zgo wrote: Mon Nov 15, 2021 2:24 pm I add a lot of textures using properties:
scene.textures.83042_265_kd.file = "/mnt/nodeHDD1/cache/Texture/595/17922.jpg"

If I create Properties should I dispose them after calling scene->Parse(properties); ?
Yes, you allocated the object, it is up to you to delete it.

Note also that all unused LuxCore binding of objects, materials, textures, etc. are deleted by LuxCore only if you call:

- Scene::RemoveUnusedImageMaps()
- Scene::RemoveUnusedTextures()
- Scene::RemoveUnusedMaterials()
- Scene::RemoveUnusedMeshes()

If some definition is not used, it is still kept (because you may re-use it in the future). It is removed only if call Scene::Remove*().
bartek_zgo wrote: Mon Nov 15, 2021 2:24 pm In my test with with a big scene
after the render the process use 12GB or RAM.

Code: Select all

delete session 
RAM USAGE 11.5GB

Code: Select all

delete renderConfig 
RAM USAGE 11.5GB

Code: Select all

delete scene
RAM USAGE 8.8GB
You should end with the same amount of free memory you had at the start.
Support LuxCoreRender project with salts and bounties
bartek_zgo
Posts: 116
Joined: Mon Oct 26, 2020 11:42 am

Re: memory cleanup

Post by bartek_zgo »

Please tell me if I'm correct. I create scene from 0 and start render. So all objects, materials, textures etc. are used. So after render calling Scene::RemoveUnused*() is useless. All resources are attached to scene so they will not remove anything.
I can set DeleteSceneOnExit() but this is identical to call delete scene;

So there is no way do dispose scene with all resources automatically, right?

So as I understand each time I create properties with new Properties() I have to preserve pointer and dispose them. But when I have to dispose them? Can I do this after calling Scene::Parse(Properties) or I have to do this after render, after disposing Scene?

Another question. Let say that I call properies = new Properties(). Properties are composed by Property that are composed by PropertyValue. If I call delete properties; does Property and ProperyValue dispose, or I have to dispose each one separately?
User avatar
Dade
Developer
Developer
Posts: 5672
Joined: Mon Dec 04, 2017 8:36 pm
Location: Italy

Re: memory cleanup

Post by Dade »

bartek_zgo wrote: Tue Nov 16, 2021 8:38 am Please tell me if I'm correct. I create scene from 0 and start render. So all objects, materials, textures etc. are used. So after render calling Scene::RemoveUnused*() is useless. All resources are attached to scene so they will not remove anything.
Yes, correct.
bartek_zgo wrote: Tue Nov 16, 2021 8:38 am I can set DeleteSceneOnExit() but this is identical to call delete scene;
Yes, correct.
bartek_zgo wrote: Tue Nov 16, 2021 8:38 am So there is no way do dispose scene with all resources automatically, right?
Uh, no, why ? If you delete the RenderSession, the RenderConfig and either set DeleteSceneOnExit() or delete the Scene, you have deleted everything.
bartek_zgo wrote: Tue Nov 16, 2021 8:38 am So as I understand each time I create properties with new Properties() I have to preserve pointer and dispose them. But when I have to dispose them? Can I do this after calling Scene::Parse(Properties) or I have to do this after render, after disposing Scene?
Your Properties is used only for the time of the execution of Scene::Parse(Properties).
bartek_zgo wrote: Tue Nov 16, 2021 8:38 am Another question. Let say that I call properies = new Properties(). Properties are composed by Property that are composed by PropertyValue. If I call delete properties; does Property and ProperyValue dispose, or I have to dispose each one separately?
No, everything is disposed as soon as you delete "properies".

But are you working in C# or C++ ? In C++, you don't usually write "properies = new Properties();" but "Properties properies;" so the object is automatically disposed by the compiler, in any case, as soon as it is out of scope. This is also true for "Property", etc. You don't have to manage explicit memory allocation at all in this case.
Support LuxCoreRender project with salts and bounties
bartek_zgo
Posts: 116
Joined: Mon Oct 26, 2020 11:42 am

Re: memory cleanup

Post by bartek_zgo »

I'm working in C# and I have a wrapper written in C++. So we call new Properties() to pass the pointer to managed code. I will try to call delete properties after scene->Parse

But something is strange. I call delete session; delete renderConfig; delete scene;
So theoretically all resources has been disposed except Properties() created with new Properies(). It is hard to believe that this can use 8.8GB
User avatar
Dade
Developer
Developer
Posts: 5672
Joined: Mon Dec 04, 2017 8:36 pm
Location: Italy

Re: memory cleanup

Post by Dade »

Doesn't C# have a garbage collector like Java ?

Automatic garbage collectors are a nasty thing: you free an object but the actual real memory is freed by the garbage collector some time after (or may be is re-used for a new object of the same type, etc.). You have very little control on the amount of memory they use (this is true for Java, I don't know if C# is the same).

As test, you should run the rendering (create the scene, render, delete everything) in a loop: if you run out of memory at some point, there is a problem otherwise is just the garbage collector doing its own things.
Support LuxCoreRender project with salts and bounties
bartek_zgo
Posts: 116
Joined: Mon Oct 26, 2020 11:42 am

Re: memory cleanup

Post by bartek_zgo »

This is not a problem of garbage collector. In C# I can force GC to collect (what I'm doing after render). What more I can see amount of memory used by managed and unmanaged code.
bartek_zgo
Posts: 116
Joined: Mon Oct 26, 2020 11:42 am

Re: memory cleanup

Post by bartek_zgo »

Hi Guys, it seems that my memory leak is caused by instances. I am creating 1 million of instances. All instances have the same material but different location. I created a huge array that is composed by concatenating the transformation matrix of each instance. After rendering I call delete scene, but my application still uses 3GB of memory. If I comment out the pScene.Parse(props) line, I don't find any memory leak. Any idea?

Code: Select all

std::string instancePrefix_s = "myInstanceId_";
std::string meshName_s = "myShape";
std::string materialName_s = "myMaterial";
std::string propertyPrefix;
for (int n = 0; n < instanceCount; n++)
{
	propertyPrefix = "scene.objects." + instancePrefix_s + std::to_string(n);
	Properties props = Properties();
	props.Set(Property(propertyPrefix + ".id")(colorId));
	props.Set(Property(propertyPrefix + ".shape")(meshName_s));
	props.Set(Property(propertyPrefix + ".material")(materialName_s));

	Property transformProp = Property(propertyPrefix + ".transformation");
	for (size_t i = 0; i < 16; i++)
		transformProp.Add(transform[n * 16 + i]);
	props.Set(transformProp);

	if (hideToCamera)
		props.Set(Property(propertyPrefix + ".camerainvisible")(hideToCamera));

	pScene->Parse(props);
}
Post Reply