OpenVDB support

Discussion related to the LuxCore functionality, implementations and API.
User avatar
Dade
Developer
Developer
Posts: 5672
Joined: Mon Dec 04, 2017 8:36 pm
Location: Italy

Re: OpenVDB support

Post by Dade »

B.Y.O.B. wrote: Mon Apr 09, 2018 9:29 pm By the way can we accelerate the export of the normal point cache with C++?
Have to investigate. I think we would need a C++ function "DefineDensityGridData()" (like DefineMesh etc.) because the main bottleneck seems to be that we currently pass a HUGE array to a property.
Yes but isn't this problem totally solved by OpenVDB ?
Support LuxCoreRender project with salts and bounties
User avatar
B.Y.O.B.
Developer
Developer
Posts: 4146
Joined: Mon Dec 04, 2017 10:08 pm
Location: Germany
Contact:

Re: OpenVDB support

Post by B.Y.O.B. »

Yes, but OpenVDB is not the default and most users will still use the old point cache.
But yes, if it turns out to be too much work then I'll just add a label to the UI that says "use OpenVDB for fast smoke rendering".
User avatar
Dade
Developer
Developer
Posts: 5672
Joined: Mon Dec 04, 2017 8:36 pm
Location: Italy

Re: OpenVDB support

Post by Dade »

I merged all branches and now OpenVDB is part of the main LuxCoreRender branch.

I'm going to add the support for half storage in order to reduce the memory footprint.

Don't we need support for 3xFloat data too ? To express density grids with RGB values (not just a single scalar) :?:
Support LuxCoreRender project with salts and bounties
User avatar
B.Y.O.B.
Developer
Developer
Posts: 4146
Joined: Mon Dec 04, 2017 10:08 pm
Location: Germany
Contact:

Re: OpenVDB support

Post by B.Y.O.B. »

Dade wrote: Tue Apr 10, 2018 4:07 pm Don't we need support for 3xFloat data too ? To express density grids with RGB values (not just a single scalar)
Would be great!
I think Blender supports smoke sims with mixing different colors and has a color_grid, but currently we don't support it.

edit: had a look and the Blender color grid is 4 times the size of the density grid.
So I guess it is a one dimensional float array with values r, g, b, a, r, g, b, a, ... (note the alpha component).
Obviously it is not documented in the API. https://docs.blender.org/api/current/bp ... color_grid
Attachments
scrn_2018-04-10_18-21-11.png
User avatar
Dade
Developer
Developer
Posts: 5672
Joined: Mon Dec 04, 2017 8:36 pm
Location: Italy

Re: OpenVDB support

Post by Dade »

B.Y.O.B. wrote: Tue Apr 10, 2018 4:10 pm
Dade wrote: Tue Apr 10, 2018 4:07 pm Don't we need support for 3xFloat data too ? To express density grids with RGB values (not just a single scalar)
Would be great!
I think Blender supports smoke sims with mixing different colors and has a color_grid, but currently we don't support it.

edit: had a look and the Blender color grid is 4 times the size of the density grid.
So I guess it is a one dimensional float array with values r, g, b, a, r, g, b, a, ... (note the alpha component).
Obviously it is not documented in the API. https://docs.blender.org/api/current/bp ... color_grid
I will add the support for RGB values.
Support LuxCoreRender project with salts and bounties
User avatar
Dade
Developer
Developer
Posts: 5672
Joined: Mon Dec 04, 2017 8:36 pm
Location: Italy

Re: OpenVDB support

Post by Dade »

I have added the support for DensityGrid storage type with the ".storage" property. The supported storage are the same of image maps:

- byte
- half
- float
- auto

The default value (and auto) is half format. The "half" format cut in half the memory requirements and it is now the default. The "byte" format may not have enough resolution for some application but it is very effective to reduce the memory usage. This is the bunny rendered with byte storage:

bunny.jpg
It uses 1/4 of memory required by float storage.

I have also added the support for vector data (i.e. RGB). They can be defined with the ".data3" property instead of ".data". Indeed the number of expected values for ".data3" is nx * ny * nz * 3. OpenVDB supports automatically detect if the grid is scalar or vector.
Support LuxCoreRender project with salts and bounties
User avatar
B.Y.O.B.
Developer
Developer
Posts: 4146
Joined: Mon Dec 04, 2017 10:08 pm
Location: Germany
Contact:

Re: OpenVDB support

Post by B.Y.O.B. »

B.Y.O.B. wrote: Mon Apr 09, 2018 9:29 pm I have started to work on support in BlendLuxCore however apparently it is non-trivial to find out the path of the cache files...
I can't find the relevant Python attribute in Blender or in the documentation. Have to look further.
Found it!
object.modifiers["Smoke"].domain_settings.point_cache.filepath
Now I have to figure out where exactly to find the cache (because if the filepath is empty, I think a folder is created in the save location of the .blend file, and it can have a name but it does not have to, and I have to account for smoke sims linked in through a library etc. pp.)

To be honest it would be much more elegant to have a fast way to transfer Blender's grid data to LuxCore.
It would even save RAM if we could just specify a pointer to the start of the array in Blender's memory (no need to copy the data).
(But then again this only works for CPU renders...)

I'm sure the OpenVDB support will be used by someone else someday, but for Blender it is a bit complicated.
I should have investigated before you started to implement it... :oops:
User avatar
Dade
Developer
Developer
Posts: 5672
Joined: Mon Dec 04, 2017 8:36 pm
Location: Italy

Re: OpenVDB support

Post by Dade »

How do you access the Blender gird data ? Is it a Python list of scalar float ? How is it if they are RGB values ?
Support LuxCoreRender project with salts and bounties
User avatar
B.Y.O.B.
Developer
Developer
Posts: 4146
Joined: Mon Dec 04, 2017 10:08 pm
Location: Germany
Contact:

Re: OpenVDB support

Post by B.Y.O.B. »

From Python, we can access this class: https://docs.blender.org/api/current/bp ... inSettings
It contains the various grids: density_grid, color_grid, ... (but only in Python - in the C code behind the scenes they are somewhere else)
They are of type "bpy_prop_array", which is some kind of Blender-defined Python data structure: https://github.com/sobotka/blender/blob ... na.c#L6011

Another way would be to get a pointer to the SmokeDomainSettings struct.
But it seems complicated to retrieve the smoke grids from this struct (requires mutex locking, and the memory has to be copied anyway):
https://github.com/sobotka/blender/blob ... oke.c#L235

I will look into how to construct the correct path to the OpenVDB files, it should be doable.
User avatar
Dade
Developer
Developer
Posts: 5672
Joined: Mon Dec 04, 2017 8:36 pm
Location: Italy

Re: OpenVDB support

Post by Dade »

B.Y.O.B. wrote: Thu Apr 12, 2018 6:42 am From Python, we can access this class: https://docs.blender.org/api/current/bp ... inSettings
It contains the various grids: density_grid, color_grid, ... (but only in Python - in the C code behind the scenes they are somewhere else)
They are of type "bpy_prop_array", which is some kind of Blender-defined Python data structure: https://github.com/sobotka/blender/blob ... na.c#L6011
I have added a new set of methods to Property Python object: AddAllBool, AddAllInt, AddUnsignedLongLong and AddAllFloat. It is a fast path for adding large Python array (or list) to a property (like with density grid ".data" field). It should be as fast as having a dedicated method but it is a general solution we can use in other place too.

This code:

Code: Select all

	size = 2000000
	a = array('f', [i for i in range(size)])
	prop = pyluxcore.Property("test.array", [])
	
	# Test Add
	start = time.clock()
	for i in range(size):
		prop.Add([a[i]])
	end = time.clock()
	print("Add test: %.2gs" % (end-start))

	prop = pyluxcore.Property("test.array", [])
	
	# Test AddAll
	start = time.clock()
	prop.AddAllFloat(a)
	end = time.clock()
	print("AddAll test: %.2gs" % (end-start))
The result is:

Code: Select all

Add test: 3.2s
AddAll test: 0.042s
So it is 76 times faster. It should work "bpy_prop_array" object so you can add the complete grid with just a single "AddAllFloat()" call.
Support LuxCoreRender project with salts and bounties
Post Reply