OpenVDB support

Discussion related to the Engine functionality, implementations and API.
User avatar
Dade
Developer
Posts: 1187
Joined: Mon Dec 04, 2017 8:36 pm

Re: OpenVDB support

Post by Dade » Mon Apr 09, 2018 9:42 pm

B.Y.O.B. wrote:
Mon Apr 09, 2018 9:29 pm
By the way can we accelerate the export of the normal point cache with C++?
Have to investigate. I think we would need a C++ function "DefineDensityGridData()" (like DefineMesh etc.) because the main bottleneck seems to be that we currently pass a HUGE array to a property.
Yes but isn't this problem totally solved by OpenVDB ?
Support LuxCoreRender project with salts and bounties

User avatar
B.Y.O.B.
Developer
Posts: 1316
Joined: Mon Dec 04, 2017 10:08 pm
Location: Germany
Contact:

Re: OpenVDB support

Post by B.Y.O.B. » Mon Apr 09, 2018 9:47 pm

Yes, but OpenVDB is not the default and most users will still use the old point cache.
But yes, if it turns out to be too much work then I'll just add a label to the UI that says "use OpenVDB for fast smoke rendering".
Support LuxCoreRender project with salts and bounties

User avatar
Dade
Developer
Posts: 1187
Joined: Mon Dec 04, 2017 8:36 pm

Re: OpenVDB support

Post by Dade » Tue Apr 10, 2018 4:07 pm

I merged all branches and now OpenVDB is part of the main LuxCoreRender branch.

I'm going to add the support for half storage in order to reduce the memory footprint.

Don't we need support for 3xFloat data too ? To express density grids with RGB values (not just a single scalar) :?:
Support LuxCoreRender project with salts and bounties

User avatar
B.Y.O.B.
Developer
Posts: 1316
Joined: Mon Dec 04, 2017 10:08 pm
Location: Germany
Contact:

Re: OpenVDB support

Post by B.Y.O.B. » Tue Apr 10, 2018 4:10 pm

Dade wrote:
Tue Apr 10, 2018 4:07 pm
Don't we need support for 3xFloat data too ? To express density grids with RGB values (not just a single scalar)
Would be great!
I think Blender supports smoke sims with mixing different colors and has a color_grid, but currently we don't support it.

edit: had a look and the Blender color grid is 4 times the size of the density grid.
So I guess it is a one dimensional float array with values r, g, b, a, r, g, b, a, ... (note the alpha component).
Obviously it is not documented in the API. https://docs.blender.org/api/current/bp ... color_grid
Attachments
scrn_2018-04-10_18-21-11.png
Support LuxCoreRender project with salts and bounties

User avatar
Dade
Developer
Posts: 1187
Joined: Mon Dec 04, 2017 8:36 pm

Re: OpenVDB support

Post by Dade » Tue Apr 10, 2018 6:56 pm

B.Y.O.B. wrote:
Tue Apr 10, 2018 4:10 pm
Dade wrote:
Tue Apr 10, 2018 4:07 pm
Don't we need support for 3xFloat data too ? To express density grids with RGB values (not just a single scalar)
Would be great!
I think Blender supports smoke sims with mixing different colors and has a color_grid, but currently we don't support it.

edit: had a look and the Blender color grid is 4 times the size of the density grid.
So I guess it is a one dimensional float array with values r, g, b, a, r, g, b, a, ... (note the alpha component).
Obviously it is not documented in the API. https://docs.blender.org/api/current/bp ... color_grid
I will add the support for RGB values.
Support LuxCoreRender project with salts and bounties

User avatar
Dade
Developer
Posts: 1187
Joined: Mon Dec 04, 2017 8:36 pm

Re: OpenVDB support

Post by Dade » Wed Apr 11, 2018 1:20 pm

I have added the support for DensityGrid storage type with the ".storage" property. The supported storage are the same of image maps:

- byte
- half
- float
- auto

The default value (and auto) is half format. The "half" format cut in half the memory requirements and it is now the default. The "byte" format may not have enough resolution for some application but it is very effective to reduce the memory usage. This is the bunny rendered with byte storage:

bunny.jpg
It uses 1/4 of memory required by float storage.

I have also added the support for vector data (i.e. RGB). They can be defined with the ".data3" property instead of ".data". Indeed the number of expected values for ".data3" is nx * ny * nz * 3. OpenVDB supports automatically detect if the grid is scalar or vector.
Support LuxCoreRender project with salts and bounties

User avatar
B.Y.O.B.
Developer
Posts: 1316
Joined: Mon Dec 04, 2017 10:08 pm
Location: Germany
Contact:

Re: OpenVDB support

Post by B.Y.O.B. » Wed Apr 11, 2018 6:37 pm

B.Y.O.B. wrote:
Mon Apr 09, 2018 9:29 pm
I have started to work on support in BlendLuxCore however apparently it is non-trivial to find out the path of the cache files...
I can't find the relevant Python attribute in Blender or in the documentation. Have to look further.
Found it!
object.modifiers["Smoke"].domain_settings.point_cache.filepath
Now I have to figure out where exactly to find the cache (because if the filepath is empty, I think a folder is created in the save location of the .blend file, and it can have a name but it does not have to, and I have to account for smoke sims linked in through a library etc. pp.)

To be honest it would be much more elegant to have a fast way to transfer Blender's grid data to LuxCore.
It would even save RAM if we could just specify a pointer to the start of the array in Blender's memory (no need to copy the data).
(But then again this only works for CPU renders...)

I'm sure the OpenVDB support will be used by someone else someday, but for Blender it is a bit complicated.
I should have investigated before you started to implement it... :oops:
Support LuxCoreRender project with salts and bounties

User avatar
Dade
Developer
Posts: 1187
Joined: Mon Dec 04, 2017 8:36 pm

Re: OpenVDB support

Post by Dade » Wed Apr 11, 2018 7:07 pm

How do you access the Blender gird data ? Is it a Python list of scalar float ? How is it if they are RGB values ?
Support LuxCoreRender project with salts and bounties

User avatar
B.Y.O.B.
Developer
Posts: 1316
Joined: Mon Dec 04, 2017 10:08 pm
Location: Germany
Contact:

Re: OpenVDB support

Post by B.Y.O.B. » Thu Apr 12, 2018 6:42 am

From Python, we can access this class: https://docs.blender.org/api/current/bp ... inSettings
It contains the various grids: density_grid, color_grid, ... (but only in Python - in the C code behind the scenes they are somewhere else)
They are of type "bpy_prop_array", which is some kind of Blender-defined Python data structure: https://github.com/sobotka/blender/blob ... na.c#L6011

Another way would be to get a pointer to the SmokeDomainSettings struct.
But it seems complicated to retrieve the smoke grids from this struct (requires mutex locking, and the memory has to be copied anyway):
https://github.com/sobotka/blender/blob ... oke.c#L235

I will look into how to construct the correct path to the OpenVDB files, it should be doable.
Support LuxCoreRender project with salts and bounties

User avatar
Dade
Developer
Posts: 1187
Joined: Mon Dec 04, 2017 8:36 pm

Re: OpenVDB support

Post by Dade » Thu Apr 12, 2018 11:16 am

B.Y.O.B. wrote:
Thu Apr 12, 2018 6:42 am
From Python, we can access this class: https://docs.blender.org/api/current/bp ... inSettings
It contains the various grids: density_grid, color_grid, ... (but only in Python - in the C code behind the scenes they are somewhere else)
They are of type "bpy_prop_array", which is some kind of Blender-defined Python data structure: https://github.com/sobotka/blender/blob ... na.c#L6011
I have added a new set of methods to Property Python object: AddAllBool, AddAllInt, AddUnsignedLongLong and AddAllFloat. It is a fast path for adding large Python array (or list) to a property (like with density grid ".data" field). It should be as fast as having a dedicated method but it is a general solution we can use in other place too.

This code:

Code: Select all

	size = 2000000
	a = array('f', [i for i in range(size)])
	prop = pyluxcore.Property("test.array", [])
	
	# Test Add
	start = time.clock()
	for i in range(size):
		prop.Add([a[i]])
	end = time.clock()
	print("Add test: %.2gs" % (end-start))

	prop = pyluxcore.Property("test.array", [])
	
	# Test AddAll
	start = time.clock()
	prop.AddAllFloat(a)
	end = time.clock()
	print("AddAll test: %.2gs" % (end-start))
The result is:

Code: Select all

Add test: 3.2s
AddAll test: 0.042s
So it is 76 times faster. It should work "bpy_prop_array" object so you can add the complete grid with just a single "AddAllFloat()" call.
Support LuxCoreRender project with salts and bounties

Post Reply