Network rendering support

Discussion related to the Engine functionality, implementations and API.
User avatar
B.Y.O.B.
Developer
Posts: 1600
Joined: Mon Dec 04, 2017 10:08 pm
Location: Germany
Contact:

Re: Network rendering support

Post by B.Y.O.B. » Sun Jan 28, 2018 1:22 pm

Thanks for the summary. Then I'll look into wxPython.
Support LuxCoreRender project with salts and bounties

User avatar
Sharlybg
Supporting Users
Posts: 747
Joined: Mon Dec 04, 2017 10:11 pm
Location: Ivory Coast

Re: Network rendering support

Post by Sharlybg » Sun Jan 28, 2018 1:56 pm

Dade wrote:
Sun Jan 28, 2018 1:06 pm
B.Y.O.B. wrote:
Sun Jan 28, 2018 12:42 pm
Yes it looks a bit experimental to me, maybe not the best idea to use it.
To be honest I'm also not against writing the GUI using C++ ImGUI, I've already used it for a small tool a while ago.
The problem is the network rendering code is now written in Python so we would have C++ GUI calling Python network code (something still to explore) and than Python network code calling C++ pyluxcore, it looks like a recipe for troubles.

As far as I have seen at the moment:

- https://www.wxpython.org/ looks like a good option;
- I really like Qt but I have been able to understand the dual license mess of https://www.riverbankcomputing.com/soft ... /download5 and if we can use it;
- https://wiki.qt.io/PySide is another Qt, the license is ok but it looks like it is not actively developed anymore.
- Tk: I was using Tcl/Tk 30 years ago, the GUI looks prehistoric but Tk is included in Python, this is a major selling point.

Wx looks like the best option at the moment for developing all PyLuxCoreTools GUIs but I'm quite confused.
fall on this on the web : https://www.josephrex.me/getting-starte ... in-python/

and this the conclusion of the author :
Conclusion:
Building GUI programs may be a challenge but as the world evolves, it is very necessary. You should also not directly use generated code from pyuic because you may want to make changes to the UI at any point. Each compilation with pyuic will make you lose your changes you’ve made on the generated code. I’ve chosen Qt as my best module for building GUI. You may choose something else that works for you but from what I hear, Tkinter has a lot of downsides so it’s not recommended.
Support LuxCoreRender project with salts and bounties

Portfolio : https://www.behance.net/DRAVIA

User avatar
B.Y.O.B.
Developer
Posts: 1600
Joined: Mon Dec 04, 2017 10:08 pm
Location: Germany
Contact:

Re: Network rendering support

Post by B.Y.O.B. » Sun Jan 28, 2018 2:12 pm

Sure, PyQt looks good and I already know Qt from C++ programming, but if the licensing situation is unclear as Dade said we might not be able to use it.
PyQt is licensed under GPL. LuxCore API is using Apache license.
Maybe we just need to license the network manager GUI under GPL - is it a problem if GPL code calls an Apache library?

edit: I also found this, which seems to make the same point as I:
http://hselab.org/pyqt5-getting-started.html wrote:Since hillmaker is Apache 2.0 licensed, should be no problem to use PyQt but then the combined package might have to be GPL licensed (not that that's a deal breaker).
edit2: Something else: https://gist.github.com/takluyver/f7d95b0354a01bae2428

(Really annoying that you have to become a hobby lawyer if you are a developer)
Support LuxCoreRender project with salts and bounties

User avatar
Dade
Developer
Posts: 1340
Joined: Mon Dec 04, 2017 8:36 pm

Re: Network rendering support

Post by Dade » Sun Jan 28, 2018 4:59 pm

Some progress with network rendering:

- it is now possible to rendering nodes with OpenCL support and nodes without (i.e. mix PATHOCL and PATHCPU for instance).

- I have done the first rendering with a AWS EC2 node in the mix. I was using my home i7 3930K+i7 860 and a remote Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz (I think it has 18 cores/36 threads) with 60GB ram from AWS Frankfurt data center. The network rendering seems to works well even over internet: It was transferring a 1024x1024 film in less than 2 seconds.

The rendering speed of the AWS Xeon isn't very impressive (even if I like to compile LuxCore with a "make -j 36" :D ) however, the point here is that you can literally lunch any number of AWS EC2 instances you want/can pay.
Support LuxCoreRender project with salts and bounties

User avatar
Sharlybg
Supporting Users
Posts: 747
Joined: Mon Dec 04, 2017 10:11 pm
Location: Ivory Coast

Re: Network rendering support

Post by Sharlybg » Sun Jan 28, 2018 6:05 pm

- it is now possible to rendering nodes with OpenCL support and nodes without (i.e. mix PATHOCL and PATHCPU for instance).
__ Well done and great flexibility for a render engine in Alpha stage ;)
Support LuxCoreRender project with salts and bounties

Portfolio : https://www.behance.net/DRAVIA

jensverwiebe
Supporting Users
Posts: 141
Joined: Tue Jan 09, 2018 6:48 pm

Re: Network rendering support

Post by jensverwiebe » Sun Jan 28, 2018 6:11 pm

@Dade
Latest pyluxcorenet is broken:

Code: Select all

yaddayadda
Traceback (most recent call last):
  File "samples/pyluxcorenetconsole/pyluxcorenetconsole.py", line 38, in <module>
    cmd.main(sys.argv)
  File "./lib/pyluxcoretools.zip/pyluxcoretools/pyluxcorenetconsole/cmd.py", line 148, in main
    
  File "./lib/pyluxcoretools.zip/pyluxcoretools/pyluxcorenetconsole/cmd.py", line 95, in Exec
    self.cmdqueue = []
TypeError: 'NoneType' object is not iterable
yaddayadda

I added an default to fix that:

Code: Select all

		parser.add_argument("-n", "--nodes", metavar="IPADDRESS", nargs="+",
							default="",
							help="rendering nodes ip addresses")

Another oddity: since a while i cannot get to render ocl in pyluxcorenetnode.py anymore, it looks like all is initiated fine as well as kernels compile, but
the stats are always showing no contributions happen. This is only on localhost, another machine node renders in all cases.

Code: Select all

...
[MainThread][2018-01-28 19:05:19,524] [LuxCore][1468.316] [PathOCLBaseRenderThread::2] Kernels compilation time: 8ms
[MainThread][2018-01-28 19:05:19,551] Received command: GET_STATS
[MainThread][2018-01-28 19:05:19,576] [Elapsed time:   1sec][Samples    0][Avg. samples/sec  0.00M on 0.0K tris]
[MainThread][2018-01-28 19:05:29,577] Received command: GET_STATS
[MainThread][2018-01-28 19:05:29,631] [Elapsed time:  11sec][Samples    0][Avg. samples/sec  0.00M on 0.0K tris]
...
Same with an cpu scene works. Odd, any ideas ?
Btw: pyluxcoreconsole or luxcoreui blenderaddon all do ocl.

Jens

User avatar
Dade
Developer
Posts: 1340
Joined: Mon Dec 04, 2017 8:36 pm

Re: Network rendering support

Post by Dade » Sun Jan 28, 2018 7:59 pm

I should have fixed both problems.

The second was probably happening because I was setting opencl.cpu.workgroup.size/opencl.gpu.workgroup.size to 0 to use the default driver value but Mac OS must have some problem with that.
Support LuxCoreRender project with salts and bounties

jensverwiebe
Supporting Users
Posts: 141
Joined: Tue Jan 09, 2018 6:48 pm

Re: Network rendering support

Post by jensverwiebe » Sun Jan 28, 2018 8:01 pm

I check in a minute. Btw: i am mainly on linux, here the node refusing ocl happend with a localhost setup only.
More in a few minutes.

#1 aka non iterable arg is fixed in cmd.py

Jens
Last edited by jensverwiebe on Sun Jan 28, 2018 8:14 pm, edited 1 time in total.

jensverwiebe
Supporting Users
Posts: 141
Joined: Tue Jan 09, 2018 6:48 pm

Re: Network rendering support

Post by jensverwiebe » Sun Jan 28, 2018 8:08 pm

#2 is also working again. ( not sure i understand why your change applies for my situation on linux net + node locally )
[MainThread][2018-01-28 21:05:01,538] [LuxCore][19.120] [PathOCLBaseRenderThread::2] Kernels compilation time: 8ms
[MainThread][2018-01-28 21:05:01,565] Received command: GET_STATS
[MainThread][2018-01-28 21:05:01,591] [Elapsed time: 1sec][Samples 0][Avg. samples/sec 0.00M on 0.0K tris]
[MainThread][2018-01-28 21:05:11,591] Received command: GET_STATS
[MainThread][2018-01-28 21:05:11,615] [Elapsed time: 11sec][Samples 330][Avg. samples/sec 62.15M on 0.0K tris]
[MainThread][2018-01-28 21:05:21,615] Received command: GET_STATS
[MainThread][2018-01-28 21:05:21,660] [Elapsed time: 21sec][Samples 657][Avg. samples/sec 64.67M on 0.0K tris]
Going now to recompile osx too and check later cpu ocl ( i normally don't use it along with way faster gpu's )
That said, i had no probs with an osx node found atm. using only 2 gpu ocl.
Same on Linux: i disable normally the intel.icd to not use cpu for ocl additional to my 3 gpu, its ofter even slower else.

Atm. obvious probs fixed, thx Dade.

Jens
Last edited by jensverwiebe on Sun Jan 28, 2018 8:14 pm, edited 1 time in total.

User avatar
Dade
Developer
Posts: 1340
Joined: Mon Dec 04, 2017 8:36 pm

Re: Network rendering support

Post by Dade » Sun Jan 28, 2018 8:14 pm

You can use a command line option like "-D opencl.devices.select 010" to have a specific setting in every rendering node. You can use any number of custom properties and they will be used to override the settings coming from the network.
Support LuxCoreRender project with salts and bounties

Post Reply