LuxCore Network Render

[Jack Eden]

Thanks again. Numpy is now installed. Although one step closer I am still not having success.

I just downloaded BlendLuxCore 2.4 and LuxCore Standalone 2.4 (official releases) and when I try to use Network Rendering I am getting the error:

[Errorno 61] Connection refused

My first though was that it could be due to "Port 18018" being closed, but as LuxRender v1.6 uses the same port, and I am able to run networker rendering with that version then I think I can rule out that possibility. So then I thought I might still have a version mismatch, but after double checking I confirmed that both master and slave are running 2.4 official releases.

NetConsole is running on MacOS 10.14.6 (launched from Blender v2.3, and the NetConsole window shows LuxCore 2.4)
NetNode is running on Ubuntu Linux 20.04 (run from Terminal, and shows LuxCore 2.4)

Any guesses as to what might be causing the error? Thanks again for the support.

[Dade]

I have the feeling you are using the wrong tool for your needs: with a single node, network rendering is a bit redundant. You can just transfer the .bcf file by hand and than do an headless rendering by just:

Code: Select all

python3 pyluxcoretools.zip console xyz.bcf

[Jack Eden]

I have the feeling you are using the wrong tool for your needs: with a single node, network rendering is a bit redundant. You can just transfer the .bcf file by hand and than do an headless rendering by just:

That's an interesting idea, and thank you for the suggestion I will give it a try. However, I have access to a 12 node render farm and was hoping to help test the large scale distributed processing of large render jobs, so I was testing on one node before installing on the render farm. For now I suppose I could just use scp to move files one at a time between the nodes but as you said it is a bit more tedious.

[Jack Eden]

Code: Select all

python3 pyluxcoretools.zip console xyz.bcf

I am actually quite encouraged by the possibilities of using the console as you have suggested. Is it possible to pass multiple files to to console with one argument? Something like:

Code: Select all

python3 pyluxcoretools.zip console xyz1.bcf xyz2.bcf xyz3.bcf

I have tried multiple possibilities and I have not found anything that works yet and I get an error of unrecognized arguments.

[Jack Eden]

After thinking about this a little bit more I realized that I could automate the process of sending of multiple render files to the Console with a shell script:

Code: Select all

for file in *.bcf; do python3 pyluxcoretools.zip console Animation/"${file}"; done

But then I have ran into the problem that every image is named

RGB_IMAGEPIPELINE_0.png

and so as it goes through and batch processes the frames one by one each previous image is overwritten by the next and in the end you are just left with one image named RGB_IMAGEPIPELINE_0.png.

[Dade]

But then I have ran into the problem that every image is named

RGB_IMAGEPIPELINE_0.png

and so as it goes through and batch processes the frames one by one each previous image is overwritten by the next and in the end you are just left with one image named RGB_IMAGEPIPELINE_0.png.

Just use:

Code: Select all

for file in *.bcf; do python3 pyluxcoretools.zip console -D film.outputs.0.type RGB_IMAGEPIPELINE -D film.outputs.0.index 0 -D film.outputs.0.filename "${file%.*}.png" Animation/"${file}"; done

It will overwrite the first output defined in .bcf file:

Code: Select all

film.outputs.0.type = RGB_IMAGEPIPELINE
film.outputs.0.index = 0
film.outputs.0.filename = xyz1.png

[Dade]

However, I have access to a 12 node render farm and was hoping to help test the large scale distributed processing of large render jobs, so I was testing on one node before installing on the render farm.

The reason I consider network rendering something of the past is because it is really required only in one single case: when the rendering time of single image is so long to requires multiple PCs.

This was something common in the past with old Lux (when a single rendering could easily take more than an entire night).

Thanks to the progress with software (LuxCoreRender) and hardware (GPUs and multi-core CPUs), the average rendering time of a single image is now more in the 15-30 minutes range.

If you need to render multiple images or animation frames, you need a different tool (see below).

For now I suppose I could just use scp to move files one at a time between the nodes but as you said it is a bit more tedious.

There are many commercial and opensource/free solutions available for rendering task queue management. You can submit any number of rendering jobs, they include multiple users support, priorities management, multiple queues management, user rights management, etc.
A fast Google search give me solutions like:

https://www.opencue.io
https://github.com/DrQueue/drqueue

They can be quite complex to setup and you may have also to write custom shell scripts to add LuxCore support but they are very powerful and handy once the setup is done.

They are a solution with a very high initial "cost" (i.e. the setup) but they will pay back if you use them a lot.

On the opposite side, you can do all by hands with no initial setup "cost" but a little work to do each time.

I suggest you to setup, at least, a shared file system between all rendering nodes and Mac client(s) so you can save there all .bcf files and read back the images without having to use scp all the times.

[Jack Eden]

Dad, thank you again for your detailed and kind response.

I understand now that I need to change my paradigm of how LuxCoreRender works for my particular workflow. I was trying to take the workflow I had been using for LuxRender and simply transpose it onto LuxCoreRender, and now I see that I shouldn't be attempting to do this. I also now realize that my particular workflow may be fairly unique to my particular use case.

I will definitely take a look at the render task queue management options.