Unified executable (for CPU-only, OpenCL and CUDA)

Discussion related to the LuxCore functionality, implementations and API.
User avatar
Dade
Developer
Posts: 4240
Joined: Mon Dec 04, 2017 8:36 pm
Location: Italy

Unified executable (for CPU-only, OpenCL and CUDA)

Post by Dade » Thu May 21, 2020 3:27 pm

Introduction

The aim of this project is to have an unified executable capable to work on any platform supporting: CPU-only, OpenCL and CUDA. This task was triggered by the problem that you can not directly use CUDA in a Blender add-on on Linux (i.e. calling cuInit() crashes Blender) but, at the end it offers many advantages.

Advantages

There has many advantages:

- CUDA works with Blender and Linux;
- a single executable instead of 3 different one;
- you can compile the LuxCore sources without having to use OpenCL or CUDA SDK;
- it may be possible to extend the same approach to Optix in the future (avoiding to have 4 different executable !);

Testing

I have done the very first test of runtime loading of CUDA on the "cuda_ocl_wranglers" branch, @Acasta69 and @u3dreal can you check if CUDA still works on Windows and MacOS.
If does I can go on, I have to change a not trivial amount of stuff because having/not having CUDA and OpenCL will become a run-time condition, not more compilation time #ifdef.
My first test has solved the problem of running CUDA on Linux with Blender (i.e. no more crash).
Support LuxCoreRender project with salts and bounties

acasta69
Developer
Posts: 310
Joined: Tue Jan 09, 2018 3:45 pm

Re: Unified executable (for CPU-only, OpenCL and CUDA)

Post by acasta69 » Thu May 21, 2020 5:15 pm

Dade wrote:
Thu May 21, 2020 3:27 pm
- you can compile the LuxCore sources without having to use OpenCL or CUDA SDK;
Do you mean this should work even if compiled with OpenCL "disabled"?
There is a problem, though:
https://dev.azure.com/LuxCoreRender/Lux ... ew=results
These errors happen with no-ocl compilation after your fix for Apple texture mapping.
I don't really understand: shouldn't "cl" files be ignored in that case?

Compiling with Cuda "enabled" is working as usual.
Support LuxCoreRender project with salts and bounties

Windows 10 64 bits, i7-4770 3.4 GHz, RAM 16 GB, GTX 970 4GB v445.87

User avatar
Dade
Developer
Posts: 4240
Joined: Mon Dec 04, 2017 8:36 pm
Location: Italy

Re: Unified executable (for CPU-only, OpenCL and CUDA)

Post by Dade » Thu May 21, 2020 5:52 pm

acasta69 wrote:
Thu May 21, 2020 5:15 pm
Dade wrote:
Thu May 21, 2020 3:27 pm
- you can compile the LuxCore sources without having to use OpenCL or CUDA SDK;
Do you mean this should work even if compiled with OpenCL "disabled"?
Not for the moment, for the first test, use OpenCL and CUDA enabled.
acasta69 wrote:
Thu May 21, 2020 5:15 pm
There is a problem, though:
https://dev.azure.com/LuxCoreRender/Lux ... ew=results
These errors happen with no-ocl compilation after your fix for Apple texture mapping.
I don't really understand: shouldn't "cl" files be ignored in that case?

Compiling with Cuda "enabled" is working as usual.
I will check.
Support LuxCoreRender project with salts and bounties

User avatar
u3dreal
Developer
Posts: 389
Joined: Tue Dec 03, 2019 3:23 pm
Location: Ulm
Contact:

Re: Unified executable (for CPU-only, OpenCL and CUDA)

Post by u3dreal » Thu May 21, 2020 6:00 pm

All works fine here .. There was even a driver update a week ago. :)
check out my newest stuff http://q3de.com/research/
portfolio http://q3de.com/


MB Pro i7 2.3Ghz, IrisPro 1.5GB, GTX750m 2GB - High Sierra
Xeon X5650@4Ghz, 2x GTX 770 Phantom - High Sierra, Windows 10, Ubuntu 18.04

acasta69
Developer
Posts: 310
Joined: Tue Jan 09, 2018 3:45 pm

Re: Unified executable (for CPU-only, OpenCL and CUDA)

Post by acasta69 » Thu May 21, 2020 6:53 pm

Dade wrote:
Thu May 21, 2020 5:52 pm
Not for the moment, for the first test, use OpenCL and CUDA enabled.
Everything's ok with this.
Support LuxCoreRender project with salts and bounties

Windows 10 64 bits, i7-4770 3.4 GHz, RAM 16 GB, GTX 970 4GB v445.87

User avatar
Dade
Developer
Posts: 4240
Joined: Mon Dec 04, 2017 8:36 pm
Location: Italy

Re: Unified executable (for CPU-only, OpenCL and CUDA)

Post by Dade » Thu May 21, 2020 8:30 pm

Ok, I will continue and extend the dynamic loading of libraries to OpenCL.
Support LuxCoreRender project with salts and bounties

User avatar
Dade
Developer
Posts: 4240
Joined: Mon Dec 04, 2017 8:36 pm
Location: Italy

Re: Unified executable (for CPU-only, OpenCL and CUDA)

Post by Dade » Wed May 27, 2020 11:14 am

Efff, things are never simply. I had to port all OpenCL host code from C++ bindings to plain C in order to get the OpenCL Wrangler to work. A lot of work and probably new bugs.

So now:

- CUDA works with Blender and Linux;
- a single executable instead of 3 different one;
- you can compile the LuxCore sources without having to use OpenCL or CUDA SDK or OpenCL C++ bindings;

The work should be now complete, @Acasta69 and @u3dreal can you check it works on Windows and MacOS (stand alone and inside Blender) ?
Support LuxCoreRender project with salts and bounties

acasta69
Developer
Posts: 310
Joined: Tue Jan 09, 2018 3:45 pm

Re: Unified executable (for CPU-only, OpenCL and CUDA)

Post by acasta69 » Wed May 27, 2020 1:56 pm

Dade wrote:
Wed May 27, 2020 11:14 am
The work should be now complete, @Acasta69 and @u3dreal can you check it works on Windows and MacOS (stand alone and inside Blender) ?
Quick test on Windows (LuxcoreUI only): CUDA ok, OpenCL crashes while compiling kernels:

Capture.png

I cannot investigate further at the moment, will try later.

Dade wrote:
Wed May 27, 2020 11:14 am
- you can compile the LuxCore sources without having to use OpenCL or CUDA SDK or OpenCL C++ bindings;
Indeed compilation works with OpenCL disabled flag.
However, we still need to distribute the NVRTC libraries, which are not part of the Nvidia drivers.
Those who compile their own executables will need the CUDA SDK. Or we could include the libs in the binary dependencies.
Support LuxCoreRender project with salts and bounties

Windows 10 64 bits, i7-4770 3.4 GHz, RAM 16 GB, GTX 970 4GB v445.87

User avatar
Dade
Developer
Posts: 4240
Joined: Mon Dec 04, 2017 8:36 pm
Location: Italy

Re: Unified executable (for CPU-only, OpenCL and CUDA)

Post by Dade » Wed May 27, 2020 4:19 pm

acasta69 wrote:
Wed May 27, 2020 1:56 pm
Those who compile their own executables will need the CUDA SDK. Or we could include the libs in the binary dependencies.
You don't need the SDK to compile the executables but you need the NVIDIA drivers + NVRTC DLL to run PATHOCL with CUDA.
Support LuxCoreRender project with salts and bounties

acasta69
Developer
Posts: 310
Joined: Tue Jan 09, 2018 3:45 pm

Re: Unified executable (for CPU-only, OpenCL and CUDA)

Post by acasta69 » Wed May 27, 2020 8:13 pm

acasta69 wrote:
Wed May 27, 2020 1:56 pm
Dade wrote:
Wed May 27, 2020 11:14 am
The work should be now complete, @Acasta69 and @u3dreal can you check it works on Windows and MacOS (stand alone and inside Blender) ?
Quick test on Windows (LuxcoreUI only): CUDA ok, OpenCL crashes while compiling kernels
Looking at the new sources, I thought it was wrong to have LUXRAYS_DISABLE_OPENCL defined, but even without it, it crashes.
A stack overflow happens in line 310, ocl.cpp:

Code: Select all

Exception thrown at 0x00007FF6FD6AA6A8 in luxcoreui.exe: 0xC00000FD: Stack overflow (parameters: 0x0000000000000001, 0x000000E2BC623000).
Unhandled exception at 0x00007FF6FD6AA6A8 in luxcoreui.exe: 0xC00000FD: Stack overflow (parameters: 0x0000000000000001, 0x000000E2BC623000).
Call stack:

Code: Select all

 	luxcoreui.exe!__chkstk() Line 109	Unknown
>	luxcoreui.exe!luxrays::oclKernelPersistentCache::Compile(_cl_context * context, _cl_device_id * device, const std::vector<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,std::allocator<std::basic_string<char,std::char_traits<char>,std::allocator<char> > > > & kernelsParameters, const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & kernelSource, bool * cached, std::basic_string<char,std::char_traits<char>,std::allocator<char> > * errorStr) Line 310	C++
 	luxcoreui.exe!luxrays::OpenCLDevice::CompileProgram(luxrays::HardwareDeviceProgram * * program, const std::vector<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,std::allocator<std::basic_string<char,std::char_traits<char>,std::allocator<char> > > > & programParameters, const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & programSource, const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & programName) Line 179	C++
 	luxcoreui.exe!slg::PathOCLBaseOCLRenderThread::InitKernels() Line 295	C++
 	luxcoreui.exe!slg::PathOCLBaseOCLRenderThread::InitRender() Line 638	C++
 	luxcoreui.exe!slg::PathOCLBaseOCLRenderThread::Start() Line 164	C++
 	luxcoreui.exe!slg::PathOCLBaseRenderEngine::StartLockLess() Line 343	C++
 	luxcoreui.exe!slg::PathOCLRenderEngine::StartLockLess() Line 155	C++
 	luxcoreui.exe!slg::RenderEngine::Start(slg::Film * flm, boost::mutex * flmMutex) Line 137	C++
 	luxcoreui.exe!slg::RenderSession::Start() Line 83	C++
 	luxcoreui.exe!luxcore::detail::RenderSessionImpl::Start() Line 1008	C++
 	luxcoreui.exe!LuxCoreApp::StartRendering(luxcore::RenderState * startState, luxcore::Film * startFilm) Line 364	C++
 	luxcoreui.exe!LuxCoreApp::RenderConfigParse(const luxrays::Properties & props) Line 209	C++
 	luxcoreui.exe!LuxCoreApp::SetRenderingEngineType(const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & engineType) Line 187	C++
 	luxcoreui.exe!LuxCoreApp::MenuEngine() Line 195	C++
 	luxcoreui.exe!LuxCoreApp::MainMenuBar() Line 546	C++
 	luxcoreui.exe!LuxCoreApp::RunApp(luxcore::RenderState * startState, luxcore::Film * startFilm) Line 610	C++
 	luxcoreui.exe!main(int argc, char * * argv) Line 222	C++
I tried to understand why that happens, but it's beyond my knowledge...
Support LuxCoreRender project with salts and bounties

Windows 10 64 bits, i7-4770 3.4 GHz, RAM 16 GB, GTX 970 4GB v445.87

Post Reply