Opencl clenqueuwritebuffer bug

Use this forum for general user support and related questions.
Forum rules
Please upload a testscene that allows developers to reproduce the problem, and attach some images.
User avatar
Sharlybg
Donor
Donor
Posts: 3101
Joined: Mon Dec 04, 2017 10:11 pm
Location: Ivory Coast

Opencl clenqueuwritebuffer bug

Post by Sharlybg »

just a couple of time now that i get a bug while animating with 2.4 Cuda build. Each 48 frames i get an error "clenqueuwritebuffer". And render stop.
look like the gpu go out of memory. but this is no matter the scene size. as it happen now with a small size scene.

will try to reproduce it with a simpler scene for debugging.
Capture d'écran 2020-05-08 12.25.05.png
Support LuxCoreRender project with salts and bounties

Portfolio : https://www.behance.net/DRAVIA
User avatar
Dade
Developer
Developer
Posts: 5672
Joined: Mon Dec 04, 2017 8:36 pm
Location: Italy

Re: Opencl clenqueuwritebuffer bug

Post by Dade »

It is likely to be some kind of memory leak with CUDA (it would be useful to understand if it happens only with CUDA or also with OpenCL).
Support LuxCoreRender project with salts and bounties
User avatar
Theo_Gottwald
Posts: 109
Joined: Fri Apr 24, 2020 12:01 pm

Re: Opencl clenqueuwritebuffer bug

Post by Theo_Gottwald »

I also get this error, but when this happens, the computer needs a reboot.
After a reboot here it works - if the GPU Memory is large enough.
Here its like that, that even a restart of Blender will not fix the problem.
Only a restart of the computer will fix the problem.
Get Blender-Quickbuttons and configure over 100 Blender-Buttons individual to your needs.
Visit my YouTube-Channel: Theo's Fun Videos and watch several Blender related Videos.
Join @Dreamstime and sell your Renderings to the world.
User avatar
Dade
Developer
Developer
Posts: 5672
Joined: Mon Dec 04, 2017 8:36 pm
Location: Italy

Re: Opencl clenqueuwritebuffer bug

Post by Dade »

Theo_Gottwald wrote: Wed May 13, 2020 9:58 pm I also get this error, but when this happens, the computer needs a reboot.
After a reboot here it works - if the GPU Memory is large enough.
Here its like that, that even a restart of Blender will not fix the problem.
Only a restart of the computer will fix the problem.
This really sounds more like a driver bug than an application bug.
Support LuxCoreRender project with salts and bounties
User avatar
Theo_Gottwald
Posts: 109
Joined: Fri Apr 24, 2020 12:01 pm

Re: Opencl clenqueuwritebuffer bug

Post by Theo_Gottwald »

I can agree to that, because also Cycles sometimes just quits rendering.
I have 2x GTX 1080 currently. But seeing the rendertime I am working on something with RTX :-)
Just nor sure if i should take it for that price :-)
https://www.alibaba.com/product-detail/ ... 723bQTMzZ8
To night i had an animation rendering. Just at frame 37 it stopped rendering with this error.
If you want to try, download it here:


This one:
42 MB Scene.
This one is also for download.
42 MB Scene.
Get Blender-Quickbuttons and configure over 100 Blender-Buttons individual to your needs.
Visit my YouTube-Channel: Theo's Fun Videos and watch several Blender related Videos.
Join @Dreamstime and sell your Renderings to the world.
User avatar
Sharlybg
Donor
Donor
Posts: 3101
Joined: Mon Dec 04, 2017 10:11 pm
Location: Ivory Coast

Re: Opencl clenqueuwritebuffer bug

Post by Sharlybg »

I still have this error plus blender crash from time to time each 48 frames render.

Btw 150§ for a RTX 2080 TI is pretty strange. would like to see how this end up :roll:
Support LuxCoreRender project with salts and bounties

Portfolio : https://www.behance.net/DRAVIA
Martini
Posts: 125
Joined: Fri Nov 23, 2018 11:36 am
Location: Australia

Re: Opencl clenqueuwritebuffer bug

Post by Martini »

I don't know if it's the same bug, but I also experienced an out-of-memory error rendering an animation on my own scene. (Apologies, I did not capture the exact error message)

I tried rendering again, and Blender crashed. It seems the memory became freed at that point. I loaded Blender up and ran a few frames again with Process Explorer open, and noticed this:
memory usage climbing between frames
memory usage climbing between frames
A couple seconds before each frame starts rendering, there is a little step up in the VRAM used, and it does not go down after the rendering finishes. The increase in VRAM seems to occur right around the instant this is printed in the console:

Code: Select all

[LuxRays][1016.266] Total BVH memory usage: 1125128Kbytes
[LuxRays][1016.375] Adding DataSet accelerator: EMBREE
[LuxRays][1016.375] Total vertex count: 71893735
[LuxRays][1016.375] Total triangle count: 24151310
[LuxRays][1018.250] EmbreeAccel build time: 1875ms
[LuxRays][1018.797] [Device GeForce RTX 2080 Ti OpenCLIntersect] BVH mesh vertices buffer size: 842504Kbytes
[LuxRays][1019.859] [Device GeForce RTX 2080 Ti OpenCLIntersect] BVH nodes buffer size: 1125128Kbytes
[LuxRays][1020.266] [BVHKernel] Compiler options: -D LUXRAYS_OPENCL_KERNEL -D PARAM_RAY_EPSILON_MIN=1e-05f -D PARAM_RAY_EPSILON_MAX=0.1f -D LUXRAYS_OPENCL_DEVICE -D LUXRAYS_OS_WINDOWS -cl-fast-relaxed-math -cl-mad-enable
[LuxRays][1020.266] [BVHKernel] Compiling kernels
[LuxRays][1020.266] [BVHKernel] Program cached
[LuxRays][1020.797] [Device GeForce RTX 2080 Ti OpenCLIntersect] BVH mesh vertices buffer size: 842504Kbytes
[LuxRays][1021.875] [Device GeForce RTX 2080 Ti OpenCLIntersect] BVH nodes buffer size: 1125128Kbytes
[LuxRays][1022.266] [BVHKernel] Compiler options: -D LUXRAYS_OPENCL_KERNEL -D PARAM_RAY_EPSILON_MIN=1e-05f -D PARAM_RAY_EPSILON_MAX=0.1f -D LUXRAYS_OPENCL_DEVICE -D LUXRAYS_OS_WINDOWS -cl-fast-relaxed-math -cl-mad-enable
[LuxRays][1022.266] [BVHKernel] Compiling kernels
[LuxRays][1022.266] [BVHKernel] Program cached
[LuxCore][1022.344] [PathOCLRenderEngine] OpenCL task count: 524288
[LuxCore][1022.344] [PathOCLBaseRenderEngine] OpenCL max. page memory size: 2883584Kbytes
Is it weird that it does the BVH and compilation twice? I do have 2 RTX 2080 Ti so I suppose that's related. But I wonder if only 1 of them is being freed after the render?

I'll try some more tests (1 GPU only... CPU only...) and see what happens...

I'm using Blender 2.83 beta with latest daily BlendLuxCore (20b0093) on Windows 10. nVIDIA GeForce Game Ready Driver 441.66. Exact hardware in my signature.

Edit: CPU-only gives no issues. Running on a single GPU gives the same problem as shown above (except that it only does the BVH and compilation once), and when the memory amount reaches the max, Blender crashes and closes instantly.

Edit 2: I get the same result in Blender 2.82a. Just before each successive frame, the memory usage seems to go up around 155mb (between approx. 154mb and 167mb; sample size: about 12 frames)

Edit 3: Increase seems relatively constant at about 155mb, even when varying the render resolution and rendering a completely different .blend file

Edit 4: Finally figured out why CUDA wasn't working. nvrtc64_101_0.dll is missing from the BlendLuxCore-latest-win64.zip! Managed to find a copy from a previous CUDA-specific build. Now CUDA is working, and the memory gets freed after a render. So maybe it's just an OpenCL specific problem right now?

Edit 5: Ha, the DLL is now back in the new daily build. :lol:

Edit 6: Decided to test it - switched from CUDA back to OpenCL and rendered a few frames. The difference is quite noticeable:
CUDA (left) vs OpenCL (right)
CUDA (left) vs OpenCL (right)
AMD Ryzen Threadripper PRO 5995WX 64-Cores | 2x Gigabyte RTX 4090 Gaming OC
ASUS Pro WS WRX80E-SAGE SE WIFI | 256GB Kingston Server Premier ECC Unbuffered DDR4
User avatar
Sharlybg
Donor
Donor
Posts: 3101
Joined: Mon Dec 04, 2017 10:11 pm
Location: Ivory Coast

Re: Opencl clenqueuwritebuffer bug

Post by Sharlybg »

This issue is a real pain. Yesterday night i had to wake up more than 10 Times to relaunch a 1000 frames test animation.
Any solution will be welcome.
Last edited by Sharlybg on Wed Jun 03, 2020 7:15 am, edited 1 time in total.
Support LuxCoreRender project with salts and bounties

Portfolio : https://www.behance.net/DRAVIA
Martini
Posts: 125
Joined: Fri Nov 23, 2018 11:36 am
Location: Australia

Re: Opencl clenqueuwritebuffer bug

Post by Martini »

Sharlybg wrote: Tue Jun 02, 2020 9:11 pm This issue is a real pain. Yesterday i had to wake more than 10 Times to relaunch a 1000 frames test animation.
Any solution will be welcome.
Ouch. :cry:

If there is a possibility to use CUDA instead of OpenCL, I might suggest that?

Otherwise, try rendering in small batches from the CLI.

Code: Select all

\path\to\blender.exe -b "\path\to\your\scene.blend" -s 1 -e 40 -a
\path\to\blender.exe -b "\path\to\your\scene.blend" -s 41 -e 80 -a
\path\to\blender.exe -b "\path\to\your\scene.blend" -s 81 -e 120 -a
etc. for however many frames you have :lol:

You can paste those all into a batch file (e.g. render.cmd) and it should basically run unattended.
AMD Ryzen Threadripper PRO 5995WX 64-Cores | 2x Gigabyte RTX 4090 Gaming OC
ASUS Pro WS WRX80E-SAGE SE WIFI | 256GB Kingston Server Premier ECC Unbuffered DDR4
User avatar
Sharlybg
Donor
Donor
Posts: 3101
Joined: Mon Dec 04, 2017 10:11 pm
Location: Ivory Coast

Re: Opencl clenqueuwritebuffer bug

Post by Sharlybg »

If there is a possibility to use CUDA instead of OpenCL, I might suggest that?
Had the same problem with CUDA before so i didn't test it. But with the new unified package CUDA work just fine :D

You litterally save me thanks.
Support LuxCoreRender project with salts and bounties

Portfolio : https://www.behance.net/DRAVIA
Post Reply