Animation render abort

Use this forum for general user support and related questions.
User avatar
epilectrolytics
Donor
Posts: 647
Joined: Thu Oct 04, 2018 6:06 am

Animation render abort

Post by epilectrolytics » Mon Dec 09, 2019 6:53 pm

I'm trying to render an animation of kintuX' pool scene but a strange problem occurs.
Every couple of frames the render, after reaching the sample limit, hangs there without proceeding to the next step which is denoising.
The screen gets refreshed and Blender UI is accessible and responsive, only the render refuses to conclude.

When I quit using escape, the denoiser starts but then Blender freezes at 1s elapsed time and I need to force quit.

Console does not show any error.

It does not happen every time but after five frames later or so.
Screenshot1.jpg
Screenshot2.jpg
pool_anim_tester.blend.zip
(110.36 KiB) Downloaded 8 times
Blender 2.81a with a recent BlendLuxCore Build from last week, Windows PC.
MBPro 15" 16GB i7-4850HQ GT750M, MacOS 10.13.6 & Win10Pro PC 16GB Ryzen 3900, 2 x RTX 2070

User avatar
Dade
Developer
Posts: 3585
Joined: Mon Dec 04, 2017 8:36 pm
Location: Italy

Re: Animation render abort

Post by Dade » Mon Dec 09, 2019 8:42 pm

Have you tried if it happens disabling PGI cache (update) ? I have the feeling it is a dead lock related to periodic PGI cache update and halt condition.
Support LuxCoreRender project with salts and bounties

User avatar
epilectrolytics
Donor
Posts: 647
Joined: Thu Oct 04, 2018 6:06 am

Re: Animation render abort

Post by epilectrolytics » Mon Dec 09, 2019 10:06 pm

Yep, confirmed!

I have rendered 50 frames with 25% resolution, 500 samples and periodic update of caustic cache disabled: no problem.
Then I have repeated this with periodic update activated and in frame #7 it hangs.
MBPro 15" 16GB i7-4850HQ GT750M, MacOS 10.13.6 & Win10Pro PC 16GB Ryzen 3900, 2 x RTX 2070

User avatar
Dade
Developer
Posts: 3585
Joined: Mon Dec 04, 2017 8:36 pm
Location: Italy

Re: Animation render abort

Post by Dade » Tue Dec 10, 2019 11:15 am

I should have fixed the problem (I think so, it is quite tricky).
Support LuxCoreRender project with salts and bounties

User avatar
epilectrolytics
Donor
Posts: 647
Joined: Thu Oct 04, 2018 6:06 am

Re: Animation render abort

Post by epilectrolytics » Tue Dec 10, 2019 4:42 pm

Thanks Dade, I think you nailed it!

First 50 samples rendered with reduced size show no dead lock any more.
I'm doing the full size animation now which will take a day :D
MBPro 15" 16GB i7-4850HQ GT750M, MacOS 10.13.6 & Win10Pro PC 16GB Ryzen 3900, 2 x RTX 2070

User avatar
epilectrolytics
Donor
Posts: 647
Joined: Thu Oct 04, 2018 6:06 am

Re: Animation render abort

Post by epilectrolytics » Tue Dec 10, 2019 9:18 pm

Dade wrote:
Tue Dec 10, 2019 11:15 am
(... it is quite tricky).
Seems you're right, another problem occurs during the full size animation render:
Not a deadlock any more but a strange dropout.
In frame 37 (which I rerendered) and now again in frame 63 one graphics card remains idle and the other works only at half speed :?
As if kernel compilation was incomplete or the synchronizatiation between CPU eye tracing and OCL path tracing failed.
The resulting image looks darker and very grainy as if a part of the rays were not included (there is denoising and compositing involved making it difficult to decide what is missing).
Screenshot.jpg
The frequency of these dropouts appears to be bigger than that of the deadlocks before.

I'll let this run overnight and maybe tomorrow I can figure out how to copy the Blender console text which may give some hints :?:

EDIT: I remember vaguely that irregular animation frame dropouts concerning caustic cache also happened with the old version before the VM refactoring.
But I have probably deleted all those scenes and results.
There was a long list of issues with the first iteration of the caustic cache which led to a temporary abandonment and the development of backward/forward path tracing instead.
MBPro 15" 16GB i7-4850HQ GT750M, MacOS 10.13.6 & Win10Pro PC 16GB Ryzen 3900, 2 x RTX 2070

User avatar
epilectrolytics
Donor
Posts: 647
Joined: Thu Oct 04, 2018 6:06 am

Re: Animation render abort

Post by epilectrolytics » Wed Dec 11, 2019 8:40 am

Update:
Percentage of faulty renders is relatively low, around 4% I guess from looking at the table (dark ones).
I will rerender them once the whole animation is through.
Screenshot.jpg
Then I will try to "catch" one of those in order to save console output.
MBPro 15" 16GB i7-4850HQ GT750M, MacOS 10.13.6 & Win10Pro PC 16GB Ryzen 3900, 2 x RTX 2070

User avatar
Dade
Developer
Posts: 3585
Joined: Mon Dec 04, 2017 8:36 pm
Location: Italy

Re: Animation render abort

Post by Dade » Wed Dec 11, 2019 10:07 am

Does it happen if you use a single GPU ? And CPU only ?
Support LuxCoreRender project with salts and bounties

User avatar
epilectrolytics
Donor
Posts: 647
Joined: Thu Oct 04, 2018 6:06 am

Re: Animation render abort

Post by epilectrolytics » Wed Dec 11, 2019 10:38 am

I will test that once the animation is finished in the afternoon and report back then.
MBPro 15" 16GB i7-4850HQ GT750M, MacOS 10.13.6 & Win10Pro PC 16GB Ryzen 3900, 2 x RTX 2070

User avatar
epilectrolytics
Donor
Posts: 647
Joined: Thu Oct 04, 2018 6:06 am

Re: Animation render abort

Post by epilectrolytics » Wed Dec 11, 2019 7:22 pm

For several hours I have now tested animation sequences but could not close in on the error.

Incomplete renders from this file apparently do not happen at lower resolution (so I could not speed up the testing much) or they happen more rarely and I don't catch them within 50 frames rendered.
At least I could replicate the issue with 50 samples halt time in the first run, but not the second (or I made a mistake).
There was nothing remarkable in the console output.

Apparently the issue does not happen with denoiser and compositor disabled.
Also not with only one GPU. Did not try CPU only, took too long, pretty sure it would work ok.

I'm stopping now before I get serious headache and instead declare this a minor issue :D
The deadlock thing was a show stopper but with the low percentage of these dropouts it was no problem to rerender the eight bad files of 250 and proceed.

I already know this will come back at me in a future project, then I will give it another tackling 8-)

In the end I'm rather happy with how the animation turned out and the potential of this unique engine.
MBPro 15" 16GB i7-4850HQ GT750M, MacOS 10.13.6 & Win10Pro PC 16GB Ryzen 3900, 2 x RTX 2070

Post Reply