Animation render abort

Use this forum for general user support and related questions.
Forum rules
Please upload a testscene that allows developers to reproduce the problem, and attach some images.
epilectrolytics
Donor
Donor
Posts: 790
Joined: Thu Oct 04, 2018 6:06 am

Animation render abort

Post by epilectrolytics »

I'm trying to render an animation of kintuX' pool scene but a strange problem occurs.
Every couple of frames the render, after reaching the sample limit, hangs there without proceeding to the next step which is denoising.
The screen gets refreshed and Blender UI is accessible and responsive, only the render refuses to conclude.

When I quit using escape, the denoiser starts but then Blender freezes at 1s elapsed time and I need to force quit.

Console does not show any error.

It does not happen every time but after five frames later or so.
Screenshot1.jpg
Screenshot2.jpg
pool_anim_tester.blend.zip
(110.36 KiB) Downloaded 136 times
Blender 2.81a with a recent BlendLuxCore Build from last week, Windows PC.
User avatar
Dade
Developer
Developer
Posts: 5672
Joined: Mon Dec 04, 2017 8:36 pm
Location: Italy

Re: Animation render abort

Post by Dade »

Have you tried if it happens disabling PGI cache (update) ? I have the feeling it is a dead lock related to periodic PGI cache update and halt condition.
Support LuxCoreRender project with salts and bounties
epilectrolytics
Donor
Donor
Posts: 790
Joined: Thu Oct 04, 2018 6:06 am

Re: Animation render abort

Post by epilectrolytics »

Yep, confirmed!

I have rendered 50 frames with 25% resolution, 500 samples and periodic update of caustic cache disabled: no problem.
Then I have repeated this with periodic update activated and in frame #7 it hangs.
User avatar
Dade
Developer
Developer
Posts: 5672
Joined: Mon Dec 04, 2017 8:36 pm
Location: Italy

Re: Animation render abort

Post by Dade »

I should have fixed the problem (I think so, it is quite tricky).
Support LuxCoreRender project with salts and bounties
epilectrolytics
Donor
Donor
Posts: 790
Joined: Thu Oct 04, 2018 6:06 am

Re: Animation render abort

Post by epilectrolytics »

Thanks Dade, I think you nailed it!

First 50 samples rendered with reduced size show no dead lock any more.
I'm doing the full size animation now which will take a day :D
epilectrolytics
Donor
Donor
Posts: 790
Joined: Thu Oct 04, 2018 6:06 am

Re: Animation render abort

Post by epilectrolytics »

Dade wrote: Tue Dec 10, 2019 11:15 am (... it is quite tricky).
Seems you're right, another problem occurs during the full size animation render:
Not a deadlock any more but a strange dropout.
In frame 37 (which I rerendered) and now again in frame 63 one graphics card remains idle and the other works only at half speed :?
As if kernel compilation was incomplete or the synchronizatiation between CPU eye tracing and OCL path tracing failed.
The resulting image looks darker and very grainy as if a part of the rays were not included (there is denoising and compositing involved making it difficult to decide what is missing).
Screenshot.jpg
The frequency of these dropouts appears to be bigger than that of the deadlocks before.

I'll let this run overnight and maybe tomorrow I can figure out how to copy the Blender console text which may give some hints :?:

EDIT: I remember vaguely that irregular animation frame dropouts concerning caustic cache also happened with the old version before the VM refactoring.
But I have probably deleted all those scenes and results.
There was a long list of issues with the first iteration of the caustic cache which led to a temporary abandonment and the development of backward/forward path tracing instead.
epilectrolytics
Donor
Donor
Posts: 790
Joined: Thu Oct 04, 2018 6:06 am

Re: Animation render abort

Post by epilectrolytics »

Update:
Percentage of faulty renders is relatively low, around 4% I guess from looking at the table (dark ones).
I will rerender them once the whole animation is through.
Screenshot.jpg
Then I will try to "catch" one of those in order to save console output.
User avatar
Dade
Developer
Developer
Posts: 5672
Joined: Mon Dec 04, 2017 8:36 pm
Location: Italy

Re: Animation render abort

Post by Dade »

Does it happen if you use a single GPU ? And CPU only ?
Support LuxCoreRender project with salts and bounties
epilectrolytics
Donor
Donor
Posts: 790
Joined: Thu Oct 04, 2018 6:06 am

Re: Animation render abort

Post by epilectrolytics »

I will test that once the animation is finished in the afternoon and report back then.
epilectrolytics
Donor
Donor
Posts: 790
Joined: Thu Oct 04, 2018 6:06 am

Re: Animation render abort

Post by epilectrolytics »

For several hours I have now tested animation sequences but could not close in on the error.

Incomplete renders from this file apparently do not happen at lower resolution (so I could not speed up the testing much) or they happen more rarely and I don't catch them within 50 frames rendered.
At least I could replicate the issue with 50 samples halt time in the first run, but not the second (or I made a mistake).
There was nothing remarkable in the console output.

Apparently the issue does not happen with denoiser and compositor disabled.
Also not with only one GPU. Did not try CPU only, took too long, pretty sure it would work ok.

I'm stopping now before I get serious headache and instead declare this a minor issue :D
The deadlock thing was a show stopper but with the low percentage of these dropouts it was no problem to rerender the eight bad files of 250 and proceed.

I already know this will come back at me in a future project, then I will give it another tackling 8-)

In the end I'm rather happy with how the animation turned out and the potential of this unique engine.
Post Reply