POV-Ray : Newsgroups : povray.windows : Strange delay for larger sized renders Server Time: 17 Jan 2019 03:28:41 GMT
  Strange delay for larger sized renders (Message 1 to 5 of 5)  
From: Bald Eagle
Subject: Strange delay for larger sized renders
Date: 5 Dec 2016 17:25:01
Message: <web.5845a275679c3437c437ac910@news.povray.org>
Probably somewhat related to:
http://news.povray.org/povray.windows/thread/%3Cweb.56edd77df9a19c455e7df57c0%40news.povray.org%3E/

When I begin a render on a large size resolution image (10240 x 7680), PPOV-Ray
seems to "hiccup" - it pauses, I get the spinny torus thing, the title bar says
"[unresponsive]" and then it kicks in, I get the usual messages, and the parsing
continues as usual.  This seems odd to me and less related the delayed stopping,
because the rendering hasn't started yet - because the parsing hasn't even begun
yet.

Has anyone else experienced this behaviour?


Post a reply to this message

From: William F Pokorny
Subject: Re: Strange delay for larger sized renders
Date: 5 Dec 2016 18:15:15
Message: <5845aeb3$1@news.povray.org>
On 12/05/2016 12:23 PM, Bald Eagle wrote:
> Probably somewhat related to:
>
http://news.povray.org/povray.windows/thread/%3Cweb.56edd77df9a19c455e7df57c0%40news.povray.org%3E/
>
> When I begin a render on a large size resolution image (10240 x 7680), PPOV-Ray
> seems to "hiccup" - it pauses, I get the spinny torus thing, the title bar says
> "[unresponsive]" and then it kicks in, I get the usual messages, and the parsing
> continues as usual.  This seems odd to me and less related the delayed stopping,
> because the rendering hasn't started yet - because the parsing hasn't even begun
> yet.
>
> Has anyone else experienced this behaviour?
>
>

Are you setting the max image buffer memory to something larger than the 
default of 128MB? I'm not sure of the exact memory footprint, but 
suppose you'd need at least than 10240x7680x4x4 ~= 1260MB to keep the 
image in memory.

Ref:
http://wiki.povray.org/content/Reference:General_Output_Options

I admit to not having played with this option myself...

Bill P.


Post a reply to this message

From: Bald Eagle
Subject: Re: Strange delay for larger sized renders
Date: 5 Dec 2016 20:05:00
Message: <web.5845c7b878a5f3ddc437ac910@news.povray.org>
William F Pokorny <ano### [at] anonymousorg> wrote:

> Are you setting the max image buffer memory to something larger than the
> default of 128MB? I'm not sure of the exact memory footprint, but
> suppose you'd need at least than 10240x7680x4x4 ~= 1260MB to keep the
> image in memory.

Hmm - I'll test that out.   I did notice that the POV-State file is HUGE, and
then it gets smooshed into a regular jpg/png after the rendering is complete.
I suppose I'll adjust it to 2GB since the POV-State file was IIRC 1.5 GB, and
the resulting image file is only 9MB or so...


Post a reply to this message

From: Le Forgeron
Subject: Re: Strange delay for larger sized renders
Date: 5 Dec 2016 21:07:15
Message: <5845d703$1@news.povray.org>
Le 05/12/2016 à 19:15, William F Pokorny a écrit :
> On 12/05/2016 12:23 PM, Bald Eagle wrote:
>> Probably somewhat related to:
>>
http://news.povray.org/povray.windows/thread/%3Cweb.56edd77df9a19c455e7df57c0%40news.povray.org%3E/
>>
>>
>> When I begin a render on a large size resolution image (10240 x 7680),
>> PPOV-Ray
>> seems to "hiccup" - it pauses, I get the spinny torus thing, the title
>> bar says
>> "[unresponsive]" and then it kicks in, I get the usual messages, and
>> the parsing
>> continues as usual.  This seems odd to me and less related the delayed
>> stopping,
>> because the rendering hasn't started yet - because the parsing hasn't
>> even begun
>> yet.
>>
>> Has anyone else experienced this behaviour?
>>
>>
> 
> Are you setting the max image buffer memory to something larger than the
> default of 128MB? I'm not sure of the exact memory footprint, but
> suppose you'd need at least than 10240x7680x4x4 ~= 1260MB to keep the
> image in memory.
> 
> Ref:
> http://wiki.povray.org/content/Reference:General_Output_Options
> 
> I admit to not having played with this option myself...
> 
> Bill P.

You probably have a point. From my memory, Windows won't make a sparse
file for the image buffer, but allocate every sectors of it when povray
write the first bytes in it (at the very end of the file, there is a few
data that are written for later recovery).

So it needs to write about 10240 x 7680 x 5 x 4 bytes (+ a few extras),
about 1.573 GB.
At average 50 MB per second (traditional speed of write on HDD), it
would take at least 30 seconds. But things can be worst.

4 bytes: for size of float
5 for the channels of the render (RGBFT)
the other numbers, you know where they come from.


Post a reply to this message

From: clipka
Subject: Re: Strange delay for larger sized renders
Date: 6 Dec 2016 00:26:08
Message: <584605a0$1@news.povray.org>
Am 05.12.2016 um 18:23 schrieb Bald Eagle:
> Probably somewhat related to:
>
http://news.povray.org/povray.windows/thread/%3Cweb.56edd77df9a19c455e7df57c0%40news.povray.org%3E/
> 
> When I begin a render on a large size resolution image (10240 x 7680), PPOV-Ray
> seems to "hiccup" - it pauses, I get the spinny torus thing, the title bar says
> "[unresponsive]" and then it kicks in, I get the usual messages, and the parsing
> continues as usual.  This seems odd to me and less related the delayed stopping,
> because the rendering hasn't started yet - because the parsing hasn't even begun
> yet.
> 
> Has anyone else experienced this behaviour?

This is perfectly normal behaviour from a certain file size onward.

There are various different data structudes in POV-Ray that hold image data:

- The image buffer.
- The state file.
- The final image.

The image buffer is essentially a chunk of memory where POV-Ray
assembles the render results into a coherent image as they trickle in
from the render threads in somewhat random order. It can be viewed as a
kind of staging area for the image data, to get it into the right order
for processing it into the final image.

By default the image buffer is kept in memory for the sake of speedy
random access, as POV-Ray needs to do a lot of jumping around in it,
which isn't exactly the thing you want to do on a hard drive. However,
this requires that the entire image buffer (which requires 20(*) bytes
per pixel) fits inside the CPU's address space alongside the scene data.
On a 32-bit machine, this would typically limit the image size to just a
little over 12k x 12k pixels, and even less for complex scenes. (Even on
64-bit machines, having the scene plus image buffer exceed the physical
memory limit may not be a good idea.)

Therefore, if the image buffer size exceeds a somewhat arbitrary limit,
POV-Ray will fall back to keeping the image buffer on hard disk.

Since the required file capacity is known beforehand, and it is also
expected that all of it will be filled with data, POV-Ray grows the
corresponding file to the desired size right from the start, so that it
doesn't have to worry about adjusting the size as it writes data to
random positions within the file. The drawback is that the operation to
grow a file to a large size can take a few moments on Windows machines
-- and probably also on other machines if the file system doesn't
support sparse files.

And that's exactly the symptom you're seeing.


The state file is a separate beast: It is used exclusively for render
abort/continue, and is essentially a log of internal messages passed
between the front- and back-end, which includes render results, but also
other status information relevant to re-create POV-Ray's status in a
continue scenario.

The state file starts out at a size of zero, and grows from there as
needed. And while it also effectively writes out image data to the hard
disk like a file-backed image buffer, it does so in a sequential
fashion, which is much easier on the system's file I/O load.

BTW, creation of the state file can be disabled using the `-CC` option.


The final image file is yet another thing: This is where the image data
ultimately ends up in, ready to be loaded by other applications, but
only as soon as image computation has completed and all the image data
has been assembled in the image buffer.


Mental notes to myself:

- On 64-bit systems it is probably a good idea to always keep the image
buffer in memory. Even if it exceeds the physical memory size, the
accesses should be sporadic and localized enough to not displace the
scene data from physical memory (which would result in fatal thrashing),
and while the file-backed image buffer absolutely positively requires
file I/O for each access, holding the image buffer in virtual memory
will give the operating system a fighting chance to avoid at least a
portion of those file I/O operations. Allowing the user to choose
file-backed mode may be useful in fringe cases, but this can easily be
achieved by choosing a higher default for the setting while still
allowing it to be set lower.

- Growing the file-backed image buffer to the final size in a single
instant is probably a bad idea (at least on file systems that don't
support sparse files): Sure, if we would grow the file incrementally the
operating system may have to perform about the same workload, just
spread out over time; however, this is mainly file I/O workload, which
wouldn't stop the render threads from jogging along -- provided they
have been started already.


Post a reply to this message

Copyright 2003-2008 Persistence of Vision Raytracer Pty. Ltd.