POV-Ray : Newsgroups : povray.general : How can we speed up the saving process? Server Time: 30 Sep 2020 23:57:45 GMT
  How can we speed up the saving process? (Message 8 to 17 of 17)  
<<< Previous 7 Messages Goto Initial 10 Messages
From: Kima
Subject: Re: How can we speed up the saving process?
Date: 12 Jul 2020 12:20:01
Message: <web.5f0aff9b1af26a06ecc0fada0@news.povray.org>
William F Pokorny <ano### [at] anonymousorg> wrote:
> On 7/11/20 12:17 AM, Kima wrote:
> ...
> > As an update, if it might be beneficial to others.
> >
> > In my experience, the slowest format to save is PNG. Of course, PNG is the best
> > and common format in terms of quality and size.
> >
> > Saving in JPEG is the fastest though it is highly compressed. Unfortunately, a
> > heavily lossy format like JPEG is not suitable for the purpose of ray tracing.
> >
> > JPEG with 100% quality is close to PNG (POV-Ray see 85% by default). The
> > official document emphasises that even with 100% quality, there might be some
> > artefacts. I am inspecting this failure but haven't found any yet. JPEG with
> > 100% quality is about 40% of PNG size.
> >
> > As a side note, I believe the default value of compression in POV-Ray (85%) is
> > too low. The default value in ImageMagick for conversion to JPEG is 92%. It does
> > not make sense to use many resources to ray trace a scene and save it as a
> > heavily lossy format.
> >
>
> Thanks for the information. I found myself thinking about your post...
> Where did exr output come out size and performance wise?
>
> There are formats other than exr which don't compress on output - ppm is
> another I think, and you can set the depth there to 16 bits per color
> channel too. If the uncompressed output is much faster, perhaps use one
> of those formats and compress/convert with external scripts?
>
> I'm no image format expert (Christoph was the guy there) - and I haven't
> spent much time in the image file handling code. Still, I'll risk adding
> a few POV-Ray code related comments. :-)
>
> - The jpeg output doesn't dither which may be part of the speed
> difference relative to png.
>
> - The jpeg library itself looks to support a no-compression jpeg output
> by setting the quality to 0, but the current POV-Ray wrap maps <=1
> values (the default compression value is -1) to 85%.
>
> - I'd say 85% jpeg compression is sometimes suitable. :-) Just depends
> on what one is doing with POV-Ray. Aside: IIRC 85% was chosen to line up
> with the effective compression relative to some previous jpeg output,
> not because it was considered the best quality choice.
>
> - POV-Ray mostly uses standard external libraries when reading and
> writing images so the compression de-compression isn't POV-Ray per sa.
> The compression approaches are not all the same - with higher quality
> and lossless compression tending to be the most expensive. Which you
> probably knew...
>
> Aside:
>
> The jpeg library efforts are somewhat fractured. Most support no
> compression option I believe, some lossless too, etc. There is though
> one library better optimized for modern CPUs (turbo something?) and it's
> not currently the library used by POV-Ray. I think google did much of
> the work IIRC. A bit of a tangle in any case.
>
> Somewhere (github?) I ran across a highly optimized for simd
> architectures png library with benchmarks that looked great compared to
> the common png library. But, during my glance, it wasn't clear how much
> real world use of it there was.
>
> Anyway... I guess, it's somewhat meaningless practically, but there are
> image libraries out there which - perhaps - could enable better image
> input/output performance in POV-Ray.
>
> Bill P.

Thanks for the detailed description.

As the number of available cores/threads, it is vital to reduce this
single-thread process when creating many scenes (e.g., for animation). I am
experimenting with different cases to better understand possible outlets.

Compression is not the key factor. In the case of tga, it is only 40% faster to
use the uncompressed output as compared with the compressed one.

ppm and exr both are faster than png but not close to JPEG.

I reckon a key factor (if not the main) is the absence of alpha channel in JPEG.

Quality 0 will lead to the default 85%. Non-compression is the quality of 100%.
Technically, it is not compressed by the JPEG standard, but it is still
compressed because of the JPEG architecture algorithm (leading to the
possibility of artefacts).

I am trying to find out the chance of artefacts when using JPEG. If it is
negligible, then I believe JPEG 100% is a better choice than PNG when
transparency is not required.


Post a reply to this message

From: Bald Eagle
Subject: Re: How can we speed up the saving process?
Date: 12 Jul 2020 12:45:08
Message: <web.5f0b05511af26a06fb0b41570@news.povray.org>
I really haven't looked into graphics file formats in a long time, so I'll just
ask this and see what happens.  ;)

What about BMP or GIF?
I also never see anyone use RAW file formats - except for professional
photographers, and it seems a bit --- complex.


Post a reply to this message

From: Kima
Subject: Re: How can we speed up the saving process?
Date: 12 Jul 2020 14:35:01
Message: <web.5f0b1f451af26a06ecc0fada0@news.povray.org>
"Bald Eagle" <cre### [at] netscapenet> wrote:
> I really haven't looked into graphics file formats in a long time, so I'll just
> ask this and see what happens.  ;)
>
> What about BMP or GIF?
> I also never see anyone use RAW file formats - except for professional
> photographers, and it seems a bit --- complex.

RAW is technically useless (at least not beneficial). In a digital camera, RAW
formats store the exact information received by the sensor. Then, the image is
simulated from the lights sensed by the sensor.

POV-Ray simulates the color of each pixel by ray tracing. For this reason,
POV-Ray has dropped supporting RAW.

BMP like PPM and TGA is a binary map of pixels (bitmap). PNG has the same
information (almost) but with better size management and zip-compression (not
affecting the information stored).

The best format (in terms of size and accuracy) is PNG.

Of course, POV-Ray is not slow in producing PNG. As I tested, converting an
uncompressed format to PNG takes the same time. As Bill correctly mentioned,
POV-Ray uses other libraries for producing each image format.


Post a reply to this message

From: Alain Martel
Subject: Re: How can we speed up the saving process?
Date: 12 Jul 2020 17:00:26
Message: <5f0b41aa$1@news.povray.org>
Le 2020-07-12 à 08:42, Bald Eagle a écrit :
> 
> I really haven't looked into graphics file formats in a long time, so I'll just
> ask this and see what happens.  ;)
> 
> What about BMP or GIF?
> I also never see anyone use RAW file formats - except for professional
> photographers, and it seems a bit --- complex.
> 
> 

BMP is close to a raw format. No compression and minimal header.

GIF only support up to 256 colours and is the grandfather of PNG. 
Meaning that PNG wad developed to address the limitation of GIF and 
improve the compression.


Post a reply to this message

From: Dick Balaska
Subject: Re: How can we speed up the saving process?
Date: 13 Jul 2020 06:17:06
Message: <5f0bfc62$1@news.povray.org>
Am 7/10/20 12:19 AM, also sprach Kima:

> 
> It takes about 5s to save output.png when almost no CPU resource is used.
> Obviously, I hope to reduce this period.
> 
> Since the file is small 5-7MB, disk speed cannot be the bottleneck, but I
> checked the saving with an SSD and no difference.
> 
> I use v3.8 on Ubuntu, and my command is
> 
> povray +Iinput.pov -GA +Q11 +MB +UL +UV +A -D -V +TH +W3840 +H2160 +UA
> +Ooutout.png
> 

I've written 1000s of 3MB .png files (1080p) and never noticed any 
bottleneck with writing the file. On SSD, HDD, Ubuntu and Windows.

One speed trick that I do is to disable the pov-state file by putting in 
my .ini file
Create_Continue_Trace_Log=Off


-- 
dik
Rendered 50,081,587,200 of 50,081,587,200 pixels (100%)


Post a reply to this message

From: Thorsten
Subject: Re: How can we speed up the saving process?
Date: 13 Jul 2020 07:17:12
Message: <5f0c0a78$1@news.povray.org>
On 10.07.2020 20:28, Kima wrote:
> You are right! I was able to reduce this period by changing the output format.
> It is not a matter of compression but data handling since the shortest time was
> to save as a lossy JPEG.
> 
> Anyhow, I should find a way to do this single-thread process in the background
> while starting next rendering.

This behavior sounds more like a configuration error of your harddisk 
drivers or some other odd configuration on your Linux system. The only 
rational reason why JPEG would be "fastest" is that is produces the 
smallest file of all file formats, and hence the actual data written to 
disk is the least.

So you may want to start checking your local configuration.

Thorsten


Post a reply to this message

From: Kima
Subject: Re: How can we speed up the saving process?
Date: 13 Jul 2020 11:25:01
Message: <web.5f0c44061af26a06ecc0fada0@news.povray.org>
Dick Balaska <dic### [at] buckosoftcom> wrote:
> Am 7/10/20 12:19 AM, also sprach Kima:
>
> >
> > It takes about 5s to save output.png when almost no CPU resource is used.
> > Obviously, I hope to reduce this period.
> >
> > Since the file is small 5-7MB, disk speed cannot be the bottleneck, but I
> > checked the saving with an SSD and no difference.
> >
> > I use v3.8 on Ubuntu, and my command is
> >
> > povray +Iinput.pov -GA +Q11 +MB +UL +UV +A -D -V +TH +W3840 +H2160 +UA
> > +Ooutout.png
> >
>
> I've written 1000s of 3MB .png files (1080p) and never noticed any
> bottleneck with writing the file. On SSD, HDD, Ubuntu and Windows.
>
> One speed trick that I do is to disable the pov-state file by putting in
> my .ini file
> Create_Continue_Trace_Log=Off
>
>
> --
> dik
> Rendered 50,081,587,200 of 50,081,587,200 pixels (100%)

Thanks, Dick.

The Create_Continue_Trace_Log=Off was quote useful. I was able to speed up the
process noticeably.


Post a reply to this message

From: Kima
Subject: Re: How can we speed up the saving process?
Date: 13 Jul 2020 11:30:00
Message: <web.5f0c458e1af26a06ecc0fada0@news.povray.org>
Thorsten <tho### [at] trfde> wrote:
> On 10.07.2020 20:28, Kima wrote:
> > You are right! I was able to reduce this period by changing the output format.
> > It is not a matter of compression but data handling since the shortest time was
> > to save as a lossy JPEG.
> >
> > Anyhow, I should find a way to do this single-thread process in the background
> > while starting next rendering.
>
> This behavior sounds more like a configuration error of your harddisk
> drivers or some other odd configuration on your Linux system. The only
> rational reason why JPEG would be "fastest" is that is produces the
> smallest file of all file formats, and hence the actual data written to
> disk is the least.
>
> So you may want to start checking your local configuration.
>
> Thorsten

Thorsten, it has nothing to do with writing the data on disk. It is about the
time required to produce JPEG or PNG file.

Save the output to a bitmap file (e.g., BMP) or even use any large image (not
produced by POV-Ray). It takes much longer to convert it to PNG as compared with
JPEG. Not just because the produce JPEG is smaller.

One reason is that PNG is zip-compressed (normally zlib).


Post a reply to this message

From: Thorsten
Subject: Re: How can we speed up the saving process?
Date: 13 Jul 2020 15:27:20
Message: <5f0c7d58$1@news.povray.org>
On 13.07.2020 13:29, Kima wrote:
> Thorsten, it has nothing to do with writing the data on disk. It is about the
> time required to produce JPEG or PNG file.

No, it is not.

> Save the output to a bitmap file (e.g., BMP) or even use any large image (not
> produced by POV-Ray). It takes much longer to convert it to PNG as compared with
> JPEG. Not just because the produce JPEG is smaller.
> 
> One reason is that PNG is zip-compressed (normally zlib).

No, believe me I know because I actually implemented the JPEG code in 
POV-Ray. The ZIP compression does not take any longer than the JPEg 
compression.

Your system very clearly has some sort of problem, and you really want 
to find out what that problem is because it most likely also leads to 
poor performance of other programs.

Thorsten


Post a reply to this message

From: Thorsten
Subject: Re: How can we speed up the saving process?
Date: 13 Jul 2020 15:29:15
Message: <5f0c7dcb$1@news.povray.org>
On 13.07.2020 13:22, Kima wrote:
> Thanks, Dick.
> 
> The Create_Continue_Trace_Log=Off was quote useful. I was able to speed up the
> process noticeably.

Again, this file produces 20 bytes per pixel plus a bit of overhead 
(believe me, I also wrote that code), which is next to nothing for a 
modern disk, even for a modern harddisk. So if you can get a speed-up, 
it is another major indication that you have a problem with your system 
configuration.

Thorsten


Post a reply to this message

<<< Previous 7 Messages Goto Initial 10 Messages

Copyright 2003-2008 Persistence of Vision Raytracer Pty. Ltd.