>> libpng only needs the current and previous row to run the pixel filter. 
>> I have no idea what the DEFLATE compressor needs.
> 
>   Compression algorithms almost invariably use a buffer for the data, and
> this buffer is often surprisingly small (like some tens of kilobytes).
> (This means that most compression algorithms will *not* find and compress
> repetitions which are gigabytes apart from each other. This is for the
> simple reason that the compression routine has to use only a limited amount
> of memory and be relatively fast.)
> 
>   Some modern algorithms might use larger buffers (like some megabytes),
> but they nevertheless don't achieve enormously better compression ratios
> (for some reason there seems to be some kind of cutoff point after which
> enlarging the compression buffer does not improve the compression of average
> data significantly, but slows down the compression more than it's worth).

DEFLATE is ancient. I doubt it needs a lot of space. Most particularly, 
the space required by DEFALTE is *unrelated* to the dimensions of the 
PNG image. The PNG spec says the pixel data is optionally "filtered", 
and then fed to DEFLATE as raw data, which DEFLATE then compresses in 
the usual way.

>   The problem with compressing exabytes of data is not the compression
> algorithm itself, but the file format into which the compressed data is
> stored.

>> I note that to store (2^31 - 1) x (2^31 - 1) pixels, where each pixel 
>> requires exactly 4 bytes, requires about 18 exabytes of storage. (!!)
> 
>   Depends on what those pixels contain, and how smart your compression
> algorithm is.

I was thinking more of the amount of RAM required if you were to store 
the entire thing uncompressed. I would expect any half-decent 
compression algorithm to improve on this significantly.

>   I don't think many file systems even support files which are exabytes
> large.

I agree.

(Maybe Google BigFile or whatever the hell it's called?)

Post a reply to this message