So, here is the current status of what I'm busy implementing:

* Philosophy

- POV-Ray shall always operate in a linear RGB color space based on 
light intensity (as opposed to perceived brightness), to properly 
compute physical effects. (In the long run, this should also include a 
clear definition of the RGB primaries and white point, but at present 
this definition is limited to gamma issues.)

- In General, all image-format input files sall be converted to 
POV-Ray's operating color space before any further processing. (In the 
long run, this should include full color profile handling; however, at 
present it is limited to gamma correction.)

- For file formats with an official specification of gamma handling (and 
later also regarding color profile handling), conversion shall be based 
on that definition or recommendation. For any other files, conversion 
shall be based on a common assumed input file color space, which shall 
be configurable in the scene file and default to sRGB.

- Image-format input files used immediately in certain language 
constructs obviously indicating that they are mere data containers (e.g. 
height fields) shall be exempt from the aforementioned conversion, and 
the raw encoded values used instead. (Note however that this does not 
apply to indirect use of input image files.)

- For any individual image input file (including files otherwise exempt 
from conversion), the user shall be able to override the conversion 
rules via SDL, in order to handle files not adhering to standards, or 
adapt to the intended use of the file.

- To the extent possible in the framework designed to achieve these 
goals, reasonable effort shall be made to emulate POV-Ray 3.6 behaviour 
for legacy scenes. However, where such behaviour is obscure and/or 
difficult to emulate consistently, backwards compatibility shall not be 
preserved.


* Syntax:

- The assumed input file gamma (to be used for all input file formats 
lacking a proper definition or recommendation) can be set in the 
global_settings section as follows:

     global_settings {
         file_gamma GAMMA
     }

where GAMMA is either the keyword 'srgb' (to indicate that input files 
are assumed to match sRGB) or a numeric expression (to indicate that 
input files are assumed to be pre-corrected for displays with the 
specified gamma value).

(Note that sRGB is /not/ perfectly equivalent to a gamma 2.2 
pre-correction, though it comes very close.)

At present, the setting is global, and should therefore /not/ be messed 
with in "library" include files.

- The input file gamma of a particular file can be overriden by 
specifying "file_gamma GAMMA" after the file name, e.g.:

     image_map {
         exr "foobar.exr" file_gamma 1.8
         interpolate 2
     }

(Note that for parser design reasons, the "file_gamma" statement must 
/immediately/ follow the file name.)

This will bypass any gamma or gamma-alike adjustment normally performed 
according to the file format standards, and use a decoding gamma curve 
as specified by the user instead.


* Legacy Behaviour:

- In the presence of an 'assumed_gamma 1.0' statement, gamma correction 
of input files is always turned on, but global file_gamma defaults to 
1.0, effectively disabling gamma correction of input files /by default/ 
(for most file types; see below).

- In the presence of an 'assumed_gamma 2.2' statement (or whatever 
happens to be the hard-coded default Display_Gamma on the particular 
machine), gamma correction of input files is always turned /off/. The 
file_gamma statement can still be used on individual files to enforce 
gamma-adjustment, in order to allow for some flexibility, but it will 
/not/ automatically compensate for the assumed overall scene gamma.

- Any other value of 'assumed_gamma' will be ignored (matching the 
behaviour of all recent betas).

- Global 'file_gamma' and 'assumed_gamma' will not mix; if both are 
encountered, 'assumed_gamma' will be ignored.

- In POV-Ray 3.6, non-paletted PNG files with gAMA chunks were the 
/only/ input files which were ever gamma-adjusted, and these were 
/always/ gamma-adjusted (unless used in a data-container context). This 
will be changed even for legacy scenes; fortunately, this will not give 
different results for standard PNG files; however, it /will/ change the 
look of PNG files with a gAMA of 1.0.


* Related Changes:

- 8-bit paletted PNG files will now be gamma-adjusted (or not) just like 
ordinary 24/32-bit PNG fils.

- An sRGB chunk in PNG files will now take precedence over a gAMA chunk, 
and will actually cause the sRGB transport function to be applied 
instead of a plain 2.2 power law gamma; this is likely to cause minor 
differences in the look of most PNG input images, but will be "more 
correct".

- Gamma adjustment of files is now exclusively performed /after/ 
conversion to floating-point format, in order to improve quality.

- Several bugs in the file readers have been eliminated.

- "Out-of-the box" transparency support for GIF files has been added.


* Implementation

- High dynamic range images (presently only OpenEXR and Radiance HDR) 
are gamma-corrected "on the fly" while reading the file.

- Paletted images are gamma-corrected by adjusting their palette data 
accordingly.

- Non-paletted integer images are kept in "native" format without 
applying gamma adjustment, in order to save memory. Gamma correction is 
instead applied at render time, using a pre-computed lookup table in 
hope of gaining a bit of a speed advantage. (Lookup tables are shared 
between images with the same gamma curve of course.)


Hmm... did I forget to mention anything? Probably yes, but I guess I 
can't help it :-)

Post a reply to this message

clipka schrieb:
> * Legacy Behaviour:
> [...]
> - In POV-Ray 3.6, non-paletted PNG files with gAMA chunks were the 
> /only/ input files which were ever gamma-adjusted, and these were 
> /always/ gamma-adjusted (unless used in a data-container context). This 
> will be changed even for legacy scenes; fortunately, this will not give 
> different results for standard PNG files; however, it /will/ change the 
> look of PNG files with a gAMA of 1.0.
> [...]
> 
> Hmm... did I forget to mention anything? Probably yes, but I guess I 
> can't help it :-)

Ah yes, one more thing:

The above claim that results will not differ for most PNG files does 
/not/ hold true if assumed_gamma is not specified at all. In that case, 
POV-Ray 3.6 applied some particularly mental gamma-'correction', which 
will /not/ be matched by the new behaviour.

Post a reply to this message

Sounds good to me. One minor thing, though:

clipka <ano### [at] anonymousorg> wrote:
> - The assumed input file gamma (to be used for all input file formats 
> lacking a proper definition or recommendation) can be set in the 
> global_settings section as follows:

>      global_settings {
>          file_gamma GAMMA
>      }

  I think the keyword is a bit confusing because it doesn't make clear
whether it affects input files or output files. Could perhaps a more
unambiguous keyword be chosen, something like "input_gamma"?

-- 
                                                          - Warp

Post a reply to this message

Warp wrote:
>   Sounds good to me. One minor thing, though:
> 
> clipka <ano### [at] anonymousorg> wrote:
>> - The assumed input file gamma (to be used for all input file formats 
>> lacking a proper definition or recommendation) can be set in the 
>> global_settings section as follows:
> 
>>      global_settings {
>>          file_gamma GAMMA
>>      }
> 
>   I think the keyword is a bit confusing because it doesn't make clear
> whether it affects input files or output files. Could perhaps a more
> unambiguous keyword be chosen, something like "input_gamma"?

Globals are evil. I would instead recommend making this a bitmap only 
feature, not supplying a global setting.

	Thorsten

Post a reply to this message

Warp schrieb:
>   I think the keyword is a bit confusing because it doesn't make clear
> whether it affects input files or output files. Could perhaps a more
> unambiguous keyword be chosen, something like "input_gamma"?

I think that would be even more ambiguous, as it might be taken to 
include color values as well.

It also allows for using the same keyword both for individual files as 
well as the fallback setting.


But I'm thining about moving it to the "default" section instead anyway. 
Aside from appearing to be more fitting there, I guess that would also 
reduce the risk of people expecting it to affect the output file.

Post a reply to this message

Thorsten Froehlich schrieb:
> Globals are evil.

Why is that?

We're not talking about C global variables here I guess :-)

Post a reply to this message

Thorsten Froehlich schrieb:
> Globals are evil. I would instead recommend making this a bitmap only 
> feature, not supplying a global setting.

I think a global setting (not necessarily in the global_settings block - 
the default block would do as well) should be made available.

Suppose some guy who prefers to work with images having a gamma 
pre-correction of 1.8 for whatever reason, in some file format that 
isn't linear by default: Why shouldn't we make his life easier?

If we'd want to force users to change their standard processes to match 
the sRGB color space, then why is there a File_Gamma INI file setting to 
adjust the output file gamma?

Post a reply to this message

Le 15/09/2009 23:12, Thorsten Froehlich nous fit lire :
> Warp wrote:
>>   Sounds good to me. One minor thing, though:
>>
>> clipka <ano### [at] anonymousorg> wrote:
>>> - The assumed input file gamma (to be used for all input file formats
>>> lacking a proper definition or recommendation) can be set in the
>>> global_settings section as follows:
>>
>>>      global_settings {
>>>          file_gamma GAMMA
>>>      }
>>
>>   I think the keyword is a bit confusing because it doesn't make clear
>> whether it affects input files or output files. Could perhaps a more
>> unambiguous keyword be chosen, something like "input_gamma"?
> 
> Globals are evil. I would instead recommend making this a bitmap only
> feature, not supplying a global setting.
> 
>     Thorsten

OMHG... I'm agreeing with Thorsten.
That's a parallel univers, I will wake up soon.

More seriously, put it where it is needed: near the file picture!
That's a setting which is really tuned to the related file.

A global would have no more purpose than a global setting for text (you
can always justify the usage of a global text font setting... but nobody
ever came so far with such suggestion)

Post a reply to this message

Le_Forgeron schrieb:
> More seriously, put it where it is needed: near the file picture!
> That's a setting which is really tuned to the related file.

You didn't pay much of attention, did you? There /is/ an (optional) 
per-file setting.

But I can well imagine that there are people out there who prefer to 
/consistently/ work with files pre-corrected for some other gamma space 
than sRGB for one reason or the other, hence the additional global 
setting: To customize the default.

> A global would have no more purpose than a global setting for text (you
> can always justify the usage of a global text font setting... but nobody
> ever came so far with such suggestion)

Maybe that's because the text{} statement /requires/ you to specify a 
font, and there's no default that a user could possibly hope to customize.

You could just as well argue that there's not much purpose in a default 
{ finish {...} } statement - after all, don't the finish settings belong 
to the individual material?

Post a reply to this message

clipka schrieb:
> - Image-format input files used immediately in certain language 
> constructs obviously indicating that they are mere data containers (e.g. 
> height fields) shall be exempt from the aforementioned conversion, and 
> the raw encoded values used instead. (Note however that this does not 
> apply to indirect use of input image files.)

It would OTOH be nice if POV-Ray would treat images the same way 
everywhere (it does not in 3.6 either but that's a different story).

> 
> - Non-paletted integer images are kept in "native" format without 
> applying gamma adjustment, in order to save memory. Gamma correction is 
> instead applied at render time, using a pre-computed lookup table in 
> hope of gaining a bit of a speed advantage. (Lookup tables are shared 
> between images with the same gamma curve of course.)

A dynamic caching scheme for converted values would be an alternative - 
significantly more complex to implement of course.

As long as this is only about gamma correction this is probably 
unnecessary but with color space conversions on-the-fly conversion might 
be too expensive.

-- Christoph

Post a reply to this message

Christoph Hormann schrieb:

>> - Image-format input files used immediately in certain language 
>> constructs obviously indicating that they are mere data containers 
>> (e.g. height fields) shall be exempt from the aforementioned 
>> conversion, and the raw encoded values used instead. (Note however 
>> that this does not apply to indirect use of input image files.)
> 
> It would OTOH be nice if POV-Ray would treat images the same way 
> everywhere (it does not in 3.6 either but that's a different story).

Well, I think there's some reason to both sides; with the per-file gamma 
setting it becomes less of a problem at any rate.


>> - Non-paletted integer images are kept in "native" format without 
>> applying gamma adjustment, in order to save memory. Gamma correction 
>> is instead applied at render time, using a pre-computed lookup table 
>> in hope of gaining a bit of a speed advantage. (Lookup tables are 
>> shared between images with the same gamma curve of course.)
> 
> A dynamic caching scheme for converted values would be an alternative - 
> significantly more complex to implement of course.

... and quite definitely more overhead than gain for an 8-bit/channel 
image, and possibly so as well for a 16-bit/channel image (except for 
images so small that it would become worthwhile to convert them to float 
right away at load time). It might also be questionable whether it would 
be faster than just recomputing the value.

> As long as this is only about gamma correction this is probably 
> unnecessary but with color space conversions on-the-fly conversion might 
> be too expensive.

For some "norm" color spaces, I think the per-color lookup table 
approach might still be both possible and worthwhile - especially for 
8-bit/channel images - if separate lookup tables are used for each color 
channel, and each entry would store an RGB color instead of a scalar; 
those would then be just added - sort of a shortcat halfway through the 
matrix transformation found in various color space transformations.

Of course for full-fledged device color profiles this wouldn't be 
sufficient.

Post a reply to this message

clipka wrote:

> - POV-Ray shall always operate in a linear RGB color space based on 
> light intensity (as opposed to perceived brightness), to properly 
> compute physical effects. (In the long run, this should also include a 
> clear definition of the RGB primaries and white point, but at present 
> this definition is limited to gamma issues.)
> 
> - In General, all image-format input files sall be converted to 
> POV-Ray's operating color space before any further processing. (In the 
> long run, this should include full color profile handling; however, at 
> present it is limited to gamma correction.)
> 

So meanwhile you do agree with me that POV-Ray should have "in the long 
run" a well defined RGB color space while I do remember a few months ago 
we had some struggle about this issue ;)


> - For file formats with an official specification...
 > [SNIP (a lot)]

Sounds all very good to me.


> * Implementation
 > [SNIP]
> - Non-paletted integer images are kept in "native" format without 
> applying gamma adjustment, in order to save memory. Gamma correction is 
> instead applied at render time, using a pre-computed lookup table in 
> hope of gaining a bit of a speed advantage. (Lookup tables are shared 
> between images with the same gamma curve of course.)
> 

I already wanted to ask you about this - as at some point it seemed to 
me you intend to store all data in floating point format - but then I 
did completely forget about it.
While I think it is a very good idea to use lut's for 8 bps I do believe 
it should not be done for 16 bps as they are meant to apply things like 
inverse gamma correction without *any* visual impact. But this is not an 
important issue.


And BTW I've rewritten tiff.cpp from scratch and the most simple and 
basic cases is already working: RGB data that is ordered interleaved 
(and not in separate color planes) arranged in strips (and not tiles) 
and uses unsigned 8 or 16 bit integers (not signed ones).
The nice thing is that full floating point HDR TIFF support for 16bit 
(the same 'half' format as used by OpenEXR), 24bit (yes, such a thing 
exists) and 32bit (the 'normal' float) is already working.
And I intend to implement TIFF *output* for 32bit floats as I think it 
would be nice to have this feature available especially within an 
already well defined image file format.
So sooner or later I need to know more details about your implementation 
of the file gamma correction but most probably I will not finish the new 
TIFF implementation befor the next beta is out...


-Ive

Post a reply to this message

Ive schrieb:
> So meanwhile you do agree with me that POV-Ray should have "in the long 
> run" a well defined RGB color space while I do remember a few months ago 
> we had some struggle about this issue ;)

Ah yes? Did we?

If you're thinking of statements from me like "what's the point in 
having perfect ICC profile handling if POV-Ray gets the gamma wrong" 
(about a workaround I had described for the gamma issues with POV-Ray, 
involving an external tool that according to you doesn't do proper ICC 
handling), and consider those an argument about ICC profiles, then yes, 
in that case I guess we did :-)


> I already wanted to ask you about this - as at some point it seemed to 
> me you intend to store all data in floating point format - but then I 
> did completely forget about it.
> While I think it is a very good idea to use lut's for 8 bps I do believe 
> it should not be done for 16 bps as they are meant to apply things like 
> inverse gamma correction without *any* visual impact. But this is not an 
> important issue.

Indeed it isn't: The framework has been set up now, and adjustments to 
the handling should be quite easy. Including stuff like, checking for 
the file size and depending on that choosing whether to convert directly 
to float or use lookup.

> And BTW I've rewritten tiff.cpp from scratch and the most simple and 
> basic cases is already working: RGB data that is ordered interleaved 
> (and not in separate color planes) arranged in strips (and not tiles) 
> and uses unsigned 8 or 16 bit integers (not signed ones).
> The nice thing is that full floating point HDR TIFF support for 16bit 
> (the same 'half' format as used by OpenEXR), 24bit (yes, such a thing 
> exists) and 32bit (the 'normal' float) is already working.
> And I intend to implement TIFF *output* for 32bit floats as I think it 
> would be nice to have this feature available especially within an 
> already well defined image file format.

Great to hear those things!

> So sooner or later I need to know more details about your implementation 
> of the file gamma correction but most probably I will not finish the new 
> TIFF implementation befor the next beta is out...

It's actually quite easy as far as the image reading code is concerned. 
Nothing that would require a dramatic change in image reading code.

Post a reply to this message

clipka schrieb:
> 
> ... and quite definitely more overhead than gain for an 8-bit/channel 
> image, and possibly so as well for a 16-bit/channel image (except for 
> images so small that it would become worthwhile to convert them to float 
> right away at load time). It might also be questionable whether it would 
> be faster than just recomputing the value.

Could be.

One field where i consider performance to be possibly critical is use of 
images in isosurfaces.  This leads to a real lot of image evaluations. 
There however you nearly always use a linear gamma curve.  So a special 
case for images where file_gamma is exactly 1.0 would be a good idea.

-- Christoph

Post a reply to this message

Christoph Hormann schrieb:

> One field where i consider performance to be possibly critical is use of 
> images in isosurfaces.  This leads to a real lot of image evaluations. 
> There however you nearly always use a linear gamma curve.  So a special 
> case for images where file_gamma is exactly 1.0 would be a good idea.

That's already implemented :-)

In case the effective decoding gamma curve happens to be neutral, the 
file readers choose an image container that does not do gamma lookup at 
all, and just does the standard "/255.0f" decoding instead.

Post a reply to this message

Ive <"ive### [at] lilysoftorg"> wrote:
> While I think it is a very good idea to use lut's for 8 bps I do believe
> it should not be done for 16 bps as they are meant to apply things like
> inverse gamma correction without *any* visual impact. But this is not an
> important issue.

What if it's a lookup table that maps a 16-bit integer into a 32-bit float?
There wouldn't be any visual impact there...

Post a reply to this message

Nicolas Alvarez schrieb:
> Ive <"ive### [at] lilysoftorg"> wrote:
>> While I think it is a very good idea to use lut's for 8 bps I do believe
>> it should not be done for 16 bps as they are meant to apply things like
>> inverse gamma correction without *any* visual impact. But this is not an
>> important issue.
> 
> What if it's a lookup table that maps a 16-bit integer into a 32-bit float?
> There wouldn't be any visual impact there...

I think what Ive means is that 16 bit per color channel should already 
provide enough "headroom" to do the gamma correction in that format.

Post a reply to this message