Tone curve application

Submitted by duxy on

Hi,

I am using LibRaw to decode raw images and modify the raw pixel values. I am comparing the raw pixels (no processing applied in my code) to the 'raw' pixels from the same image after conversion to DNG format using Adobe's DNG Converter. I am accessing the raw pixels by calling LibRaw::open_file() -> LibRaw::unpack() -> LibRaw::raw2image(), and then accessing the pixel values in the imgdata.image structure.

My problem is that sometimes they match with the DNG and sometimes they do not. (At least) one of the discrepancies is caused by the application of the embedded tone curve - it appears that for some cameras this is applied by default during LibRaw::unpack(), but for other cameras it is not (e.g. is for Canon, is not for Sony). Is this correct? If so, how can I determine when it is applied in unpack() and when it is not?

I am writing the raw pixel values back to dng (by modifying the DNG Converted image), and therefore I need to be able to match the pixel values by inverting the tone curve as required. Also, are there any other operations that are applied to the 'raw' pixel values during unpack() that I will need to invert?

Many thanks,

Duxy

Forums: 

The tone curve for DNG files

Submitted by lexa on

The tone curve for DNG files (Linearization Curve in DNG specs terms) is always applied on unpack() phase. There are no 'documented' options to turn this off (because RAW application always want to see linearized data), but you may use this trick:

LibRaw lr;
lr.open_file(filename);
// Replace tone curve read from metadata by linear one:
for(int i=0;i<0x10000;i++)
  lr.imgdata.color.curve[i]=i;
// Do the unpack with linear curve:
 lr.unpack();

This trick will work with data format with separate curve in metadata (all DNG files; Sony ARW2 format), but will not work with files where curve is calculated on unpack() phase (e.g. Nikon lossy NEF files).

Also please note, that raw2image() will skip 'masked pixels' (pixels outside of ActiveArea in DNG terms). To avoid this you may do the same trick: set imgdata.sizes.width and iwidth to raw_width and zero left_margin (and same with height, iheight and top_margin).

-- Alex Tutubalin @LibRaw LLC

Followup: the only format

Submitted by lexa on

Followup: the only format with linearization data applied after unpack() phase (within raw2image or dcraw_process()) is PhaseOne files (both compressed and uncompressed).

-- Alex Tutubalin @LibRaw LLC

Thanks Alex, I used your

Submitted by duxy on

Thanks Alex, I used your suggested code snippet. I'm also testing for application of the tone curve during unpack(), and applying the inverse tone curve as necessary at that point.

Inversed tone curve after

Submitted by lexa on

Inverted tone curve after unpack() should work fine too.

BTW, it is better to manipulate raw data in 'linear domain' (so, after real tone curve applied) and apply inverted curve before packing your data back (to DNG).

-- Alex Tutubalin @LibRaw LLC

Thanks Alex. Could you please

Submitted by duxy on

Thanks Alex. Could you please explain to me why it is better to manipulate raw data in the linear domain? I have seen this mentioned before, but the reasoning behind it isn't clear to me. The number of individual brightness levels available is determined by the bit depth of the camera, so any interpolated values in the linear domain (calculated during image manipulation) will have to be remapped to this same set of brightness levels. What am I missing?

Another question: I have noticed that some images (e.g. ARW images from the Sony A7) are reported as having top_margin=0 and left_margin=0, but width does not equal raw_width. Is this just a case of masked pixels being on the *right* of the image (in my experience they are usually on the left)? The DNG SDK reports the width as the full raw_width, and on examination the DNG converted images fill this extra image area with repeated values of the pixel at position(x,width), i.e. it duplicates the last pixel in the row to fill the image area between width and raw_width.

Any comments would be greatly appreciated!

Linear domain is very clear

Submitted by lexa on

Linear domain is very clear in noise characteristics and so. If you work in compressed domain than noise stats becomes non-symmetric.

Sony A7 definitely has masked pixels on right side. You may use, RawDigger (for example) to examine the file and see black area clearly.

-- Alex Tutubalin @LibRaw LLC

confirming linearization table for .ARW file

Submitted by Josh Knill (not verified) on

I was looking into confirming if the linearization table given for a number of files is "correct". For ARW files, I was hoping to compare it to linearization data retrieved from dcraw; or to check to see if the raw cfa data has applied the linearization table correctly; however both libraw and dcraw do not seem to apply the linearization table for ARW files, nor is there a command I can run (that I know of) with dcraw to get the linearization table. I was curious if you knew the best means of confirming the correctness of the linearization table output for an ARW file/was curious how you have done so yourselves.

Sony/compressed (ARW2.3)

Submitted by lexa on

Sony/compressed (ARW2.3) linearization curve is applied on LibRaw::unpack() phase.
Curve is accessible via imgdata.color.curve[] array (also, one could use RawDigger application to dump linearization curve)

-- Alex Tutubalin @LibRaw LLC