Hi,
I am trying to reproduce (in my own code, due to be converted into a CUDA tool) libraw's pre-processing and post-processing steps. So far I have a working "pipeline" but I need some help in understand a few steps.
What I do:
- load the image by "open_file(char*)"
- unpack the file content by "unpack()"
- call subtract_black (but it seems like black levels aren't set in my images, all imgdata.rawdata.color.cblack[x] entries are 0), min/max values before and after "subtract_black()" are identical
- safely multiply imgdata.rawdata.raw_image by 65535/imgdata.color.maximum ("safely" because I clip data higher than imgdata.color.maximum to the maximum value)
- divide imgdata.color.cam_mul[0-3] by its smallest component and use the resulting color multiplier to ...
- scale and clip the raw data at imgdata.rawdata.raw_image
- do my GPU demosaic
- resulting RGB image is supposed to be in "camera color space" now (looks too dark and colors are off, see below)
- apply imgdata.color.rgb_cam[][] matrix to RGB image (expecting sRGB result). This step completely fails, the resulting image is clipping and running into "weird color"
Notes:
In comment https://www.libraw.org/comment/4277#comment-4277 Alex says that rgb_cam is "from RAW to sRGB". I am not sure that this is what he meant, because he uses "RAW" in the same comment for the RAW, mosaic data. I assume that what he meant is "camera colorspace" or "raw color space" or whatever term works best for "unaltered color space from camera".
In comment https://www.libraw.org/comment/4519#comment-4519 Alex says that "data scaling to full range" (may I call this "normalization"?) is to be done AFTER demosaic. Is that correct, is the normalization step ("scale to full range") to be done after demosaic (i.e. in RGB colorspace)?
My questions are:
- should normalization be done before demosaic or after? If I follow above comment (scale-to-full-range after demosaic) I get "wrong" color balancing, but if I do the normalization step after my "brute force" multiply-by-maximum (see above), I get a plausible color balance (yet too dark)
- I am pretty sure that using imgdata.color.maximum is bad, since my color balance is slightly "off" compared to a standard libraw-run on the same data using gamma 1.0/1.0 (linear) and output colorspace 0 ("raw" or "unaltered" colorspace). What would be the better scale-to-max approach?
- I am unable to get the matrix multiplication to output correct color. This clearly is a problem in my workflow (see above). Since the fourth column in the matrix is 0, I don't think it has to be applied to the RAW data as Alex' comment seems to indicate, as that would set to 0 the second green pixel, effectively losing resolution. So it must be applied to RGB data after demosaic, but here I get clipping. Could you please give me a bit of pseudo-code to check against mine (see below)? Or point out the error I made (above, missing a step in the workflow?)
Thank you so much in advance!
Pseudo-code for applying the rgb_cam matrix:
for x/y in RGB-image: // loop over all pixels in RGB image
double rgbM[3]; // 3-component-vector for RGB values, "single column matrix"
rgbM[0]=rgbM[1]=rgbM[2]=0; // clear single column matrix
for(int c=0;c <3; c++) // Matrix multiplication (based on libraw code)
{
rgbM[0]+=imgdata.color.rgb_cam[0]
*RGB_channel[c];
rgbM[1]+=imgdata.color.rgb_cam[1][c]*RGB_channel[c];
rgbM[2]+=imgdata.color.rgb_cam[2][c]*RGB_channel[c];
}
for(int c=0;c<3;c++) if(rgbM[c]>65535) rgbM[c]=65535; // clipping
RGB-image(x,y) = rgbM; // set pixel color code to 3-component vector content
</cite>
Thanks again, and apologies for the lengthy message!
Mact
code messed up
Drupal messed up the code, sorry for that - I thought I double-checked. If you are having problems understanding the pseudocode, let me know, please!
Mact
dcraw_process: This is
dcraw_process: This is borrowed from dcraw proper, please make sure you've read https://ninedegreesbelow.com/files/dcraw-c-code-annotated-code.html
"seems like black levels aren't set in my images" - what camera is it, please?
Generally, if you want quality: use floating point, apply white balance and some gamma / tone curve before demosaicking, calculate and use appropriate forward colour transforms (device to destination colour space, sRGB/Adobe RGB, etc.). dcraw_process is sort of a hack, you may want to skip it altogether.
--
Iliah Borg
Thank you
Thanks, I remember having read that niedegreesbelow article before, but I will go through it again. I must have mixed up something.
> "seems like black levels aren't set in my images" - what camera is it, please?
I have tested with various firmware versions from Sony A7r3 and also (if I remember correctly, I may be wrong there) with Canon 800d, 750d and others. The above statement is definitely true for A7r3, though.
> Generally, if you want quality: use floating point, apply white balance and some gamma / tone curve before demosaicking
Agreed on floating point, that will be a (simple) step ASAP. I am interested in the gamma curve statement, though, since my goal is to match about 12 different camera types as good as possible by NOT messing with "random" (excuse the term) curves. I would like to have the process as documented, stable, reproducible and non-"guessed" as I can manage.
> calculate and use appropriate forward colour transforms
This is most probably the step where I have broken something: My conversion (see above) from "camera color space" to anything more or less well-defined (sRGB is only for testing, it is NOT a "good" colorspace of course, I am targeting linear ACES in the end).
What about my question about normalization? Alex said that normalization is to be done after demosaicing, while that sounds not quite right to me (from my understanding you actually NEED to normalize - "scale to max" - the mosaic image before demosaic, but Alex said the opposite in the note mentioned above).
The main difference I see is actually about the "spreading" of the values: If we "cut"/clip the black end (lower end) and move data to the upper end (maximize values by multiplying with "maximum") normalization here must create a different set of colors. If we would normalize BEFORE cutting/clipping, that effect would be smaller. If we normalize AFTER demosaicing, using 3 channels in parallel, we'd only really change luminance, not color.
Or am I wrong somewhere?
Mact
For ILCE-7RM3 typical black
For ILCE-7RM3 typical black level is 512. If it is not so in your process, I would fix it first thing.
Gamma curve "statement" is based on error / artifacts analysis.
Forget colour for the moment, make other things, including white balance, work.
--
Iliah Borg
good call
:-)
In my world, white balance *IS* about color ^_^
I will re-read the ninedegrees-article, still wondering about Alex' statement about normalization/scale-to-max, though.
It is possible that I checked the black-values before they got populated, which would explain why they turned up 0.
Mact
White balance is what
White balance is what normalizes the responses of red and blue channels to the response of the green ones, so as to provide equal data numbers over a neutral (grey) surface. It's not about colour, it's about space metrics, making it uniform across all (3, 4, or more - depends on CFA) colour channel axes.
None of the operations we are discussing here are about colour per se, in fact. They are about projecting the device space into a colorimetric space, where colour is known. Operations that are about colour are those that change colour.
--
Iliah Borg
no worries
That's nomenclature - I understand what you mean and have no problems in using the word "color" in that sense (in the context of libraw, that is). I normally use it in a somewhat different sense - but that does not matter at all here and is of no concern for the discussion.
> They are about projecting the device space into a colorimetric space, where colour is known
Exactly. I would not put it that way, as I consider it misunderstandable, but we are talking about the same thing here.
Mact
Colour is what camera
Colour is what camera measured and recorded (less metameric error). The goal is to preserve colour it measured and present it in a colorimetric space. One of the methods is to find and apply an optimized transform between device data space, which isn't colorimetric, and XYZ or Lab colour space. There are few methods of dealing with metameric error, too. It's one of those calibration class tasks. Next, we move to perceptual matching, and those operations do change colour.
--
Iliah Borg
some investigative results
Just an interim update:
libraw reports cblack values as all 0 for me after unpack()/subtract_black(). Yet, the actual data (RAW image at imgdata.rawdata.raw_image) has minimum 509 and maximum 6069 before and after subtract_black().
To me it looks like subtract_black() either does not use "imgdata.rawdata.raw_image" (since its data has not been changed) or I need to add another step between unpack() and subtract_black() to actually get the black data populated. I will give reading the code another go.
To be continued.
Mact
There are two black-level
There are two black-level fields.
Quote from docs: https://www.libraw.org/docs/API-datastruct.html#libraw_colordata_t
unsigned black;
Black level. Depending on the camera, it may be zero (this means that black has been subtracted at the unpacking stage or by the camera itself), calculated at the unpacking stage, read from the RAW file, or hardcoded.
unsigned cblack[4102];
Per-channel black level correction. First 4 values are per-channel correction, next two are black level pattern block size, than cblack[4]*cblack[5] correction values (for indexes [6....6+cblack[4]*cblack[5]).
For Sony files (checked w/ A7R-IV sample), black is 512 and cblack[0..3] are zero after open_file() call.
-- Alex Tutubalin @LibRaw LLC
might be the docs
Hi, Alex,
thanks!
The docs only say "can be zero" about the "black" value (which, in my case, is !=0), but not about cblack (which IS 0 for me). I think the docs could be improved by mentioning the same about the cblack values, would have helped me :-)
The 0 is in cblack even after "unpack()".
I don't see a problem here, it's really just minimizing headache for users ...
Update on my progress will come, I'm now trying to better understand the code with help of the ninedegrees article.
Mact
cblack is *per-channel
cblack is *per-channel correction to (base) black*.
-- Alex Tutubalin @LibRaw LLC
better!
Yes, that phrasing is better - you could copy&paste that into documentation, the simple addition of "correction TO BASE black" makes it clearer!
Mact
Update
So ... by not using cam_mul[] for scaling, but instead pre_mul[] and reproducing dcraw's one-step scaling-and-white-balancing I am getting a stable result.
Interestingly, this is not the (exact) same hue that dcraw (and libraw) is creating with gamma 1/1, no-auto-bright and output "raw" (unaltered camera color) *or* output "sRGB": dcraw/libraw's output is slightly (not much, really) more on the blue side, ever so slightly less saturated. Not sure why that is. Maybe because I am doing most calculations on double-precision, not integer? Since the difference is really small, I am fine with it.
I do have to normalize the image for dark/maximum (the black value and the maximum value from imgdata.rawdata.color) in order for dcraw's scaling to work correctly. I could not find the function in dcraw that is equivalent to this normalization step, but leaving it out gives too dark results, so it must be correct (I assume that scale_colors actually looks for high/low points in the image, there are some parts of the code that are still opaque to me).
This also means that my assumption above seems to be correct: Normalization is pre-demosaic, not post-demosaic.
My steps now are:
- load file
- unpack
- call "subtract_black()" (just in case black has not been subtracted by "unpack")
- normalize to embedded dark/white values (or "minimum/maximum", dcraw's mixture of terms is not helping ...)
- scale/white-balance based on pre_mul[]
- demosaic
- apply color profile (using the pre-calculated cam-rgb-to-xyz-to-output-rgb)
- apply rotation
BTW: I found lclevy’s discussion on scaling quite helpful (the site is linked from the ninedegreesbelow website).There's a caveat about that website, though (and that MAY have misled me, I cannot remember): The code quoted there about dark-subtraction uses cblack[], NOT the "global" black value. That is due to the code having populated cblack[] with "complete" black values before. This might add to why my scaling was off.
Mact
"... - call "subtract_black()
"... - call "subtract_black()" (just in case black has not been subtracted by "unpack")..."
blacks are NOT subtracted by unpack().
If you wish to use Libraw's imgdata.image[] array (not imgdata.rawdata.*), you also need to call raw2image() or raw2image_ex(). The second one could subtract black if called with (true) arg (and if other parameters you use do not prevent black subtraction on copy).
-- Alex Tutubalin @LibRaw LLC
Thanks but
hi,
no need to shout at me, I was documenting my progress here for others to hopefully help. Maybe I should have kept it to myself.
> blacks are NOT subtracted by unpack().
... may I humbly refer you to your documentation, which clearly states:
I do not wish to use the 4-component-distribution version of the data for several reasons (one, not the least, being performance, my current very crude reverse-engineered version is slightly faster than the original and that is without optimization and converting large loops to CUDA). That is why I was asking questions in order to better understand what happens under the hood.
In the end I want to use libraw for reading/unpacking RAW data and do all post-processing after that step myself, to better integrate it into pipelines. Recreating the reading part would be possible since my usecases only cover about 10 different camera models/brands and I have limitless access to all of them, but libraw (dcraw) is doing a great job there, so why reinvent the wheel ...
Thanks for the advice you gave (pointing me back to ninedegrees)! If we were able to communicate better I would volunteer for reworking the documentation, obviously, that would not be wise. Therefor: Thank you for the work you invest in maintaining libraw!
Mact
The "has been subtracted at
The "has been subtracted at the unpacking stage" statement covers "possibility": if black pattern is not covered by black/cblack[], the only way is to subtract it on unpack to prevent LibRaw user from camera-specific details.
From user's point, there is no difference between 'black subtracted by camera' (e.g. old Nikons) and 'black subtracted on unpack()'
-- Alex Tutubalin @LibRaw LLC