Yes this is monochrome CFA data. So we need to apply the debayer algo on it.
If I do it, one file is well demosaiced, the other not. While as you said, Bayer Pattern is the same.
This code is for FITS image where data is read bottom up. BUT, that changes anything: one file will be ok (not the same) but the other one not.
The problem is this lack of consistency: this is not normal. Before 0.20 everything was ok.
Followup: EOS 1500D/200D reported visible height is odd (4015), while EOS 2000D is even (3464), If bayer pattern is calculated on top row (fcol(raw, i, c), where i,c are 0...) and applied from bottom row this will may into observed mess.
So before 0.20 everything was ok, but now my results are not consistent while I would expect it it. What I understand it is just a compatibility issue with 0.19 and 0.20.
In the image I send you, both of camera has same Bayer pattern as stated by LibRaw. But as you can see, one is well demosaiced in GBRG, the other one no. Like if the Bayer Pattern was wrong for one image, of like I had to apply an offset for debayer (If I do it works of course).
1) Canon uses different marketing names for the same model, for example:
1500D / 2000D / Rebel T7 / Kiss X90 or 450D / Digital Rebel XSi / Kiss X2
For Canons, both model and normalized_model are set based on camera ID, not string specified in EXIF
2) Yes, if pre-LibRaw 0.20 margins was not even, new code will adjust it to even value. In that case, bayer pattern will become the same for full sensor area and for visible area within margins.
I'm not familiar with FITS, so could not help here.
I appreciate your efforts to support GPR files in LibRaw. I followed the steps but after spending a few hours, was unable to get things working (I'm building on a Mac). I am wondering if you would consider posting a fork of the GPR SDK with the necessary modifications somewhere that people could build (with the correct Adobe 1.4 SDK if possible). Thank you for your consideration.
Alex,
Does that mean that when I scale the CFA data using say Tungsten multipliers, the result can be treated as though it was taken against a D65 illuminant?
As for "color profiles for specific light source", are they better because these profiles map the camera space data to XYZ and incorporate the chroma adaptation to D65 in them?
I understand these are more conceptual questions, but I am unable to find good resources that answer them.
(dmax is max(pre_mul[0...3], pre_mul is WB coeffs).
<p>
Final scaling is performed in copy_mem_image unless no_auto_bright is set:
<ul>
<li>histogram is analyzed (from right to left) to get level # where 1% of pixels are saturated
<li>output curve will map 0...level# to full output range
</ul>[/geshifilter-c]
I have two images, one saturated where the max raw value is 14355 and another unsaturated, where the raw max values are (7560,11708,4489). Turning on interpolation (color_option=0), I get max values of (20055,19270,20076) for the saturated and for the unsaturated (8481,11708,5168). If I turn auto-scale one with user_mul[1.99,1.0,1.77,1.0], I get max values of 65535 for the saturated image (fine, its saturated here also). However, for the unsaturated image, the max values are (55594,49872,39231). So, IN FACT, it did not scale the unsaturated image up to 65535. What does the scaling actually do?
Without auto-bright/auto-maximum, scaling scales (theoretical) maximum to 2^16-1.
This is somewhat tricky, for example for 14-bit canons maximum assumed 16383-black_value, while in reality data maximum is lower (as low as ~12000 for 5D Mark II at intermediate ISOs), that's why maximum adjustment was introduced (there is no way to determine real maximum value from metadata for these camera(s)).
For most other vendors, maximum is not that tricky.
I tried converting an image that did not saturate anywhere. The maximum values I get in the image are lower that 2^16. So, I am actually not certain that the procedure really does scale to the full range. I'd really like to understand what the scaling does, as it also does not seem to be just the multiplication factors. Something else is going on.
Ok, thanks for that update. I wouldn't say it is useless though! For one, you don't actually get more precision by multiplying all values by a constant, unless you are talking about interpolation. Two: for scientific images count numbers are actually important. For example, if you have astronomy images with different exposures, it is nice to be able to scale them to the same level, based on exposure count.
As long as I can multiply each image output by its scaling value in my external program, that's ok. What I really need is for output_color=0 to work with scaling off.
There is no standard way to avoid full range (0..65535) scaling in LibRaw::scale_colors (not to scale is useless because of limited ushort precision).
Possible workaround:
params.no_auto_scale = 1 to avoid LibRaw::scale_colors() call
implement own scale_colors replacement (use scale_colors() source as a reference) and provide it via callbacks.pre_preinterpolate_cb
(you'll need to subclass LibRaw and implement the callback in the subclass to allow access to internal data fields)
no_auto_scale is very special use parameter, quote from docs/API-datastruct.html:
Disables pixel values scaling (call to LibRaw::scale_colors()) in LibRaw::dcraw_process().
This is special use value because white balance is performed in scale_colors(), so skipping it will result in non-balanced image.
This setting is targeted to use with no_interpolation, or with own interpolation callback call.
For normal processing you probably need no_auto_bright=1 (and, may be, adjust_maximum_thr=0.0 too)
P.S. www.libraw.org/docs need update w/ actual LibRaw/doc/ contents (forgot to do it on LibRaw 0.20 release)
1) Ok, I'll check them out!
2) I found the problem by digging into the code. The values set in user_mul only get applied when no_autoscale is false, i.e. auto_scaling is applied. This is counterintuitive, as setting your own multipliers does not count as auto_scaling, in my opinion. I noticed the pixel values increasing by a lot more than the multipliers I entered though, and it seems to scale the image by a value determined from a sum. Is this ok? Is this consistent from image to image? I.e., if I take a 2x long exposure, will the values scale by a factor of 2 (if gammas=1.0). I will want to combine and do arithmetics with the images and I don't want artificial scales.
3) I see a change when setting output_color=0. That's great. I'll see if it helped with the interpolation.
Yes this is monochrome CFA data. So we need to apply the debayer algo on it.
If I do it, one file is well demosaiced, the other not. While as you said, Bayer Pattern is the same.
This code is for FITS image where data is read bottom up. BUT, that changes anything: one file will be ok (not the same) but the other one not.
The problem is this lack of consistency: this is not normal. Before 0.20 everything was ok.
Debayer ok: https://postimg.cc/0KMm0zVz
Debayer ko: https://postimg.cc/0bv2f6Wh
Followup: EOS 1500D/200D reported visible height is odd (4015), while EOS 2000D is even (3464), If bayer pattern is calculated on top row (fcol(raw, i, c), where i,c are 0...) and applied from bottom row this will may into observed mess.
The files you sent are monochrome tiffs, I do not have any idea what I could see from these files.
With LibRaw 0.20: both for Canon 2000D/1500D and Canon 1200D files, imgdata.idata.filters is 0x4b4b4b4b
I do not see any problem with colors when demosaicing using LibRaw::dcraw_process().
BTW, in your code i see:
g_snprintf(fit->row_order, FLEN_VALUE, "%s", "BOTTOM-UP");
Is there any possibility that this results in bayer pattern use from the bottom row, not top?
1 - It is always complicated with Canon ;). Thank you, I did not know that 1500 = 2000.
2 - OK. I send you a small example of two images of 2000D and 1200D. These images are pure RAW, in TIFF.
I get it with this code: https://gitlab.com/free-astro/siril/-/blob/master/src/io/image_formats_l...
So before 0.20 everything was ok, but now my results are not consistent while I would expect it it. What I understand it is just a compatibility issue with 0.19 and 0.20.
In the image I send you, both of camera has same Bayer pattern as stated by LibRaw. But as you can see, one is well demosaiced in GBRG, the other one no. Like if the Bayer Pattern was wrong for one image, of like I had to apply an offset for debayer (If I do it works of course).
https://filesender.renater.fr/?s=download&token=3c99e777-040b-44e9-9ecf-...
1) Canon uses different marketing names for the same model, for example:
1500D / 2000D / Rebel T7 / Kiss X90 or 450D / Digital Rebel XSi / Kiss X2
For Canons, both model and normalized_model are set based on camera ID, not string specified in EXIF
2) Yes, if pre-LibRaw 0.20 margins was not even, new code will adjust it to even value. In that case, bayer pattern will become the same for full sensor area and for visible area within margins.
I'm not familiar with FITS, so could not help here.
I find the same behavior with 1300D.
I appreciate your efforts to support GPR files in LibRaw. I followed the steps but after spending a few hours, was unable to get things working (I'm building on a Mac). I am wondering if you would consider posting a fork of the GPR SDK with the necessary modifications somewhere that people could build (with the correct Adobe 1.4 SDK if possible). Thank you for your consideration.
Alex,
Does that mean that when I scale the CFA data using say Tungsten multipliers, the result can be treated as though it was taken against a D65 illuminant?
As for "color profiles for specific light source", are they better because these profiles map the camera space data to XYZ and incorporate the chroma adaptation to D65 in them?
I understand these are more conceptual questions, but I am unable to find good resources that answer them.
Regards,
Dinesh
I've just created a pull request for this.
David
Thank you for the samples.
Current (github) LibRaw/master rejects these files.
Most likely, the fix is implemented by this commit: https://github.com/LibRaw/LibRaw/commit/cfb66cb2372b373961321cb7ebeae733...
Prior to this fix (i.e. LibRaw 0.20 release), CR3 parse may loop infinite on truncated files.
(we'll release 0.20.1 soon with this fix and couple of other fixes)
Definitely not 1-2 days, because we have some not fixed (yet) bugs.
Most likely one week or so.
Thank you. By soon do you mean 1-2 days, or weeks? If the latter I'll pull the latest updates from git.
Thanks again
David
Yes, manually changing imgdata.color.maximum after LibRaw::unpack() will effectively disable the scaling.
Hi, does this discussion & workaround also help: https://github.com/letmaik/rawpy/issues/101#issuecomment-595704322 ?
This is not what I am seeing.
I have two images, one saturated where the max raw value is 14355 and another unsaturated, where the raw max values are (7560,11708,4489). Turning on interpolation (color_option=0), I get max values of (20055,19270,20076) for the saturated and for the unsaturated (8481,11708,5168). If I turn auto-scale one with user_mul[1.99,1.0,1.77,1.0], I get max values of 65535 for the saturated image (fine, its saturated here also). However, for the unsaturated image, the max values are (55594,49872,39231). So, IN FACT, it did not scale the unsaturated image up to 65535. What does the scaling actually do?
Without auto-bright/auto-maximum, scaling scales (theoretical) maximum to 2^16-1.
This is somewhat tricky, for example for 14-bit canons maximum assumed 16383-black_value, while in reality data maximum is lower (as low as ~12000 for 5D Mark II at intermediate ISOs), that's why maximum adjustment was introduced (there is no way to determine real maximum value from metadata for these camera(s)).
For most other vendors, maximum is not that tricky.
... something to add here.
I tried converting an image that did not saturate anywhere. The maximum values I get in the image are lower that 2^16. So, I am actually not certain that the procedure really does scale to the full range. I'd really like to understand what the scaling does, as it also does not seem to be just the multiplication factors. Something else is going on.
Hi Alex,
Ok, thanks for that update. I wouldn't say it is useless though! For one, you don't actually get more precision by multiplying all values by a constant, unless you are talking about interpolation. Two: for scientific images count numbers are actually important. For example, if you have astronomy images with different exposures, it is nice to be able to scale them to the same level, based on exposure count.
As long as I can multiply each image output by its scaling value in my external program, that's ok. What I really need is for output_color=0 to work with scaling off.
Thanks!
There is no standard way to avoid full range (0..65535) scaling in LibRaw::scale_colors (not to scale is useless because of limited ushort precision).
Possible workaround:
params.no_auto_scale = 1 to avoid LibRaw::scale_colors() call
implement own scale_colors replacement (use scale_colors() source as a reference) and provide it via callbacks.pre_preinterpolate_cb
(you'll need to subclass LibRaw and implement the callback in the subclass to allow access to internal data fields)
Hi Alex,
Well, the only way I got user_mul to be applied was then no_auto_scale=0. I have always had no_auto_bright=1 and adjust_maximum_thr=0.0.
in Libraw::scale_colors() user_mul is copied to pre_mul if it is set. And then, I only see pre_mul used if use_auto_wb or use_camera_wb is set.
What I would need is to use the camera multipliers (i.e. 1.99,1.0,1.77,1.0, as defined by Canon for my camera), but no scaling done to the image.
Thanks
BTW, postprocessing in LibRaw is (very) similar to dcraw.c one, so 'dcraw annotated' may help too: https://ninedegreesbelow.com/files/dcraw-c-code-annotated-code.html
no_auto_scale is very special use parameter, quote from docs/API-datastruct.html:
Disables pixel values scaling (call to LibRaw::scale_colors()) in LibRaw::dcraw_process().
This is special use value because white balance is performed in scale_colors(), so skipping it will result in non-balanced image.
This setting is targeted to use with no_interpolation, or with own interpolation callback call.
For normal processing you probably need no_auto_bright=1 (and, may be, adjust_maximum_thr=0.0 too)
P.S. www.libraw.org/docs need update w/ actual LibRaw/doc/ contents (forgot to do it on LibRaw 0.20 release)
Oh, F'in aces! The interpolation is nice now. The output_color=0 helped!
All I need to know now is the scaling to the images when using user_mul.
Thanks!
-A
Hi Alex,
1) Ok, I'll check them out!
2) I found the problem by digging into the code. The values set in user_mul only get applied when no_autoscale is false, i.e. auto_scaling is applied. This is counterintuitive, as setting your own multipliers does not count as auto_scaling, in my opinion. I noticed the pixel values increasing by a lot more than the multipliers I entered though, and it seems to scale the image by a value determined from a sum. Is this ok? Is this consistent from image to image? I.e., if I take a 2x long exposure, will the values scale by a factor of 2 (if gammas=1.0). I will want to combine and do arithmetics with the images and I don't want artificial scales.
3) I see a change when setting output_color=0. That's great. I'll see if it helped with the interpolation.
Thanks again!
Pages