I agree, it seems that there are pretty consistent biases, so I have the answer to my original question, thank you all!
One thing I'm wondering though is why is there a difference across channels? Because electronically, they should be the same, just with different filters on top, right? But then again comparing histograms there are clearly differences in which values which channel have lots of or none of. I guess I'll do some tests to figure this out, comparing different regions.
A side note: in LibRaw we read Canon's metadata black level too, have a look at libraw_types.h, libraw_canon_makernotes_t -- it is in ChannelBlackLevel[4]. You can add something like
if (Canon.ChannelBlackLevel[0])
printf("ChannelBlackLevel (from makernotes): %d %d %d %d\n",
Canon.ChannelBlackLevel[0],
Canon.ChannelBlackLevel[1],
Canon.ChannelBlackLevel[2],
Canon.ChannelBlackLevel[3]);
to raw-identify and it will print those levels if called with '-v'.
I had a look at full raw picture histograms with daylight pictures at ISO 125 or even darker shots at ISO 6400 and we've got the same pattern around 2048.
In fact I just looked at a much older daylight ISO 125 shot taken with my old G9 X (I currently have both a G9 X and a G9 X II) and it's no better: http://i.imgur.com/sfMDaKr.png < the histogram is perfectly antialiased, it's no visualisation artifact, that's exactly how it is, you've got the strict 12-bit comb pattern that gradually fades as you go up in values.
As for any noise reduction I just looked and I do have "High ISO speed NR" set to "Standard", but I don't think it (and sure hope it doesn't) affects the RAW. According to this it doesn't http://www.learn.usa.canon.com/resources/articles/2011/high_iso_noise_re.... But again I get the same histogram pattern with ISO 125 shots.
As for the noise level yep there's a lot of it but I stack bursts (usually at least 20 frames, though I'm thinking of going higher) which reduces noise quite a bit, so at that point getting the offset right can make a difference I suppose, mostly if like I said added gain and WB end up boosting that offset quite a bit. But yeah I suppose a bit of extra quantisation noise from this histogram quirk doesn't do much.
Wow you're right, but look at the histogram for the whole picture, it's very strange, around 2048 you have values that never occur which gives these huge peaks and troughs but the rest of the histogram is much more normal or at least has a much tamer version of the problem. I knew that ADCs could have such quirks but it still looks pretty odd.
G9X-II samples from photographyblog all have Black Level about 2047-2050. Do you use some special mode (noise reduction??) on your camera?
UPD: also, these files do not show 'one of 4' histogram even for higher ISO (3200-6400-12800). Unfortunately, no samples with intermediate ISO (2000) in this site.
standard deviation value for black masked pixels is about 37 for this shot. So 5 lower bits (for 14-bit scale) are useless, difference between black at 2048 and 2044 to be masked under the noise.
Please note, that only one value of 4 is used (so camera effectively 12 bit at this ISO).
With such high noise (screenshot #1, see histogram bell width) difference in one 'real' level is very hard to see in practice (and I do not see any visible difference between automatic black level and manual set to 2047 or 2048).
Strange, that would explain why only I thought this was a problem. Here's the result from a burst I took (Canon G9 X II):
IMG_0293.CR2: black 2044, cblack 2 0 1 0
IMG_0294.CR2: black 2044, cblack 1 0 1 0
IMG_0295.CR2: black 2044, cblack 1 0 2 0
IMG_0296.CR2: black 2045, cblack 1 1 0 0
IMG_0297.CR2: black 2044, cblack 0 1 1 2
IMG_0298.CR2: black 2043, cblack 1 2 2 0
IMG_0299.CR2: black 2044, cblack 1 0 2 2
IMG_0300.CR2: black 2044, cblack 1 2 1 0
IMG_0301.CR2: black 2043, cblack 3 2 2 0
IMG_0302.CR2: black 2044, cblack 1 1 3 0
IMG_0303.CR2: black 2045, cblack 0 1 1 1
IMG_0304.CR2: black 2043, cblack 2 0 3 1
IMG_0305.CR2: black 2044, cblack 1 0 3 0
IMG_0306.CR2: black 2044, cblack 1 1 1 0
IMG_0307.CR2: black 2044, cblack 0 0 3 1
IMG_0308.CR2: black 2045, cblack 0 0 0 0
IMG_0309.CR2: black 2042, cblack 3 0 4 3
IMG_0310.CR2: black 2044, cblack 1 1 1 0
IMG_0311.CR2: black 2044, cblack 1 1 0 0
IMG_0312.CR2: black 2044, cblack 1 2 1 0
IMG_0313.CR2: black 2042, cblack 3 5 4 0
For what it's worth I took another burst just a couple of minutes later (same settings except higher ISO) and all these files in that burst have exactly "black 2049, cblack 0 0 0 0"
I've examined several samples (from latest cameras with approx. black level 2048) and I could not find any sample with black level below 2047 (LibRaw's value)
All cameras, we're talking about all cameras, well at least the modern ones, or at the very least the ones that are supposed to have a known offset, like Canon and 2048. There must be a good reason why you chose one method over the other, right? On the 0.17 release notes you wrote "LibRaw do not rely on hardcoded black levels", well, why not? Is it because hardcoded black levels are unreliable or is it because they can't be found for all camera models? That's what I'm asking, why did you choose to do it this way and not the "it's close to 512 or 2048 so let's set it to 512 or 2048" way? I think that's a pretty fundamental question.
Right, but the question remains, which is closer to ideal, the fixed 2048 offset or the measured black levels? Seems to me like using the masked pixels makes it quite prone to noise mostly at high ISOs, and that they deviate from the fixed offset in statistically insignificant ways (while the ideal offset might well remain the same), which is why I'm thinking the measured black levels should probably be disregarded entirely, except maybe as a hint of what the true fixed offset should be.
But since you're using the measured black levels instead then you must think that logic is wrong, this is what I'm asking about.
LibRaw tries to extract as much as possible from RAW.
Other software (e.g. in-camera JPEG processor) may crop more or less (it also may depend on camera/raw processor settings, for example barrel distortion correction on/off).
So, pixel level RAW/JPEG match looks not possible in general case
If you are working with archive, edit dcraw_common.cpp in /internal, after the line 5479
I'm looking at the source for LibRaw 0.18.2 and in dcraw/dcraw.c I don't see this at all at line 7662. Lines 7662 and 7663 read this:
{ "Olympus E-PL7", 0, 0, { 9197,-3190,-659,-2606,10830,2039,-458,1250,5458 } },In LibRaw/dcraw.c after the line 7662 you have:
if ((unique_id == 0x03970000) || // G7 X Mark II
(unique_id == 0x80000394)) // EOS M5
make it:
if ((unique_id == 0x03970000) || // G7 X Mark II
(unique_id == 0x04100000) || // G9 X Mark II
(unique_id == 0x80000394)) // EOS M5
Here is histogam of visible area: https://www.dropbox.com/s/09p4dcys46ldx86/Screenshot%202017-03-26%2017.3...
Black subtraction is turned off, log vertical scale.
Entire histogram looks very strange.
I agree, it seems that there are pretty consistent biases, so I have the answer to my original question, thank you all!
One thing I'm wondering though is why is there a difference across channels? Because electronically, they should be the same, just with different filters on top, right? But then again comparing histograms there are clearly differences in which values which channel have lots of or none of. I guess I'll do some tests to figure this out, comparing different regions.
I've got 0.18.2, I literally just got started with LibRaw a few days ago after looking at the dcraw code made me despair.
I'm tempted to think it might be something more upstream, something with the ADC or a sort of calibration curve applied at some point around the ADC.
Maybe your LibRaw version is not current?
I tried these (on my dark frame shot) and I get 10000 0 0 389. and AverageBlackLevel says 2597, these values look strange.
Btw I was gonna printf everything but turns out it's much quicker to use your debugger to have a quick look at everything ;)
Yes, histogram of your _0298 shot looks strange, I suspect some kind of raw data processing applied in camera.
Thank you for the dark frame.
I've averaged central part of the image using RawDigger: https://www.dropbox.com/s/ciftfhp8zgfkxfp/Screenshot%202017-03-26%2008.4...
Black subtraction is off, averaged values are 2044.5, 2045.6, 2047.7, 2046.2 (rounded in red).
Entire image is purple because black subtraction is turned off.
Next screenshot shows black values determined by LibRaw using black frame integration: https://www.dropbox.com/s/fcxybepvflf0ec9/Screenshot%202017-03-26%2008.5...
(bottom left corner): 2044, 2045, 2047, 2046
Looks like masked pixels averaging is right (may be 1 level off for G2), it is in very good agreement with black frame ('real black') data.
I do not see any reason to switch from masked pixels averaging to metadata value.
Thank you for the samples, most interesting.
A side note: in LibRaw we read Canon's metadata black level too, have a look at libraw_types.h, libraw_canon_makernotes_t -- it is in ChannelBlackLevel[4]. You can add something like
if (Canon.ChannelBlackLevel[0])
printf("ChannelBlackLevel (from makernotes): %d %d %d %d\n",
Canon.ChannelBlackLevel[0],
Canon.ChannelBlackLevel[1],
Canon.ChannelBlackLevel[2],
Canon.ChannelBlackLevel[3]);
to raw-identify and it will print those levels if called with '-v'.
I had a look at full raw picture histograms with daylight pictures at ISO 125 or even darker shots at ISO 6400 and we've got the same pattern around 2048.
In fact I just looked at a much older daylight ISO 125 shot taken with my old G9 X (I currently have both a G9 X and a G9 X II) and it's no better: http://i.imgur.com/sfMDaKr.png < the histogram is perfectly antialiased, it's no visualisation artifact, that's exactly how it is, you've got the strict 12-bit comb pattern that gradually fades as you go up in values.
Here's the dark frame http://www.mediafire.com/file/1823atsyamr1xx5/IMG_1554.CR2 , I'll spare you any suspense, here's the histogram: http://i.imgur.com/pSVFFLK.png
As for any noise reduction I just looked and I do have "High ISO speed NR" set to "Standard", but I don't think it (and sure hope it doesn't) affects the RAW. According to this it doesn't http://www.learn.usa.canon.com/resources/articles/2011/high_iso_noise_re.... But again I get the same histogram pattern with ISO 125 shots.
As for the noise level yep there's a lot of it but I stack bursts (usually at least 20 frames, though I'm thinking of going higher) which reduces noise quite a bit, so at that point getting the offset right can make a difference I suppose, mostly if like I said added gain and WB end up boosting that offset quite a bit. But yeah I suppose a bit of extra quantisation noise from this histogram quirk doesn't do much.
Wow you're right, but look at the histogram for the whole picture, it's very strange, around 2048 you have values that never occur which gives these huge peaks and troughs but the rest of the histogram is much more normal or at least has a much tamer version of the problem. I knew that ADCs could have such quirks but it still looks pretty odd.
http://i.imgur.com/iqXfe1E.png
To be honest it kind of makes you want to reconsider relying on values in the vicinity of 2048 to determine anything.
Followup #3:
G9X-II samples from photographyblog all have Black Level about 2047-2050. Do you use some special mode (noise reduction??) on your camera?
UPD: also, these files do not show 'one of 4' histogram even for higher ISO (3200-6400-12800). Unfortunately, no samples with intermediate ISO (2000) in this site.
And another request: could you please provide dark frame shot (lens caps on) with same exposure (1/80, ISO 2000)?
Followup:
standard deviation value for black masked pixels is about 37 for this shot. So 5 lower bits (for 14-bit scale) are useless, difference between black at 2048 and 2044 to be masked under the noise.
Thank you for the sample (I've checked only EOS cameras, will inspect G9X2 in more depth).
Here is histogram of masked pixels: https://www.dropbox.com/s/06nyv7dhu56n4i1/Screenshot%202017-03-25%2022.4...
(black subtraction turned off).
And here is the same histogram, but zoomed in: https://www.dropbox.com/s/d1grqdcl343tncj/Screenshot%202017-03-25%2022.5...
Please note, that only one value of 4 is used (so camera effectively 12 bit at this ISO).
With such high noise (screenshot #1, see histogram bell width) difference in one 'real' level is very hard to see in practice (and I do not see any visible difference between automatic black level and manual set to 2047 or 2048).
Strange, that would explain why only I thought this was a problem. Here's the result from a burst I took (Canon G9 X II):
For what it's worth I took another burst just a couple of minutes later (same settings except higher ISO) and all these files in that burst have exactly "black 2049, cblack 0 0 0 0"
Here is IMG_0298.CR2 http://www.mediafire.com/file/b2xzd3tit3gww85/IMG_0298.CR2
I've examined several samples (from latest cameras with approx. black level 2048) and I could not find any sample with black level below 2047 (LibRaw's value)
Could you please provide one?
All cameras, we're talking about all cameras, well at least the modern ones, or at the very least the ones that are supposed to have a known offset, like Canon and 2048. There must be a good reason why you chose one method over the other, right? On the 0.17 release notes you wrote "LibRaw do not rely on hardcoded black levels", well, why not? Is it because hardcoded black levels are unreliable or is it because they can't be found for all camera models? That's what I'm asking, why did you choose to do it this way and not the "it's close to 512 or 2048 so let's set it to 512 or 2048" way? I think that's a pretty fundamental question.
Very hard to tell 'which is closed to ideal' while discussing abstract matters.
What camera we're talking about? Could you please provide dark frames shot at same shutter speed/temperature/camera settings?
Right, but the question remains, which is closer to ideal, the fixed 2048 offset or the measured black levels? Seems to me like using the masked pixels makes it quite prone to noise mostly at high ISOs, and that they deviate from the fixed offset in statistically insignificant ways (while the ideal offset might well remain the same), which is why I'm thinking the measured black levels should probably be disregarded entirely, except maybe as a hint of what the true fixed offset should be.
But since you're using the measured black levels instead then you must think that logic is wrong, this is what I'm asking about.
Please note, that imgdata.color.black is only 'base' value with offsets stored in imgdata.cblack[], so final black level may be different.
For Canon cameras, LibRaw calculates black levels based on masked pixels.
LibRaw tries to extract as much as possible from RAW.
Other software (e.g. in-camera JPEG processor) may crop more or less (it also may depend on camera/raw processor settings, for example barrel distortion correction on/off).
So, pixel level RAW/JPEG match looks not possible in general case
Pages