Which black level should I trust?

Submitted by MichelRouzic on

I know that my Canon camera's raw offset is meant to be 2048, however often libraw reports slightly different values in rawdata.color.black like 2044 for instance. Which one is more accurate? The reported value seems like it might be prone to being influenced by noise. Can the true offset vary this way or should I take libraw's reported value as an approximation of the ideal value? And if libraw's value is to be trusted then why is that?

Using 2044 instead of 2048 can result in blacks being raised to a value of 1 (in 8-bit sRGB) and proportionally more if the gain is boosted as well as with WB corrections (for instance on a night shot it can easily raise blacks on the blue channel to 7 in sRGB), so this is quite important.

Thank you!

Forums: 

Please note, that imgdata

Submitted by lexa on

Please note, that imgdata.color.black is only 'base' value with offsets stored in imgdata.cblack[], so final black level may be different.

For Canon cameras, LibRaw calculates black levels based on masked pixels.

-- Alex Tutubalin @LibRaw LLC

Right, but the question

Submitted by MichelRouzic on

Right, but the question remains, which is closer to ideal, the fixed 2048 offset or the measured black levels? Seems to me like using the masked pixels makes it quite prone to noise mostly at high ISOs, and that they deviate from the fixed offset in statistically insignificant ways (while the ideal offset might well remain the same), which is why I'm thinking the measured black levels should probably be disregarded entirely, except maybe as a hint of what the true fixed offset should be.

But since you're using the measured black levels instead then you must think that logic is wrong, this is what I'm asking about.

Very hard to tell 'which is

Submitted by lexa on

Very hard to tell 'which is closed to ideal' while discussing abstract matters.

What camera we're talking about? Could you please provide dark frames shot at same shutter speed/temperature/camera settings?

-- Alex Tutubalin @LibRaw LLC

All cameras, we're talking

Submitted by MichelRouzic on

All cameras, we're talking about all cameras, well at least the modern ones, or at the very least the ones that are supposed to have a known offset, like Canon and 2048. There must be a good reason why you chose one method over the other, right? On the 0.17 release notes you wrote "LibRaw do not rely on hardcoded black levels", well, why not? Is it because hardcoded black levels are unreliable or is it because they can't be found for all camera models? That's what I'm asking, why did you choose to do it this way and not the "it's close to 512 or 2048 so let's set it to 512 or 2048" way? I think that's a pretty fundamental question.

I've examined several samples

Submitted by lexa on

I've examined several samples (from latest cameras with approx. black level 2048) and I could not find any sample with black level below 2047 (LibRaw's value)

Could you please provide one?

-- Alex Tutubalin @LibRaw LLC

Strange, that would explain

Submitted by MichelRouzic on

Strange, that would explain why only I thought this was a problem. Here's the result from a burst I took (Canon G9 X II):

IMG_0293.CR2: black 2044, cblack 2 0 1 0
IMG_0294.CR2: black 2044, cblack 1 0 1 0
IMG_0295.CR2: black 2044, cblack 1 0 2 0
IMG_0296.CR2: black 2045, cblack 1 1 0 0
IMG_0297.CR2: black 2044, cblack 0 1 1 2
IMG_0298.CR2: black 2043, cblack 1 2 2 0
IMG_0299.CR2: black 2044, cblack 1 0 2 2
IMG_0300.CR2: black 2044, cblack 1 2 1 0
IMG_0301.CR2: black 2043, cblack 3 2 2 0
IMG_0302.CR2: black 2044, cblack 1 1 3 0
IMG_0303.CR2: black 2045, cblack 0 1 1 1
IMG_0304.CR2: black 2043, cblack 2 0 3 1
IMG_0305.CR2: black 2044, cblack 1 0 3 0
IMG_0306.CR2: black 2044, cblack 1 1 1 0
IMG_0307.CR2: black 2044, cblack 0 0 3 1
IMG_0308.CR2: black 2045, cblack 0 0 0 0
IMG_0309.CR2: black 2042, cblack 3 0 4 3
IMG_0310.CR2: black 2044, cblack 1 1 1 0
IMG_0311.CR2: black 2044, cblack 1 1 0 0
IMG_0312.CR2: black 2044, cblack 1 2 1 0
IMG_0313.CR2: black 2042, cblack 3 5 4 0

For what it's worth I took another burst just a couple of minutes later (same settings except higher ISO) and all these files in that burst have exactly "black 2049, cblack 0 0 0 0"

Here is IMG_0298.CR2 http://www.mediafire.com/file/b2xzd3tit3gww85/IMG_0298.CR2

Thank you for the sample (I

Submitted by lexa on

Thank you for the sample (I've checked only EOS cameras, will inspect G9X2 in more depth).

Here is histogram of masked pixels: https://www.dropbox.com/s/06nyv7dhu56n4i1/Screenshot%202017-03-25%2022.4...
(black subtraction turned off).
And here is the same histogram, but zoomed in: https://www.dropbox.com/s/d1grqdcl343tncj/Screenshot%202017-03-25%2022.5...

Please note, that only one value of 4 is used (so camera effectively 12 bit at this ISO).

With such high noise (screenshot #1, see histogram bell width) difference in one 'real' level is very hard to see in practice (and I do not see any visible difference between automatic black level and manual set to 2047 or 2048).

-- Alex Tutubalin @LibRaw LLC

Wow you're right, but look at

Submitted by MichelRouzic on

Wow you're right, but look at the histogram for the whole picture, it's very strange, around 2048 you have values that never occur which gives these huge peaks and troughs but the rest of the histogram is much more normal or at least has a much tamer version of the problem. I knew that ADCs could have such quirks but it still looks pretty odd.

http://i.imgur.com/iqXfe1E.png

To be honest it kind of makes you want to reconsider relying on values in the vicinity of 2048 to determine anything.

Thank you for the samples,

Submitted by ib on

Thank you for the samples, most interesting.

A side note: in LibRaw we read Canon's metadata black level too, have a look at libraw_types.h, libraw_canon_makernotes_t -- it is in ChannelBlackLevel[4]. You can add something like
if (Canon.ChannelBlackLevel[0])
printf("ChannelBlackLevel (from makernotes): %d %d %d %d\n",
Canon.ChannelBlackLevel[0],
Canon.ChannelBlackLevel[1],
Canon.ChannelBlackLevel[2],
Canon.ChannelBlackLevel[3]);
to raw-identify and it will print those levels if called with '-v'.

--
Iliah Borg

I tried these (on my dark

Submitted by MichelRouzic on

I tried these (on my dark frame shot) and I get 10000 0 0 389. and AverageBlackLevel says 2597, these values look strange.

Btw I was gonna printf everything but turns out it's much quicker to use your debugger to have a quick look at everything ;)

I've got 0.18.2, I literally

Submitted by MichelRouzic on

I've got 0.18.2, I literally just got started with LibRaw a few days ago after looking at the dcraw code made me despair.

In dcraw.c after the line

Submitted by ib on

In LibRaw/dcraw.c after the line 7662 you have:
if ((unique_id == 0x03970000) || // G7 X Mark II
(unique_id == 0x80000394)) // EOS M5

make it:
if ((unique_id == 0x03970000) || // G7 X Mark II
(unique_id == 0x04100000) || // G9 X Mark II
(unique_id == 0x80000394)) // EOS M5

--
Iliah Borg

I'm looking at the source for

Submitted by MichelRouzic on

I'm looking at the source for LibRaw 0.18.2 and in dcraw/dcraw.c I don't see this at all at line 7662. Lines 7662 and 7663 read this:

    { "Olympus E-PL7", 0, 0,
	{ 9197,-3190,-659,-2606,10830,2039,-458,1250,5458 } },

If you are working with

Submitted by ib on

If you are working with archive, edit dcraw_common.cpp in /internal, after the line 5479

--
Iliah Borg

Yes, histogram of your _0298

Submitted by lexa on

Yes, histogram of your _0298 shot looks strange, I suspect some kind of raw data processing applied in camera.

-- Alex Tutubalin @LibRaw LLC

I'm tempted to think it might

Submitted by MichelRouzic on

I'm tempted to think it might be something more upstream, something with the ADC or a sort of calibration curve applied at some point around the ADC.

Indeed, and look at what

Submitted by MichelRouzic on

Indeed, and look at what happens when you superimpose all 4: http://i.imgur.com/EovPwIF.png

So for every 4 values in the histogram you have one with green (not sure which but it's one of the two), the next one has both blue and the other green, the next one has red only, and finally there's one with no values at all. This would make sense if they were all gathered in the same bins but instead they're spread differently, like they're being digitised differently maybe?

It might be worth noting that the G7 X Mark I (which preceded the related G9 X) only had 12-bit values, so maybe the bump to 14-bit was done with cutting a corner somewhere.

Followup:

Submitted by lexa on

Followup:

standard deviation value for black masked pixels is about 37 for this shot. So 5 lower bits (for 14-bit scale) are useless, difference between black at 2048 and 2044 to be masked under the noise.

-- Alex Tutubalin @LibRaw LLC

And another request: could

Submitted by lexa on

And another request: could you please provide dark frame shot (lens caps on) with same exposure (1/80, ISO 2000)?

-- Alex Tutubalin @LibRaw LLC

I had a look at full raw

Submitted by MichelRouzic on

I had a look at full raw picture histograms with daylight pictures at ISO 125 or even darker shots at ISO 6400 and we've got the same pattern around 2048.

In fact I just looked at a much older daylight ISO 125 shot taken with my old G9 X (I currently have both a G9 X and a G9 X II) and it's no better: http://i.imgur.com/sfMDaKr.png < the histogram is perfectly antialiased, it's no visualisation artifact, that's exactly how it is, you've got the strict 12-bit comb pattern that gradually fades as you go up in values.

Here's the dark frame http://www.mediafire.com/file/1823atsyamr1xx5/IMG_1554.CR2 , I'll spare you any suspense, here's the histogram: http://i.imgur.com/pSVFFLK.png

As for any noise reduction I just looked and I do have "High ISO speed NR" set to "Standard", but I don't think it (and sure hope it doesn't) affects the RAW. According to this it doesn't http://www.learn.usa.canon.com/resources/articles/2011/high_iso_noise_re.... But again I get the same histogram pattern with ISO 125 shots.

As for the noise level yep there's a lot of it but I stack bursts (usually at least 20 frames, though I'm thinking of going higher) which reduces noise quite a bit, so at that point getting the offset right can make a difference I suppose, mostly if like I said added gain and WB end up boosting that offset quite a bit. But yeah I suppose a bit of extra quantisation noise from this histogram quirk doesn't do much.

Thank you for the dark frame.

Submitted by lexa on

Thank you for the dark frame.
I've averaged central part of the image using RawDigger: https://www.dropbox.com/s/ciftfhp8zgfkxfp/Screenshot%202017-03-26%2008.4...
Black subtraction is off, averaged values are 2044.5, 2045.6, 2047.7, 2046.2 (rounded in red).

Entire image is purple because black subtraction is turned off.

Next screenshot shows black values determined by LibRaw using black frame integration: https://www.dropbox.com/s/fcxybepvflf0ec9/Screenshot%202017-03-26%2008.5...
(bottom left corner): 2044, 2045, 2047, 2046

Looks like masked pixels averaging is right (may be 1 level off for G2), it is in very good agreement with black frame ('real black') data.

I do not see any reason to switch from masked pixels averaging to metadata value.

-- Alex Tutubalin @LibRaw LLC

I agree, it seems that there

Submitted by MichelRouzic on

I agree, it seems that there are pretty consistent biases, so I have the answer to my original question, thank you all!

One thing I'm wondering though is why is there a difference across channels? Because electronically, they should be the same, just with different filters on top, right? But then again comparing histograms there are clearly differences in which values which channel have lots of or none of. I guess I'll do some tests to figure this out, comparing different regions.

Okay I did some tests with a

Submitted by MichelRouzic on

Okay I did some tests with a black frame by doing averages on various non-overlapping windows and the biases across channels are consistent.

Followup #3:

Submitted by lexa on

Followup #3:

G9X-II samples from photographyblog all have Black Level about 2047-2050. Do you use some special mode (noise reduction??) on your camera?

UPD: also, these files do not show 'one of 4' histogram even for higher ISO (3200-6400-12800). Unfortunately, no samples with intermediate ISO (2000) in this site.

-- Alex Tutubalin @LibRaw LLC

Speaking of photographyblog

Submitted by MichelRouzic on

Speaking of photographyblog samples and crazy histograms, I've tested my libraw code on samples from other cameras and I haven't seen anything crazier than a Sony RX100 V's histogram. It's coded on 14-bits (actually perhaps more since it says the maximum is 17220), but at the lower end it only needs 12-bits, and it goes down in steps as you go up in values! So there's a 11-bit section, then 10-bit, then 9-bit in the highlights! It probably helps with the compression? Anyway I thought you'd be interested to know about this in case you didn't already.

http://i.imgur.com/4GdKyxk.png