open_bayer() creating All-Black 8-bit Images

Submitted by Falka on

For several weeks, I have been using LibRaw's 0.19 snapshot open_bayer() call to open and eventually demosaic some 16bit raw images. I got the original source from this thread: https://www.libraw.org/node/2261
It's been working beautifully.

Today I'm trying to add support for 8-bit raw images. I have an 8-bit raw image which is legitimate, as it has been successfully demosaiced using other software. My LibRaw calls were creating all-black images with the 8bit picture, so I went back to basics and tried to open the file using the original sample demonstration of open_bayer(). Still get all-0 images. Here is the code I'm using. No errors are triggered at runtime.

For reference, the first pixels of the RGGB raw image have values of 255. Each pixel is littleendian R-G-G-B one after the other in the file, with no header or borders.

/*
Opened and read the file into unsigned char * buffer, just like in the code sample. Everything appears fine, 
including the filesize, fsz.
*/
 
LibRaw rp;
rp.imgdata.params.output_tiff = 1; // Also tried setting this to 0 for PPM. No luck
rp.imgdata.params.output_bps = 8; // Adding this did not fix the problem
 
int ret = rp.open_bayer(buffer, fsz, width, height, 0, 0, 0, 0, 0, LIBRAW_OPENBAYER_RGGB, 0, 0, 0);
// Check for error using LIBRAW_SUCCESS. I never get an error here
 
std::cout << "Opened bayer... first thing at buffer is " << (int)buffer[0] << "\n"; // This prints 255 :)
 
ret = rp.unpack();
// Check for error using LIBRAW_SUCCESS. I never get an error here
 
std::cout << "Unpacked! first thing at buffer is " << (int)buffer[0] << "\n"; // This prints 255 :)
std::cout << "First thing from imgdata: " << (int)rp.imgdata.image[0] << "\n"; // This prints 0.
// Perhaps imgdata.image is where the error is? 
 
ret = rp.dcraw_process(); 
// Check for error using LIBRAW_SUCCESS. I never get an error here
 
ret = rp.dcraw_ppm_tiff_writer(fileName); // fileName is a char[] with ".tif" at the end. No problems with name.
// Check for error using LIBRAW_SUCCESS. I never get an error here

Are any of those calls incorrect? Could the problem be somewhere in the LibRaw source code? I understand that the 0.19 snapshot is not an official release, so I could have simply found a bug.
I can provide test images.

Thank you for your time!

Forums: 

Sorry for delayed publishing

Submitted by lexa on

Sorry for delayed publishing of this topic, summer is vacation time even for libraw.org moderator :)

Could you please provide image sample to test it with. open_bayer should work with 8-bit images, but it looks like it needs additional testing.

version 0.19 expected to be released this year or in Q1-2019 (we definitely will publish 1-2 snapshots before that). Snapshots are used in our applications (RawDigger, FastRawViewer) and stable for production use.

-- Alex Tutubalin @LibRaw LLC

Hi Alex, no worries! Hope it

Submitted by Falka on

Hi Alex, no worries! Hope it was good vacation time.

Here is my 8bit raw file on Google Drive:
https://drive.google.com/file/d/0B41mEzFmIzubcmxKMnVEcE14MEE/view?usp=sh...

Here it is demosaiced to a .tif using other software (with worse algorithms!):
https://drive.google.com/file/d/0B41mEzFmIzubZW0wMXE3YWo3eFk/view?usp=sh...

I found a workaround to make LibRaw accept the file (and other 8bit files from the same camera)... I store each 8bit pixel in memory as if it were a 16bit pixel, then have LibRaw output 8bits. I cannot keep using this because it is exactly twice the amount of IO that should be done!

Hope it's an easy fix

P.S. might be helpful -
This one does not have 255-level pixels at the beginning, they are quite dark in the upper left.
It is in the "RGGB" RGRGRG.../GBGBGB.../ Bayer pattern

Thanks for the sample.

Submitted by lexa on

Thanks for the sample.

Yes, this is bug in open_bayer() code, the standard tone curve (color.curve[] array) was not initialized to linear curve, but it is used in 8-bit decoder.

Just add these lines:

  for (int i = 0; i < 0x10000; i++)
    curve[i] = i;

To void CLASS initdata() (anywhere in the function body). This call is in
* dcraw/dcraw.c if you're using snapshot (and re-generate script/makefile)
* internal/dcraw_common.cpp (if you're do not regenerate this file from LibRaw's dcraw.c).

(you may patch both source files if you're not sure, if re-generate script is used, than dcraw_common.cpp to be generated from dcraw.c if needed)

Link to full patch: https://www.dropbox.com/s/xqi4ox7ui4r5cqg/8bit-openbayer.patch?dl=0
(sorry, our forum does not allow attachments to topic comment, it is easier to put it on dropbox than re-tune our forum).

This full patch also patches openbayer_sample.cpp to get correct parameters to this call.

-- Alex Tutubalin @LibRaw LLC

Success

Submitted by Falka on

Just rebuilt raw.dll (only ended up having to edit dcraw_common.cpp, thanks for the tips), and openbayer_sample() works with the patch! Thank you so much for your fast help. This is awesome!

I have found that to get what I want from LibRaw, I need to set no_auto_bright = 0 for 8bit images, and no_auto_bright = 1 for 16bit images. This is no problem to me, but the behavior is a little bizarre. The rest of this comment is to help you figure out if this is an unintentional error within LibRaw.

.............................................................................................................................

I do some precise exposure compensation and color correction before using LibRaw. After some testing, I have found a bizarre result when I set imgdata.params.no_auto_bright:
__________________|___8bit image____|__16bit image___
no_auto_bright = 0 |___100% correct___|__Earlier exposure / color adjustments are ruined
no_auto_bright = 1 |_Completely black_|_100% correct___

(By the way, when I say "100% correct," I mean that my earlier exposure adjustments were conserved :) )

The results for 16bit images in the table above make perfect sense.
The results for 8bit images don't make any sense to me! I would expect results exactly like 16bit. Instead, in 8bit, no_auto_bright = 0 behaves like I would expect no_auto_bright = 1 to behave, and no_auto_bright = 1 makes the entire image all-black no matter what.

.............................................................................................................................

Hope that was helpful for moving on with Release 0.19!

These results are expected

Submitted by lexa on

These results are expected (more or less).

auto brightness calculates image histogram, than adjust (increase) values to fit full 16-bit range (with 1% of pixels at saturation with default auto_bright_thr). So, if you use linear exposure correction (w/o highlights compression), than your changes will be ruined (indeed), because brightness is adjusted to use full data range.
Do not know what 'color adjustment' you use,because no such mechanics exists in dcraw_process() options (excluding white balance).

Without auto-bright, 8-bit images use only 0..255 range (to be correct,it may be more if white balance is used). This is 8 stop below full range (0..65535), so resulting image will look 'near black' (not completely black).

-- Alex Tutubalin @LibRaw LLC

Oh, they're non-dcraw adjustments which I coded.

Submitted by Falka on

I was serious that I do color, whitebalance, exposure, etc corrections to the raw data before putting it into LibRaw. I wrote the adjustments, and only use LibRaw to demosaic. No dcraw or LibRaw emulation of dcraw involved for color-correction / whitebalance / etc.

I wish it were the case that the resulting 8bit images without auto_bright were 'near black,' but take a histogram of this result and see for yourself:
https://drive.google.com/file/d/0B41mEzFmIzubTkhfSk9zMmtVVnc/view?usp=sh...

(Output with output_bps = 8, no_auto_bright = 1, no_auto_scale = 1, user_black = 0.) Every pixel is 0, and this was an overexposed 8bit raw. Do what you will with this information; I was just trying to repay a favor, thinking I might have found unintended behavior. I see now that LibRaw uses 16bit internally, even if the input buffer was only 8 BPP.

So, what seems to be happening...
1. 8 BPP data in the buffer has max value of 255
2. The buffer is read into LibRaw, which uses 16 BPP data internally
3. no_auto_bright = 1, so values are never scaled up to 65535
4. output_bps = 8, so values are all divided by 257 to scale down to 255 maxval... but at most, we had (255 / 257), which with integer rounding down equals 0. Anything less than 255 will of course also become 0.
5. Every pixel is 0 in the resulting image!

If that's correct (?), let this post just be a warning to any 8bit - LibRaw users who read this!

Meanwhile, I could not

Submitted by lexa on

Meanwhile, I could not confirm the problem.

Here is the changes made for openbayer_sample.cpp:
--- a/samples/openbayer_sample.cpp
+++ b/samples/openbayer_sample.cpp
@@ -45,7 +45,9 @@ int main(int ac, char *av[])
return 3;
LibRaw rp;
rp.imgdata.params.output_tiff = 1;
- int ret = rp.open_bayer(buffer, fsz, 640, 480, 0, 0, 0, 0, 0, LIBRAW_OPENBAYER_RGGB, 0, 0, 1400);
+ rp.imgdata.params.no_auto_bright = 1;
+ //int ret = rp.open_bayer(buffer, fsz, 640, 480, 0, 0, 0, 0, 0, LIBRAW_OPENBAYER_RGGB, 0, 0, 1400);
+ int ret = rp.open_bayer(buffer, fsz,4112,3008,0,0,0,0,0,LIBRAW_OPENBAYER_RGGB,0,0,0);
if (ret != LIBRAW_SUCCESS)
return 4;
if ((ret = rp.unpack()) != LIBRAW_SUCCESS)

(no_auto_bright=1 added to default settings).

Here is TIFF produced from your sample (screenshot in viewer): https://www.dropbox.com/s/alm7jqumy7fmf12/Screenshot%202017-08-02%2019.1...

-- Alex Tutubalin @LibRaw LLC