I was trying to understand how an error could be being returned and not being caught. Turns out, I have callbacks turned off (as in LIBRAW_OPIONS_NO_MEMERR_CALLBACK | LIBRAW_OPIONS_NO_DATAERR_CALLBACK) and this error is being reported with the dcraw_emu program via a callback!
So, it would seem that this error is not being reported or returned when the callback is turned off. That is an important note as to the importance of the callbacks which I did not understand originally.
Turns out for the image in question "derror()" is being called from canon_load_raw(). Not exactly clear what is going on with this, but it seems like it is a problem with pixel values. What isn't at all clear is how the image is being processed correctly when this is not asserted.
I am not so sure I am missing something in my use of libraw any longer, other than it should probably pick up data error callbacks and treat the image as unsupported if this occurs.
Well, that is the problem. No error codes. It is difficult to describe the contents of the image, but what I am getting is some fairly random noise.
Here is a link to everything: https://dl.dropboxusercontent.com/u/43288013/Photos.zip
There are three files: 600_0001.CRW (original), 600_0001.CRW.ppm (what it is supposed to show as) and 600_0001.CRW.jpg (what I am seeing).
It seems most of the failures are .CRW (Canon) images, at least so far. Some of the .CRW are working and .NEF is working fine for all of them.
I will see what I can do with the prebuilt binary for dcraw_emu and experiment with the -mem switch. My use of the library has the image loaded into memory already which is why I am using open_buffer. Basically where I am in this is that I took an older dcraw-conversion and ripped it out replacing it with libraw. It was mostly working before (and did work with .CRW files) but it was a dcraw from around 2009. I wanted to pick up more current camera support. Two weeks into the project... here I am.
LibRaw's internal/* files are generated from (patched) dcraw.c source.
If the warnings are from this code (dcraw_common.cpp, dcraw_fileio.cpp), your changes will *not* be merged into main LibRaw source because changing of these files will broke dcraw auto-import.
Thank you for clarification, that makes sense now. I'm using LibRaw under Mac with Xcode/clang and have lots of warnings (>150), so I'm considering to fix them or not. It is worth doing only if fixes are mainlined somehow.
0.17 is latest public code.
We do not push 'work in progress' code to github, but use own private repo for it. Only big pieces, known to work (in our RawDigger/FastRawViewer) are pushed.
(I answer anyway: this comments are public and may be helpful for other users)
LibRaw extract full visible area, 5202x3465 pixels. Canon's suggestion of smaller area, '5184x3456', is ignored.
Probably, Canon's suggestion is smaller to make some room for demosaic, do not know.
Can you think of any other ways I could attempt to get around this this with LibRaw? My application is running on a Rasberry Pi 2. Unfortunately my processing pipeline seems to be bottlenecked by the time it takes to write to the Pi's SD card. Despite having the fasted SD card available, writing is quite slow with the Pi 2. With 16bit TIFF's and then having to write out another PNG or JPEG to disk I think the cumulative effect is adding roughly 25-30% extra processing time to each photo.
Since my application converts 100s of pictures per job this can add quite few more minutes for each capture session. Perhaps I need to look in to modifying one of the source code examples and trying to figure out how to modify the code to do this myself. Do you think this would be possible? I'm not an experienced C++ programmer though and so any pointers would be most welcome. If you think it's a non-trivial task then perhaps I'd be best to just live with the extra time this steps takes.
LibRaw is library, it is not targeted to be full dcraw replacement. So, there is no such thing as 'document mode' (-D or -d in dcraw), programmer have full access to unprocessed raw data, but no ready samples for direct emulation of dcraw -D (may be unprocessed_raw sample will solve your problem)
Second one (-o1 is sRGB output, so the default) is emulated by dcraw_emu -T
Ok. Thank you for your advice. After the first feedback I hacked a small batch processing for preview images which works quite well. Even if I wondered that there is no Cam_XYZ populated with Blackmagic Production Camera raw files. But this should be another thread maybe later.
Pixels outside of image area are not image area (in most cases).
Unlike Adobe tools (e.g. DNG Converter), LibRaw tries to save as much image area as possible, without any space for demosaic to run. So, you need to crop slightly more to get demosaic some space.
There is small exclusion from 'as much as possible rule': border size is usually even, to keep bayer patern the same from both edges (uncropped and 'active area' cropped). This is not general rule, but I hold on to it when image format do not dictates other borders (such as DNG ActiveArea or so)
thank you for your reply Alex! Cleared up some stuff and I'm somehow happy that unpack already applies the curve.
Regarding 1): How much can be relied on the pixels just outside of the active pixel area? Most algorithms I have currently implemented need 2-3 pixels on the outside to get the pixels at the edge. Therefore I currently copy the 2 rows/cols at the edge to the outside but would like to save the processing time of this step. Or is the usability of this pixels mostly depending on the raw format?
1)Yes, raw_image is raw_width*raw_height 1D array. It is uncropped, full image (including 'masked areas') is read by unpack().
Black level is not subtracted, phase one postprocessing is not performed, but linearization curve is applied to raw_image data (see below)
2) Yes, curve is populated on the early stage of open_file() with curve[i] = i; (for full range loop, from 0 to 65535)
After open_file(), the curve[] will contain linearization curve read from file metadata.
Unfortunately, some formats (e.g. Nikon NEF) store curve not in metadata, but in raw image data itself. For these formats curve is read on unpack() stage.
curve[] size is not set anywhere: curve usage is specific for specific data format/unpacker code, curve is applied at unpack() stage.
Note: same curve[] array is used for gamma correction of output.
3) Use LibRaw::COLOR(row,col) call to get color at row,col.
Please note, that row,col are 'visible area' (cropped) coordinates (i.e. uncropped at top_margin/left_margin is COLOR(0,0))
callbacks are optional.
You'll receive error code for call and, also, bits in imgdata.process_warnings for non-fatal errors (data error is not fatal).
BTW, the image is indeed broken, but raw data in place. Still investigating.
I was trying to understand how an error could be being returned and not being caught. Turns out, I have callbacks turned off (as in LIBRAW_OPIONS_NO_MEMERR_CALLBACK | LIBRAW_OPIONS_NO_DATAERR_CALLBACK) and this error is being reported with the dcraw_emu program via a callback!
So, it would seem that this error is not being reported or returned when the callback is turned off. That is an important note as to the importance of the callbacks which I did not understand originally.
Turns out for the image in question "derror()" is being called from canon_load_raw(). Not exactly clear what is going on with this, but it seems like it is a problem with pixel values. What isn't at all clear is how the image is being processed correctly when this is not asserted.
I am not so sure I am missing something in my use of libraw any longer, other than it should probably pick up data error callbacks and treat the image as unsupported if this occurs.
More likely, it means that some of internal tags pointed outside of data buffer.
This is not an error for file case (seek out of file will not read anything, so tag value is ignored), but will raise error in buffer case.
BTW, it means you *have* error code at unpack()
Thank you for file sample, to be investigated in depth.
dcraw_emu (prebuilt) says "unknown file: data corrupted at 127149" with the -mem switch. Without it, works fine.
Not sure what this means, but it is different.
Well, that is the problem. No error codes. It is difficult to describe the contents of the image, but what I am getting is some fairly random noise.
Here is a link to everything: https://dl.dropboxusercontent.com/u/43288013/Photos.zip
There are three files: 600_0001.CRW (original), 600_0001.CRW.ppm (what it is supposed to show as) and 600_0001.CRW.jpg (what I am seeing).
It seems most of the failures are .CRW (Canon) images, at least so far. Some of the .CRW are working and .NEF is working fine for all of them.
I will see what I can do with the prebuilt binary for dcraw_emu and experiment with the -mem switch. My use of the library has the image loaded into memory already which is why I am using open_buffer. Basically where I am in this is that I took an older dcraw-conversion and ripped it out replacing it with libraw. It was mostly working before (and did work with .CRW files) but it was a dcraw from around 2009. I wanted to pick up more current camera support. Two weeks into the project... here I am.
Could you please explain what is 'not working':
- error codes (on which call exactly, what code)
- error codes are OK, but resulting image is broken?
Also, dcraw_emu has -mem command-line switch to test memory I/O instead of File I/O. It it working for your case, or not?
Then I think it's not worth it :)
LibRaw's internal/* files are generated from (patched) dcraw.c source.
If the warnings are from this code (dcraw_common.cpp, dcraw_fileio.cpp), your changes will *not* be merged into main LibRaw source because changing of these files will broke dcraw auto-import.
Thank you for clarification, that makes sense now. I'm using LibRaw under Mac with Xcode/clang and have lots of warnings (>150), so I'm considering to fix them or not. It is worth doing only if fixes are mainlined somehow.
0.17 is latest public code.
We do not push 'work in progress' code to github, but use own private repo for it. Only big pieces, known to work (in our RawDigger/FastRawViewer) are pushed.
Expect big github update in 2-3 weeks or so.
(I answer anyway: this comments are public and may be helpful for other users)
LibRaw extract full visible area, 5202x3465 pixels. Canon's suggestion of smaller area, '5184x3456', is ignored.
Probably, Canon's suggestion is smaller to make some room for demosaic, do not know.
I think I have it figured out. Thanks.
So is (50,158) in 5202 x 3465 the same pixel as (56,168)
Thanks
Canon stores sensor active area in Makernotes (vendor specific), not in EXIF fields.
LibRaw do not use this data, so for Canon 7D active area (reported by LibRaw) is 5202x3465, while left-top corner is at 50,158
LibRaw::dcraw_ppm_tiff_writer(char* fn) accepts filename as parameter.
It looks like you need to create own image writing code with ability to write to stdout
Thanks for replying so quickly Alex.
Can you think of any other ways I could attempt to get around this this with LibRaw? My application is running on a Rasberry Pi 2. Unfortunately my processing pipeline seems to be bottlenecked by the time it takes to write to the Pi's SD card. Despite having the fasted SD card available, writing is quite slow with the Pi 2. With 16bit TIFF's and then having to write out another PNG or JPEG to disk I think the cumulative effect is adding roughly 25-30% extra processing time to each photo.
Since my application converts 100s of pictures per job this can add quite few more minutes for each capture session. Perhaps I need to look in to modifying one of the source code examples and trying to figure out how to modify the code to do this myself. Do you think this would be possible? I'm not an experienced C++ programmer though and so any pointers would be most welcome. If you think it's a non-trivial task then perhaps I'd be best to just live with the extra time this steps takes.
LibRaw is library, it is not targeted to be full dcraw replacement. So, there is no such thing as 'document mode' (-D or -d in dcraw), programmer have full access to unprocessed raw data, but no ready samples for direct emulation of dcraw -D (may be unprocessed_raw sample will solve your problem)
Second one (-o1 is sRGB output, so the default) is emulated by dcraw_emu -T
dcraw_emu only writes to files now, so you need to create temporary .tif file
use cmatrix (this is camera raw values to output rgb space matrix) if you need to use file built-in color profile
cam_xyz is filled in by internal LibRaw color profile, but no such profile is set for BlackMagic cameras.
Ok. Thank you for your advice. After the first feedback I hacked a small batch processing for preview images which works quite well. Even if I wondered that there is no Cam_XYZ populated with Blackmagic Production Camera raw files. But this should be another thread maybe later.
Pixels outside of image area are not image area (in most cases).
Unlike Adobe tools (e.g. DNG Converter), LibRaw tries to save as much image area as possible, without any space for demosaic to run. So, you need to crop slightly more to get demosaic some space.
There is small exclusion from 'as much as possible rule': border size is usually even, to keep bayer patern the same from both edges (uncropped and 'active area' cropped). This is not general rule, but I hold on to it when image format do not dictates other borders (such as DNG ActiveArea or so)
Hey,
thank you for your reply Alex! Cleared up some stuff and I'm somehow happy that unpack already applies the curve.
Regarding 1): How much can be relied on the pixels just outside of the active pixel area? Most algorithms I have currently implemented need 2-3 pixels on the outside to get the pixels at the edge. Therefore I currently copy the 2 rows/cols at the edge to the outside but would like to save the processing time of this step. Or is the usability of this pixels mostly depending on the raw format?
1)Yes, raw_image is raw_width*raw_height 1D array. It is uncropped, full image (including 'masked areas') is read by unpack().
Black level is not subtracted, phase one postprocessing is not performed, but linearization curve is applied to raw_image data (see below)
2) Yes, curve is populated on the early stage of open_file() with curve[i] = i; (for full range loop, from 0 to 65535)
After open_file(), the curve[] will contain linearization curve read from file metadata.
Unfortunately, some formats (e.g. Nikon NEF) store curve not in metadata, but in raw image data itself. For these formats curve is read on unpack() stage.
curve[] size is not set anywhere: curve usage is specific for specific data format/unpacker code, curve is applied at unpack() stage.
Note: same curve[] array is used for gamma correction of output.
3) Use LibRaw::COLOR(row,col) call to get color at row,col.
Please note, that row,col are 'visible area' (cropped) coordinates (i.e. uncropped at top_margin/left_margin is COLOR(0,0))
Being very similar to Adobe is surely not bad at all! ;)
LibRaw's embedded color profile (usually very similar to Adobe's) may differ from camera provided color data, so result difference is expected.
Pages