I have a collection of raw files that LibRaw now processes without many issues.
There are two types of files that still have issues, only one of which is probably important.
The first one is apparently "Nucore" (extension .bmq) images which used to be supported but for some reason support was removed from dcraw. Considering the source of the raw images that I got for testing, I find this somewhat interesting that a supported format would be withdrawn. However, I did dig around and found a comment in the dcraw RC history that indicates the removal was intentional. So I'm not going to worry too much about this, at least right now.
The second group of pictures is some Kodak images fail with -100008 errors. I did some tracing of this and it would seem that the variable data_size isn't set (sometimes?) when processing a Kodak DC120 JPEG image. I am building the LibRaw source with JPEG enabled and using the turbojpeg library in the same build for regular JPEG image processing. So it seemed like a simple thing to have LibRaw connect up to that. Except the critical data_size variable isn't being set.
Failing Kodak pictures: http://remote.infinadyne.com:5000/fbsharing/zSAK7Z3Y
Please email me directly (1st 4 characters of user name) @ infinadyne.com and I can be helpful in providing a link to download a somewhat dated collection of raw images for testing purposes. It is not my intent to make this resource public but interested parties are free to contact me about it.
data_size is not used in
data_size is not used in kodak_jpeg_load_raw(), so it is not critical.
All Kodak files from your link are processed OK by our FastRawViewer and RawDigger (LibRaw used internally).
-- Alex Tutubalin @LibRaw LLC
We must be looking at different code
What I am seeing at the beginning of kodak_jpeg_load_raw is the following:
// LibRaw's Kodak_jpeg_load_raw void CLASS kodak_jpeg_load_raw() { if(data_size < 1) throw LIBRAW_EXCEPTION_DECODE_JPEG;This is the source of the -100008 error I am getting. This is in dcraw_common.cpp.
I am building this with NO_JPEG not defined.
Paul Crowley
Sorry. looked under wrong
Sorry. looked under wrong #ifdef.
And (looked in RawDigger in debugger) data_size is correct. Looks like I need to switch to 32-bit version again?
-- Alex Tutubalin @LibRaw LLC
And 32-bit version works fine
And 32-bit version works fine too: https://www.dropbox.com/s/4azsj2qecpl4af9/Screenshot%202015-11-10%2019.1...
-- Alex Tutubalin @LibRaw LLC
I don't understand ...
I have looked around at dcraw_common.ccp and I find no instance of setting data_size... ever. I looked at the latest dcraw.c source and I see the LibRaw version is doing something interesting in copying the entire JPEG data area and byte-flipping it whereas this is apparently done in the input buffer in dcraw.c. I can certainly understand a need/desire to keep the input buffer as "const", even at the expense of a huge malloc to hold the copy.
But this doesn't explain how data_size (which controls the malloc and byte swapping) gets set in things like RawDigger.
I put in some code to set data size at the beginning of kodak_jpeg_load_raw and it works flawlessly now. This isn't something I am comfortable with. I would very much like to understand how data_size is supposed to be set, given that it is buried in an "internal" struct, I am sure it is not an external parameter. Clearly, data_size is completely new to LibRaw and doesn't appear in the original dcraw.c source. But when I search on data_size in dcraw_common.cpp the only occurrences I find are reading it. Actually, I find no occurrence anywhere in LibRaw where this value is set.
What am I missing here?
Paul Crowley
data_size is set within apply
data_size is set within apply_tiff() code.
malloc is not 'huge': Kodak JPEG-codec cameras are DC50 (0.5Mpix) and DC120 (1Mpix). Real data size is less than 1Mb.
-- Alex Tutubalin @LibRaw LLC
I guess my history is showing...
These days, a malloc under 1MB isn't huge at all.
Working on embedded systems and games with as little as 128 BYTES of RAM conditions one in different ways and makes one think a bit before doing some things.
I can see where it would be good to set data_size in apply_tiff, but in the code I downloaded from libraw.org it isn't happening. Seems like the statement
would be a good idea to have there, but in the 0.17 code it is not there.
Paul Crowley
OK, mystery solved. I use
OK, mystery solved. I use current (our development) branch for testing.
This branch to be pushed on GitHub within 2-3 weeks, our repository is here.
Or just add these lines to apply_tiff:
data_offset = tiff_ifd[i].offset; +#ifdef LIBRAW_LIBRARY_BUILD + data_size = tiff_ifd[i].bytes; +#endif tiff_flip = tiff_ifd[i].t_flip;1Mb malloc is not much in today raw processing. 50Mpix Canon RAW, processed in full-res in float (16 bytes per pixel) is slightly less than 1Gb (and my phone have 2GB RAM)
-- Alex Tutubalin @LibRaw LLC
Checked all old kodak files
Checked all old kodak files with current (-development) LibRaw branch with both open_file() and open_buffer(). All OK.
development branch to be updated on GitHub in few days
-- Alex Tutubalin @LibRaw LLC
Great - I will look for it
I will take a look at this, but I am probably not going to pick up the next release right away. I got into this because my dcraw "adaptation" wasn't working as well as it should and I thought LibRaw might save me a lot of work. It certainly has, because all in all, dcraw is a monolithic global-variable mess. It might sort of be OK for a stand-alone program, but I am not even sure of that. I do know that it is very difficult for someone to maintain the way it is.
So I have my little graphics interface DLL now working with the updated dcraw code and this brings it into the more "modern era". I have also brought in a faster JPEG decoder and I believe this will benefit the applications that use this DLL greatly. So now I get to spend a bunch of time working on those applications with a lot more confidence in the graphics DLL.
Paul Crowley