First, thanks very much for LibRaw, it is really helping me.
I'm pulling images out of files with many (many!) images,
and I've reached a strange limit. After exactly 486 frames (images),
I get a corrupted data exception. I've tried starting at the 400th
frame and I still get stopped at frame 486. The source file is
a cine file with frames of 2048 * 1080 16bit * RBG images in it.
I don't think that this creates any problem with an lseek, but I could be
wrong.
I've tried to see where the corrupted data exception is thrown,
but the code is err, somewhat opaque :-).
Any clues anyone?
thanks in advance
Mike K
It operating system dependent
After much experiment, I tried the same code on an Ubuntu 9.04 box, and it works perfectly. Backporting the same code back to OS X gives me the original problem.
Finally, I just tried dcraw_emu from the samples directory, and that suffers the
same problem, so I've eliminated my application code.
Any further input welcome...
File descriptor or memory leak?
Could you please describe how to repeat the problem with dcraw_emu? What command-line switches used?
-- Alex Tutubalin @LibRaw LLC
OS X Issues
I'm afraid it may not be very easy to repeat, as the file I'm using
is a .cine file of around 16G! I know this sounds an extreme case, but
some professional movie cameras shoot raw...
I didn't use any special switches, just something of the form of
dcraw_emu -s 700 mytestfile.cine
I'm afraid that I'm going to be very busy for the next couple of days, but
a good experiment I might perform would be to create a file with very small frames, so I
can find out if the frame count or the file size is the issue. That might
also create a test case that can be loaded via the Internet.
Mike
You say '16Gb' ? For many
You say '16Gb' ?
For many file types data_offset of frame data is read as 32-bit integer (via get4()). So, maximum frame offset is 2Gb.
To check this I need some sample with several frames. I don't need full 16-Gb file multi-multi-frame file, but file with 3-4 frames will be enough.
-- Alex Tutubalin @LibRaw LLC
Test File
Hi, many apologies for the long delay, I got food poisoning after the last post,
and I've been trying to catch up ever since.
I've put a test file on http://www.modern-industry.com/test.cine Its about 49MB.
Mike
Any progress?
gentle ping....
M
Oh, sorry. I've downloaded
Oh, sorry.
I've downloaded this file several weeks ago, than make LibRaw 0.8-b4, than forgot about file.
Will work with it tomorrow.
-- Alex Tutubalin @LibRaw LLC
There is two news, good one
There is two news, good one and not-so-good :)
1. Frame offset is stored as 64 bit in this format, so there is no *theoretical* problems.
2. There is two possible sources of problems under Mac OS X:
a) one possible source is 32-bit off_t type (i.e. some special compilation flags are needed because 64-bit offsets are possible)
b) FILE * interface under Mac OS X does not support large files.
I've MacBook with MacOS 10.5 on hands, so will continue investigations. Hope, I'll find problem source today.
-- Alex Tutubalin @LibRaw LLC
Problem found!
The problem is in MacOS X fseek() call. It takes 32-bit offset (long int). There is fseeko() function which takes off_t offset and so will work with files larger than 2Gb.
The quick fix:
in libraw/libraw_datastream.h change:
return substream?substream->seek(o,whence):fseek(f,o,whence);
to
return substream?substream->seek(o,whence):fseeko(f,o,whence);
This will break compatibility with Windows (no fseeko) and, possible, Linux. I'll prepare more universal solution (with #ifdef) today or tomorrow (to appear in 0.8-Beta5)
-- Alex Tutubalin @LibRaw LLC