Libraw and file descriptors

Submitted by amandra on

With recent changes to Android, simple files are an endangered species and we have to rely on file descriptors. My c++ is rudimentary at best, but if I follow the docs and code to open a file descriptor I'd need to use:

LibRaw::open_datastream

I'd pass in an extension of LibRaw_abstract_datastream that would very closely resemble the existing:

LibRaw_file_datastream and LibRaw_bigfile_datastream but for a FILE*.

And I'd replicate the logic to choose which stream that exists in LibRaw::open_file before calling LibRaw::open_datastream.

Does this sound logical?

Forums: 

LibRaw_file_datastream is

Submitted by lexa on

LibRaw_file_datastream is based on С++ iostreams, while bigfile_datastream uses old plain FILE*.

Iostreams implementation is slightly faster (under Win32, at least), but there is no way to limit file buffer size. Also, iostreams implementation is limited to 2GB (or 4Gb??) files on some systems because of 32-bit nature.

Most raw files are much smaller than 2/4GB (and usually fit into 50-100M with very few exceptions), so (faster) iostream implementation is used. Bigfile_datastream is used for larger files to avoid too large buffers and 2/4Gb limit. The only files that go over limit are cinema files (Red's R3D files and other multi-frame files).

-- Alex Tutubalin @LibRaw LLC

Thanks for the extremely fast

Submitted by amandra on

Thanks for the extremely fast response. I completely missed the difference in the subfile_open of the two streams in my first glance. So it should just be a simple exercise of replicating big with fdopen.

If I really want to strive for the speed in stream I could see if Android supports any method of FILE* to fstream. How much of an improvement do you think there is? It's Android, so linux, if the improvement only lies in Win.

Thanks again!

I do not think you'll see any

Submitted by lexa on

I do not think you'll see any real difference in speed for compressed RAW files.

-- Alex Tutubalin @LibRaw LLC

It looks like their are quite

Submitted by amandra on

It looks like their are quite a few references to file path throughout the code despite having the file stream.

Long story short, will I be able to create a data stream that does not have access to a true file path? I should be able to get a file name, but unfortunately that's the extent. If it's not possible the rest of this post is irrelevant, just analysis of the impact of file path in the code. If it is possible, if you don't mind elaborating on the impact it would help me greatly. Thanks.

1) I noted a few line directives from dcraw.c. Do I need to be concerned about its use of file path or just within libraw?

LibRaw_abstract_datastream::subfile_open(const char *fn)
2) I only note this used in a parse_external_jpeg() which seems to be a hack to get metadata. It appears to be a warning at worse on a filename issue. Correct?
3) What is the significance of this function? It appears to open any file really, but has a thread lock. Correct?

#define ifname ((char*)libraw_internal_data.internal_data.input->fname())
4) This seems to be more of a concern, there are 17 references in dcraw_common, a lot in the same LibRaw_abstract_datastream::subfile_open(const char *fn) function and some related to minolta. There are many more references to its own similar 'ifname' in dcraw.c itself if I need to be concerned with that?

Yes, you may implement LibRaw

Submitted by lexa on

Yes, you may implement LibRaw datastream without filename. Look into LibRaw_buffer_datastream as a sample.
Your datastream->fname() should return NULL in this case. This should work OK for most RAW data with one exception:
- old (not current) 'CHDK hack' (or DIAG RAW hack, cannot remember now) creates RAW file without any metadata, just sensor dump.
- all metadata is stored in filename.JPG (same file, different extension).

dcraw (and than LibRaw) is able to parse filename.raw + filename.JPG pair to get metadata (exposure parameters, etc) from the second file.

Another story is make_jas_stream() (from Jasper JPEG2000 library). This call is needed only if you want to decode Red cinema file. Again, you may implement it as 'return 0'; and all things will work OK (exception LIBRAW_EXCEPTION_DECODE_JPEG2000 raised within LibRaw, then converted to return code in unpack())

-- Alex Tutubalin @LibRaw LLC

Followup:

Submitted by lexa on

Followup:

just checked for ifname:
- these 'names' are under #ifdef DCRAW_VERBOSE, or #ifndef LIBRAW_LIBRARY_BUILD. The only exlusion is parse_external_jpeg(), but also after if(!ifp->ifname())... return;

-- Alex Tutubalin @LibRaw LLC

BTW, for debug code try to

Submitted by lexa on

BTW, for debug code try to
- mmap file
- pass mmap to LibRaw_buffer_datastream.

It should work under Linux (Android). It is not working right under Windows because Win32 memory mapping is some strange thing.

-- Alex Tutubalin @LibRaw LLC

Once again thanks for the

Submitted by amandra on

Once again thanks for the instant response. My unix is as rusty as my c...had to look up mmap. Can't believe I didn't think to check the buffer stream for reference, face::palm.

For testing (it uses extra

Submitted by lexa on

For testing (it uses extra memory, so not mobile friendly):

char *buffer = malloc(filesize);
read(fd,buffer,filesize);
close(fd);
LibRaw::open_buffer(buffer,filesize);

should work.
And replace with your own LibRaw_datastream implementation later.

-- Alex Tutubalin @LibRaw LLC