How to compile LibRaw with RawSpeed?

Submitted by Kuro on

I've been using LibRaw 0.18.3 with success.

Now I want to add support for RawSpeed. I read that FastRawViewer is using RawSpeed. So how do I get RawSpeed support in LibRaw 0.18.3?

I've been trying all weekend without success. I can get it to compile, but in the linking stage, it always comes back with error on "-lrawspeed". Since RawSpeed is not a library, I'm not sure why the Makefile.dist has this as an option?

I'm using RawSpeed at the master branch.

Can anyone help?

Forums: 

1) You need to compile

Submitted by lexa on

1) You need to compile rawspeed as a library (this is not a problem, just make library from object files)
2) You may need to patch RawSpeed using provided patches (in LibRaw's rawspeed folder)
to provide Dll export (for Windows/DLL) and rawspeed_get_number_of_processor_cores() call
3) You may used qmake .pro file (provided with LibRaw's patches) to build RawSpeed.a/.dll

-- Alex Tutubalin @LibRaw LLC

Can you update the file

Submitted by Kuro on

Can you update the file README.RawSpeed.txt? It is out of date and I've been trying all weekend using the instructions there without success.

Compiling RawSpeed to a library is a problem, because there is no makefile or configure file. I'm trying to do this under macOS.

Kuro

Although README.RawSpeed.txt

Submitted by lexa on

Although README.RawSpeed.txt does not look fresh, RawSpeed master branch was not updated from 2014 too.

-- Alex Tutubalin @LibRaw LLC

RawSpeed team is focused

Submitted by lexa on

RawSpeed team is focused mostly on code rewrite and cleanup and on correct handling of corner cases (e.g. fuzzer testing, etc). This is very important work, but in reality users of RAW processing software usually feeds it by files created by cameras, but not random crap generated by fuzzer.

On the other side, the real issue with real files created by real camera is not fixed for 10 months: https://github.com/darktable-org/rawspeed/issues/100 (is this library really *maintained* ?)

So, I'm somewhat skeptical about current RawSpeed state. It has very strong features (e.g. LJPEG decoder is really ~30% faster on Canon 5Ds files, this is vital for apps build for speed /like our FastRawViewer/), but focusing on 'pure software development process' (code cleanup, modern C++ rewrite, fuzzer testing) instead of handling real issues is, at least, questionable from my point of view.

Most likely, we'll support 'RawSpeed devel' (do not know exact version is it v3 or v4 now) in LibRaw after 0.19 release (so, this year).

Very probable, that support will use some 'whitelisting', so only known/tested camera files to be passed to RawSpeed decoder (and, sure, this list will be empty by default).

P.S. RawSpeed has imported our (LibRaw) compressed fuji decoder and provided feedback was very useful, we fixed two (or so) corner cases based on it.

-- Alex Tutubalin @LibRaw LLC

it seems this issue 100 had

Submitted by Nichola (not verified) on

it seems this issue 100 had been closed. Do you have any progress on new rawspeed lib?

Didn't even start.

Submitted by lexa on

Didn't even start.

-- Alex Tutubalin @LibRaw LLC

After a lot of trials and

Submitted by Kuro on

After a lot of trials and errors by following your instructions above, I've finally managed to compile RawSpeed into a library and recompiled libraw to use it.

I can see a speedup of 2x with Canon CR2 files, but not so much for NEF, and I think there is no support for X-Trans.

Kuro

I just experienced crashes

Submitted by Kuro on

I just experienced crashes when I decoded DNG and Olympus ORF files with RawSpeed turned on.

I've installed the rawspeed.uncompressed-color-dng.patch. So what else am I missing?

Kuro

Could you please provide

Submitted by lexa on

Could you please provide sample DNG to look into it?

-- Alex Tutubalin @LibRaw LLC

I reverted the patch for

Submitted by Kuro on

I reverted the patch for rawspeed.uncompressed-color-dng.patch, then RawSpeed no longer crashes on DNG files.

But now it crashes on decoding Olympus ORF files. So something is definitely wrong...

Kuro

After half a day of playing

Submitted by Kuro on

After half a day of playing around with it, I still get a crash whenever a DNG file is decoded.

Below is how I compile librawspeed and libraw (this is under macOS):

For RawSpeed:
- Unpack the rawspeed-master.zip file
- Copy it to the same folder that stores LibRaw-x.xx.x
- Rename it to RawSpeed
- Copy the patch files from LibRaw/RawSpeed to RawSpeed/RawSpeed directory
- Run patch < rawspeed.cpucount-unix.patch
- Edit Common.cpp and cut and paste the the function int rawspeed_get_number_of_processor_cores() to the end of the file, outside of the #elif
- Run patch < rawspeed.qmake-pro-files.patch
- Run qmake -makefile rawspeed-lib.pro
- Edit DngDecoderSlices, find the line with jpeg_read_header(&info, TRUE). Cast TRUE with (boolean)
- Edit RawDecoderThread.h, change 'uint32 taskNo' to 'int32 taskNo'
- Edit the Makefile. Change -mmacosx-version-min= to the version you want (such as 10.9)
- Run make

For LibRaw:
- Edit Makefile.dist
- Uncomment the two lines under '# RawSpeed Support'
- Add -mmacosx-version-min=xx.x to CFLAGS+=
- Uncomment the lines 'Demosaic Pack GPL2 and GPL3'
- Run make -f Makefile.dist

There were some warnings when compiling RawSpeed. The most notable one is bit shifting a negative number. This is undefined by the gcc compiler. This warning is in LJpegDecompressor.cpp. I tried changing -32768 to 0b1000000000000000, but the crash persists.

I found that sometime I can decode the DNG file and have it displayed on the screen. But my app crashes just by sitting there idle for a few seconds. Other times, my app crashes right at the line "rawProcessor.unpack()". It seems that the code in RawSpeed in decoding DNG file is corrupting some memory locations, and when some background garbage collection occurs, the app crashed.

As soon as I set 'imgdata.params.use_rawspeed = 0', DNG files are decoded correctly without any crash at all.

Perhaps you have fixed something in the master branch of RawSpeed to allow successful decoding of DNG files?

Kuro

LibRaw team is not

Submitted by lexa on

LibRaw team is not responsible for RawSpeed bugs.

Meanwhile, if you provide sample file that crashes RawSpeed I'll take a look (under Windows/MSVC 2010 because it my main development platform).

-- Alex Tutubalin @LibRaw LLC

I do not see any problems

Submitted by lexa on

I do not see any problems with this file. Tested with RawDigger under both Windows and OS X.

-- Alex Tutubalin @LibRaw LLC

I don't think it is the file,

Submitted by Kuro on

I don't think it is the file, because when use_rawspeed = 0 , LibRaw has no problem loading this file.

Would you be so kindly post links to the libraw_r.a and librawspeed.a compiled for macOS so I can download them to try out? PM me if it is not convenient to do it in the forum post.

Thanks!

Kuro

This will result into full

Submitted by lexa on

This will result into full build infrastructure sharing:
- other libraries used (jpeg, xml2)
- complete build environment share to make sure your app compiled with same compiler/same settings to ensure same offsets in libraw structures

Looks impossible.

-- Alex Tutubalin @LibRaw LLC

I finally found the culprit

Submitted by Kuro on

I finally found the culprit of the crash: the thread executing LibRaw ran out of stack space! I observed LibRaw crashing internally when I single step into the actual decoder for DNG.

By replacing "LibRaw rawProcessor;" with "LibRaw *rawProcessor = new LibRaw;", the crash went away :)

Kuro

Glad to hear.

Submitted by lexa on

Glad to hear.

Yes, LibRaw object is large

-- Alex Tutubalin @LibRaw LLC

if rawspeed has decoded the

Submitted by lexa on

if rawspeed has decoded the file, this bit is turned on:

imgdata.process_warnings |= LIBRAW_WARN_RAWSPEED_PROCESSED;

(this is not warning, just info)

-- Alex Tutubalin @LibRaw LLC