Address sanitizer is still on. I need to find ways to disable it in the RawSpeed3 source code (not sure how right now).
Do you see speed improvements with RAWSpeed 3 in general? Is it faster than the original RAWSpeed or does it simply offer more RAW file decode for different camera brands?
1) Sorry, we do not offer advice on building RawSpeed3 as mentioned in the provided README
It is definitely possible to build it w/o address sanitizer.
We're not Cmake users, so I do not know what is 'target_sources(rawspeed_get_number_of_processor_cores....' message
2) LIBRAW_DECODER_TRYRAWSPEED3 bit is set in get_decoder_info() code. There is no way to change it from your app and no need for that.
LIBRAW_DECODER_TRYRAWSPEED3 bit is set only for decoders that are definitely faster in RawSpeed3, not for all decoders provided by the library (for example, it is mostly useless to decode unpacked/uncompressed data via RawSpeed3: it uses additional memory buffer for the decoded file with little or no benefit)
3) If you compile LibRaw with USE_RAWSPEED_BITS define, you need to set LIBRAW_RAWSPEEDV3_USE bit in imgdata.rawparams.use_rawspeed
If you compile LibRaw with just USE_RAWSPEED3 define, imgdata.rawparams.use_rawspeed=1 is enough.
Anyway, I've it figured out so it is now compiled correctly. There was an error in the 01. patch, it should remove 'final' from CameraMetaData, and not adding to it.
There is also another error in CMakeLists.txt in RawSpeed-v3/src/librawspeed/common:
target_sources(rawspeed_get_number_of_processor_cores should be set to PUBLIC not PRIVATE. Or else it libraw would not link (can't find rawspeed_get_number_of_processor_cores).
There is one addition error in converting the cameras.xml file to cameras.cpp file. A null character was added in the last line of the cpp file. It won't compile unless it is removed. The null char was preceding the last line "";
I also need to add -fsanitize=address compile option to Makefile.dist, as RawSpeed3 source code automatically detect and enable -fsanitize=address compile flag, and it'll not link unless the same is specified in libraw and my code.
Now that I got it to compile and link, how do I specify RawSpeed3 during runtime?
I set imgdata.rawparams.use_rawspeed = 1 and for all the RAW files I tried, it doesn't decode them with RawSpeed3, including Sony ARW files or Nikon Z camera RAW files.
The doc mentioned about:
RawSpeed-v3 support is controlled via libraw_decoder_info.t->flags (LIBRAW_DECODER_TRYRAWSPEED3 bit).
First of all, there is no libraw_decoder_info.t->flags, but there is libraw_decoder_info_t->decoder_flags.
I assume you're referring to this decoder_flags?
I can assign decoder_flags = LIBRAW_DECODER_TRYRAWSPEED3, but how do I pass the libraw_decoder_info_t data structure to the raw processor so that it knows I want it to try RawSpeed3?
There is a get_decoder_info() function, but not a set_decoder_info().
1.
To make a file with camera definitions, run the command (you can add it to the build system): sh ./rawspeed3_c_api/rsxml2c.sh < path/to/RawSpeed/data/cameras.xml > rawspeed3_c_api/cameras.cpp
The rsxml2c.sh script requires cat, tr, and sed unix command-line utilities installed, there is no specific version requirements.
2.
Add the resulting file (LibRaw/RawSpeed3/rawspeed3_c_api/cameras.cpp) and LibRaw/RawSpeed3/rawspeed3_c_api/rawspeed3_capi.cpp, to the build of the RawSpeed-v3 (dynamic) library, which you learned to build in the previous step. If building Windows DLL: rawspeed3_capi.cpp should be compiled with -DRAWSPEED_BUILDLIB to create dll export entries automatically.
3.
To check that everything works, you can use the LibRaw/RawSpeed3/rawspeed3_c_api/rawspeed3_capi_test.cpp test program, it should be possible to build it and link with RawSpeed-v3 (+ C-API); and after that this test program should be ready to decode RAW files passed to it from command line.
Please clarify what is not clear in the steps quoted above.
I'm not sure what you mean, but by linear behaviour I mean that if I take images of the same identical subject with the same illumination, but with different exposure times, and then plot pixel value VS exposure time, the curve is linear (if gamma is 1). In the plot I attached to the original post you can see this behaviour.
I see, thanks. As I said though, gamma is not really a problem. Even if I set gamma to 1 I do not see the linear behaviour that I'd like to get (https://imgur.com/a/a1C3cfV for just a few images), while I do get it with the DNG SDK and unprocessed_raw.
1) when using dcraw_emu, -g 1 0 doesn't seem to have an effect on top of -W. I actually don't have a problem with having a non-unitary gamma, as long as it is the same on all images. I can compensate for that later in the processing.
2) White Balance is not really a problem neither, I have a neutral area in the target I'm imaging and I will in any case perform white balance using that.
3) I did look at unprocessed_raw and it does give a linear output. However, it returns images before demosaicing, which I'd prefer not to have to code in myself, since I'd assume libRAW has significantly better functionality for this than I can code. I'd rather invest time into trying to get linear output from built-in functionality than having to code demosaicing.
So, what I'd like to find out is if there is something we can do to get rid of that extra processing that happens in dcraw_emu -W, but without going as far back as unprocessed_raw.
There you can find
1) DNG images from the camera
2) JPG images automatically generated by the camera (I use those to get exposure times from EXIF data)
3) TIF images obtained using the stage3 output of the Adobe DNG SDK
4) PPM images obtained using dcraw_emu.exe -W (which should be equivalent to what described in this comment https://www.libraw.org/comment/6511#comment-6511)
I like the TIF images because for half the exposure time I get half the pixel value, across the exposure range. The PPM images are not predictable, which implies that some extra processing is happening. Please see the original attachment for a plot of image average (across all pixels, all channels) VS exposure time.
Sorry, I didn’t realize right away that I was already talking with two people.
Could you please formulate your problem not referring to the 'OP'. Also, it would be great to have RAW file(s) in question еo discuss something in specific detail and not in the abstract.
Thank you.
Windows: use LibRaw::open_file(wchar_t *)
macOS: LibRaw::open_file(char*) accepts UTF-8 filenames.
Other operating system: do not know :), you probably may need to implement own LibRaw datastream.
Yes, this is expected speed-up for huffman compressed files.
Managed to turned off sanitizer and RawSpeed 3 is about 20-40+% faster than RawSpeed v1. Sony A7RM4 RAW decode speed is like 2x.
Huffman decoder is faster.
Address sanitizer is still on. I need to find ways to disable it in the RawSpeed3 source code (not sure how right now).
Do you see speed improvements with RAWSpeed 3 in general? Is it faster than the original RAWSpeed or does it simply offer more RAW file decode for different camera brands?
I'm also unsure. Do you still use address sanitizer?
I recompiled without USE_RAWSPEED_BITS and RawSpeed3 appeared to have been called when I decode a certain Sony ARW images.
But performance is terrible. Like 4.5x slower.
Not sure what is going on.
1) Sorry, we do not offer advice on building RawSpeed3 as mentioned in the provided README
It is definitely possible to build it w/o address sanitizer.
We're not Cmake users, so I do not know what is 'target_sources(rawspeed_get_number_of_processor_cores....' message
2) LIBRAW_DECODER_TRYRAWSPEED3 bit is set in get_decoder_info() code. There is no way to change it from your app and no need for that.
LIBRAW_DECODER_TRYRAWSPEED3 bit is set only for decoders that are definitely faster in RawSpeed3, not for all decoders provided by the library (for example, it is mostly useless to decode unpacked/uncompressed data via RawSpeed3: it uses additional memory buffer for the decoded file with little or no benefit)
3) If you compile LibRaw with USE_RAWSPEED_BITS define, you need to set LIBRAW_RAWSPEEDV3_USE bit in imgdata.rawparams.use_rawspeed
If you compile LibRaw with just USE_RAWSPEED3 define, imgdata.rawparams.use_rawspeed=1 is enough.
It was actually step 2 on how to add the files.
Anyway, I've it figured out so it is now compiled correctly. There was an error in the 01. patch, it should remove 'final' from CameraMetaData, and not adding to it.
There is also another error in CMakeLists.txt in RawSpeed-v3/src/librawspeed/common:
target_sources(rawspeed_get_number_of_processor_cores should be set to PUBLIC not PRIVATE. Or else it libraw would not link (can't find rawspeed_get_number_of_processor_cores).
There is one addition error in converting the cameras.xml file to cameras.cpp file. A null character was added in the last line of the cpp file. It won't compile unless it is removed. The null char was preceding the last line "";
I also need to add -fsanitize=address compile option to Makefile.dist, as RawSpeed3 source code automatically detect and enable -fsanitize=address compile flag, and it'll not link unless the same is specified in libraw and my code.
Now that I got it to compile and link, how do I specify RawSpeed3 during runtime?
I set imgdata.rawparams.use_rawspeed = 1 and for all the RAW files I tried, it doesn't decode them with RawSpeed3, including Sony ARW files or Nikon Z camera RAW files.
The doc mentioned about:
RawSpeed-v3 support is controlled via libraw_decoder_info.t->flags (LIBRAW_DECODER_TRYRAWSPEED3 bit).
First of all, there is no libraw_decoder_info.t->flags, but there is libraw_decoder_info_t->decoder_flags.
I assume you're referring to this decoder_flags?
I can assign decoder_flags = LIBRAW_DECODER_TRYRAWSPEED3, but how do I pass the libraw_decoder_info_t data structure to the raw processor so that it knows I want it to try RawSpeed3?
There is a get_decoder_info() function, but not a set_decoder_info().
Quote from RawSpeed3/README.md:
1.
To make a file with camera definitions, run the command (you can add it to the build system): sh ./rawspeed3_c_api/rsxml2c.sh < path/to/RawSpeed/data/cameras.xml > rawspeed3_c_api/cameras.cpp
The rsxml2c.sh script requires cat, tr, and sed unix command-line utilities installed, there is no specific version requirements.
2.
Add the resulting file (LibRaw/RawSpeed3/rawspeed3_c_api/cameras.cpp) and LibRaw/RawSpeed3/rawspeed3_c_api/rawspeed3_capi.cpp, to the build of the RawSpeed-v3 (dynamic) library, which you learned to build in the previous step. If building Windows DLL: rawspeed3_capi.cpp should be compiled with -DRAWSPEED_BUILDLIB to create dll export entries automatically.
3.
To check that everything works, you can use the LibRaw/RawSpeed3/rawspeed3_c_api/rawspeed3_capi_test.cpp test program, it should be possible to build it and link with RawSpeed-v3 (+ C-API); and after that this test program should be ready to decode RAW files passed to it from command line.
Please clarify what is not clear in the steps quoted above.
Yes, this seems to do the trick. Thanks.
Have you tried, from the link I suggested, https://www.libraw.org/docs/Samples-LibRaw.html :
"-c float-value
This key sets params.adjust_maximum_thr parameter.
Use -c 0 to completely disable automatic maximum calculation.
Default value: 0.75"
THIS.
dcraw_emu -W -c 0 -g 1 0
gives linear response to varying exposure time. Thanks.
I'm not sure what you mean, but by linear behaviour I mean that if I take images of the same identical subject with the same illumination, but with different exposure times, and then plot pixel value VS exposure time, the curve is linear (if gamma is 1). In the plot I attached to the original post you can see this behaviour.
Final followup: you probably need to turn off auto-maximum detection via dcraw_emu -c 0
You probably need to either use unprocessed raw values or program your own (simple) postprocessing.
Please define "linear behaviour" and maybe demonstrate with numbers over a grey step wedge what you mean.
DNGs don't have brightness. First, brightness isn't an attribute of a digital image; second, there is no such thing as standard calibration for raw.
This needs to be said, sorry.
I see, thanks. As I said though, gamma is not really a problem. Even if I set gamma to 1 I do not see the linear behaviour that I'd like to get (https://imgur.com/a/a1C3cfV for just a few images), while I do get it with the DNG SDK and unprocessed_raw.
Left to right:
dcraw_emu 00.dng
dcraw_emu -W 00.dng
dcraw_emu -W -g 1 0 00.dng
https://www.dropbox.com/s/texnp4rxpwgh8i0/Screenshot%202023-01-05%201908...
If TIFF output used for -g.... one may be fooled by TIFF viewer if the viewer is ICC-aware
1) when using dcraw_emu, -g 1 0 doesn't seem to have an effect on top of -W. I actually don't have a problem with having a non-unitary gamma, as long as it is the same on all images. I can compensate for that later in the processing.
2) White Balance is not really a problem neither, I have a neutral area in the target I'm imaging and I will in any case perform white balance using that.
3) I did look at unprocessed_raw and it does give a linear output. However, it returns images before demosaicing, which I'd prefer not to have to code in myself, since I'd assume libRAW has significantly better functionality for this than I can code. I'd rather invest time into trying to get linear output from built-in functionality than having to code demosaicing.
So, what I'd like to find out is if there is something we can do to get rid of that extra processing that happens in dcraw_emu -W, but without going as far back as unprocessed_raw.
Also, your 'tif' files are not white-balanced, probably you may want to set custom 'Uni-WB' white balance.
Also, if you want to get unscaled/unaltered data extracted: take a look for unprocessed_raw and/or 4channels LibRaw samples.
>> for half the exposure time I get half the pixel value
If you need 'linear output' you probably may want to set output gamma to 1.0
You can find an example dataset here https://drive.google.com/drive/folders/1X4Cqj-hhLXOktWsfFN4e2yfWWZI5BGHK...
There you can find
1) DNG images from the camera
2) JPG images automatically generated by the camera (I use those to get exposure times from EXIF data)
3) TIF images obtained using the stage3 output of the Adobe DNG SDK
4) PPM images obtained using dcraw_emu.exe -W (which should be equivalent to what described in this comment https://www.libraw.org/comment/6511#comment-6511)
I like the TIF images because for half the exposure time I get half the pixel value, across the exposure range. The PPM images are not predictable, which implies that some extra processing is happening. Please see the original attachment for a plot of image average (across all pixels, all channels) VS exposure time.
Sorry, I didn’t realize right away that I was already talking with two people.
Could you please formulate your problem not referring to the 'OP'. Also, it would be great to have RAW file(s) in question еo discuss something in specific detail and not in the abstract.
Pages