LibRaw with Adobe DNG SDK 1.7

Submitted by lrusr19 on

Today I've tried to get LibRaw working with DNG SDK 1.7 using Visual C++ 2022.
I've downloaded the latest LibRaw snapshot from Github.

After some struggling I've managed to create a .lib for the DNG SDK (Adobe should really include a VC solution for that) and then I've following the instructions in README.DNGSDK.txt

My first tests show that this apparently works many of my sample DNG files, but it fails for others, producing "wrong" colors with a bad magenta cast. See attached image 003 (image 002 is the same photo loaded in ACR with default settings).

I'm not all to deep into RAW processing, so I really don't know where to start or which of the many LibRaw parameters and options impact this.

Here is the code I use (I save the LibRaw output to a TIF file for further processing).

dng_host host;
host.SetPreferredSize(0);
host.SetMinimumSize(2000);
host.SetMaximumSize(0);
 
auto plraw = std::make_unique<LibRaw>(LIBRAW_OPIONS_NO_DATAERR_CALLBACK);
plraw->set_dng_host(&host);
plraw->imgdata.rawparams.use_dngsdk = LIBRAW_DNG_ALL;
plraw->imgdata.rawparams.options |= LIBRAW_RAWOPTIONS_ALLOW_JPEGXL_PREVIEWS;
 
plraw->imgdata.params.use_camera_wb = 1;
plraw->imgdata.params.use_auto_wb = 1;
plraw->imgdata.params.highlight = 0;
plraw->imgdata.params.user_qual = 3;
plraw->imgdata.params.output_color = 1;
memset(plraw->imgdata.params.gamm, 0x0, sizeof(plraw->imgdata.params.gamm));
plraw->imgdata.params.gamm[0] = 1 / 2.4;
plraw->imgdata.params.gamm[1] = 12.92;
plraw->imgdata.params.bright = 1;
plraw->imgdata.params.no_auto_bright = 0;
plraw->imgdata.params.auto_bright_thr = 0.002f;
plraw->imgdata.params.exp_correc = 1;
plraw->imgdata.params.output_bps = 8;
plraw->imgdata.params.output_tiff = 1;
 
plraw->unpack();
plraw->dcraw_process();
plraw->dcraw_ppm_tiff_writer(...
...

Any help or suggestions greatly appreciated.
I've had problems uploading the second file. Hopefully this works. It shows in the preview.

Forums: 

It is usually pointless to

Submitted by lexa on

It is usually pointless to discuss problems in processing a file without having the file itself

-- Alex Tutubalin @LibRaw LLC

I understand.

Submitted by lrusr19 on

I have sent an email to the owner of the image and asked if I can provide it to you as an example.
I will post here when I have the permission and provide a download link.
It seems to use the JPXL compression, that's all I can tell so far.

Sample DNG for the color problem

Submitted by lrusr19 on

I have permission from the image owner and I have uploaded a sample image in a ZIP here:

Sample Image download (12 MB)

Please let me know which LibRaw setting or steps I need to perform to get rid of the color cast, if possible.
Or maybe I need to configure the Adobe DNG SDK host instance specifically?

All this is new to me and any help is appreciated.

Thank you for the file sample

Submitted by lexa on

Thank you for the file sample.

Here is processing results: https://www.dropbox.com/scl/fi/ooqcaseziap0vwo5ario4/2023-11-11-003-Pano...

dcraw_emu command line used for that:

dcraw_emu -dngsdk -R 3145728 -T 2023-11-11 003-Pano.dng

Technical details:

  1. The image is linearized via OpcodeList2 tag value; For similar images use of OpcodeList3 is also possible (probably depends on Adobe software version used).
  2. By default, only OpcodeList1 tag is processed by LibRaw
  3. To enable OpcodeList2/3 processing LIBRAW_RAWOPTIONS_DNG_STAGE2_IFPRESENT/LIBRAW_RAWOPTIONS_DNG_STAGE3_IFPRESENT flags should be added to imgdata.rawparams.options
  4. This translates into numeric value of (1<<20)|(1<<21) so 3145728

This is not perfect solution if one do not want to apply OpcodeList2/3 for all DNG files containing these tags.

To improve it: query decoder used via LibRaw::get_decoder_info() and set the flags only for jxl_dng_load_raw_placeholder() and/or lossy_dng_load_raw() decoders.

-- Alex Tutubalin @LibRaw LLC

Thanks, Alex!

Submitted by lrusr19 on

Thanks, Alex!
this is really helpful. I will implement your suggestions as soon as possible.

Are these LIBRAW_RAWOPTIONS_DNG_STAGE* flags documented somewhere? I have found them in the source code and see where you apply them, and I have some rough idea about opcodes in DNG files, but I would never have thought that I have to apply them or which of them.

In most cases, the DNG files I have to handle have large-enough embedded previews and that's sufficient for my use-case.

But when it comes to actual processing of RAW data into a usable image, things get a lot more difficult and finding parameters which work across a wide range of RAW files is tough.

I've got this working,

Submitted by lrusr19 on

I've got this working, following your instructions.
I call:

libraw_decoder_info_t dinfo;
plraw->get_decoder_info(&dinfo);

and compare the dinfo->name with "jxl_dng_load_raw_placeholder()" and "vc5_dng_load_raw_placeholder()", as you suggested (I hope I understood you correctly).

And this works very well for the problem image. In the debug built.

When I switch to the release built (Visual C++ 2022), the dinfo->name returned is "lossy_dng_load_raw()" for some reason I don't understand yet. I will prepare a release build with debug symbols so I can trace into this. The preprocessor defines etc. the same for both builds.

When I also set the new opcode flags when "lossy_dng_load_raw()" is returned in dinfo->name, the image looks good in the release build too.

Any idea?

vc5_dng_load_raw_placeholder

Submitted by lexa on

vc5_dng_load_raw_placeholder is GoPro files decoder name. It was not mentioned in my reply; Also this name should not be returned in get_decoder_info() for both release and debug (unless you load DLL with different LibRaw version....)

-- Alex Tutubalin @LibRaw LLC

I've tested this and it seems

Submitted by lrusr19 on

I've tested this and it seems to be some kind of code generation / optimization issue with Visual C++ 2022...?
I create LibRaw as a static library.

When I use the debug build, the decoder_name returned is "jxl_dng_load_raw_placeholder()".

When I compile the release build with optimizations disabled (/Od) or set to Optimizations (Favor Speed) (/Ox), the decoder_name returned is also "jxl_dng_load_raw_placeholder()".

But when I build release with Maximum Optimization (Favor Speed) (/O2), the decoder_name is set to "vc5_dng_load_raw_placeholder()"

Looking at the LibRaw source, this should only happen when USE_GPRSDK is defined (it is not in my build) so I can only assume that there is some sort of compiler error or wrong jump produced when optimizing that case statement when the /O2 is enabled.

Looks interesting.

Submitted by lexa on

Looks interesting.

In fact, both functions have the same body:

void LibRaw::vc5_dng_load_raw_placeholder()
{
    // placeholder only, real decoding implemented in GPR SDK
    throw LIBRAW_EXCEPTION_UNSUPPORTED_FORMAT;
}
void LibRaw::jxl_dng_load_raw_placeholder()
{
  // placeholder only, real decoding implemented in DNG SDK
  throw LIBRAW_EXCEPTION_UNSUPPORTED_FORMAT;
}

Looks like Visual studio detects this, compiles as single function, so
else if (load_raw == &LibRaw::vc5_dng_load_raw_placeholder)
and
else if (load_raw == &LibRaw::jxl_dng_load_raw_placeholder)
Are both true....

Looks like
1) these functions should be separated in additional code unit
2) All optimizations should be disabled at least for common compilers (gcc, clang, visual studio).

-- Alex Tutubalin @LibRaw LLC

#pragma optimize( "", off )

Submitted by lexa on

#pragma optimize( "", off )
before
void LibRaw::vc5_dng_load_raw_placeholder()

and
#pragma optimize( "", on )
before
void LibRaw::adobe_copy_pixel(unsigned row, unsigned col, ushort **rp)

(in src/decoders/dng.cpp)
Will probably fix the issue.

These pragmas are Visual Studio specific; we'll implement wider solution in the next update

-- Alex Tutubalin @LibRaw LLC

I've tried that. No change :-

Submitted by lrusr19 on

I've tried that. No change :-(
Visual Studio 2022 17.8.1

I've also disabled the optimization for the get_decoder_info() function and even for the declarations for vc5_dng_load_raw_placeholder() and jxl_dng_load_raw_placeholder() in the header.
Did a full rebuild of LibRaw each time, just in case.
No change in the name returned!

I then moved the call to get_decoder_info into a separate function in my code (returns true if the options should be added) and wrapped that in optimize off / on.
Same result. Still returned the wrong name.

I think this is something really strange with Maximum Optimization that somehow rolls things together on a more global level. Normal Optimization works, so I'll use that for now. Until you maybe change the code to work around this somehow.

Tried the linker option. No

Submitted by lrusr19 on

Tried the linker option. No change.
This is not available for libraries (I link LibRaw as a static lib). But setting the options in my DLL which links to LibRaw made no change.

I've also played with string pooling and the other options that differ (https://learn.microsoft.com/en-us/cpp/build/reference/o1-o2-minimize-siz...) between Optimization (Speed) which works and Maximize Optimization (Speed) which does not work. Without any success.

I'll stick to /Ox /Ob2 /Oi /Ot for now.

This is not string pooling,

Submitted by lexa on

This is not string pooling, this should be linker identical code folding (designed to merge same methods of different classes, e.g. templates)
So, two originally different function pointers are the same in the optimized code.

We'll return to this but not immediately

-- Alex Tutubalin @LibRaw LLC

I've tried all the options

Submitted by lrusr19 on

I've tried all the options that could make a difference, but there are not that many options in the Librarian when compiling LibRaw as a static lib.

I've settled to compile LibRaw with Ox instead O2 and that works just fine. Maybe it's a tick slower than it could, but I think it's fine.

Do you have a big spam problem? The post form always tells me to take more time filling the form. Maybe I type too fast ;-)