What can cause different output inside Docker container?

Submitted by dmitri on

Hello,

I'm executing `dcraw_emu` inside a Docker container that uses `node:23-alpine` and the latest version of LibRaw.

When I run LibRaw to convert a DNG file inside the container running locally, I get good results. However, when the container is running on Digital Ocean, my results look very different: there's a lot of data lost in the output TIFF file that causes the result to look way too bright.

This happens only to a specific DNG file that comes from a scanner operated by SilverFast software. Other DNG files work fine and look the same on both systems.

The only difference I can think of is that Digital Ocean uses AMD chips, and I'm testing on ARM (M1 on macOS). Could that be the issue, or could there be something else?

Forums: 

Please make sure you're using

Submitted by lexa on

Please make sure you're using same LibRaw version compiled with same options/compile flags (e.g. DNG SDK support) and with same run-time options

-- Alex Tutubalin @LibRaw LLC

Thank you for your quick

Submitted by dmitri on

Thank you for your quick response, Alex.

I'm using librav-0.21.3-r0 on both systems. Here's a screenshot of what I gathered (left is my Docker container running on Apple M1 laptop, right is a Docker container running on Digital Ocean instance): https://www.dropbox.com/scl/fi/umk20exovyygzs952ywfi/versions.png?rlkey=...

Both systems run -6 -o 1 -g 3.0 12.0 -r 1.0 1.0 2.2 1.0 -T /vol/test.dng but the results are different.

https://www.dropbox.com/scl/fi/axtohrmyfr2g903pxjvbk/small_compare.png?r... This is a screenshot of what the source DNG looks like when my macOS renders it as a file preview test.dng, the result I got with LibRaw locally LOCAL test.dng.tiff, and the result I got with LibRaw running on Digital Ocean Docker container DO test.dng.tiff. As you can see, the one rendered by DO looks off, even though, in general, there are no issues with the DO LibRaw app.

If the original DNG would be helpful, I'll ask its owner and get back. Please let me know!

Is there anything else I should consider?

Here's detailed info about the DNG file:

$ exiftool test.dng
ExifTool Version Number         : 13.25
File Name                       : test.dng
Directory                       : .
File Size                       : 109 MB
File Permissions                : -rw-r--r--
File Type                       : DNG
File Type Extension             : dng
MIME Type                       : image/x-adobe-dng
Exif Byte Order                 : Big-endian (Motorola, MM)
Make                            : Plustek
Camera Model Name               : OpticFilm 8200i
Orientation                     : Horizontal (normal)
Software                        : SilverFast 8.8.0 r25 (Jun 25 2021)  d6df961 25.06.
Image Width                     : 4824
Image Height                    : 3408
Bits Per Sample                 : 16 16 16
Compression                     : Uncompressed
Photometric Interpretation      : Linear Raw
Samples Per Pixel               : 3
Rows Per Strip                  : 1
X Resolution                    : 300
Y Resolution                    : 300
Planar Configuration            : Chunky
Resolution Unit                 : inches
XMP Toolkit                     : Adobe XMP Core 7.0-c000 1.000000, 0000/00/00-00:00:00
Company                         : LaserSoft Imaging
Original Scan Frame             : <?xml ... VERY LONG STRING ... >
Scan Frames                     : <?xml ... VERY LONG STRING ... >
Creator Tool                    : SilverFast 8.8.0 r25 (Jun 25 2021)  d6df961 25.06.
Metadata Date                   : 2025:04:13 08:42:22-07:00
Document ID                     : E3BB96E86460FD74E93B6EB4A3FA1A53
Original Document ID            : E3BB96E86460FD74E93B6EB4A3FA1A53
Instance ID                     : xmp.iid:06a79b8e-04a5-4b52-a580-95f09cfce4aa
Format                          : image/dng
Version                         : 16.5
Process Version                 : 15.4
White Balance                   : As Shot
Color Temperature               : 6500
Tint                            : +10
Exposure 2012                   : 0.00
Contrast 2012                   : 0
Highlights 2012                 : 0
Shadows 2012                    : 0
Whites 2012                     : 0
Blacks 2012                     : 0
Texture                         : 0
Clarity 2012                    : 0
Dehaze                          : 0
Vibrance                        : 0
Saturation                      : 0
Parametric Shadows              : 0
Parametric Darks                : 0
Parametric Lights               : 0
Parametric Highlights           : 0
Parametric Shadow Split         : 25
Parametric Midtone Split        : 50
Parametric Highlight Split      : 75
Sharpness                       : 40
Sharpen Radius                  : +1.0
Sharpen Detail                  : 25
Sharpen Edge Masking            : 0
Luminance Smoothing             : 0
Color Noise Reduction           : 25
Color Noise Reduction Detail    : 50
Color Noise Reduction Smoothness: 50
Hue Adjustment Red              : 0
Hue Adjustment Orange           : 0
Hue Adjustment Yellow           : 0
Hue Adjustment Green            : 0
Hue Adjustment Aqua             : 0
Hue Adjustment Blue             : 0
Hue Adjustment Purple           : 0
Hue Adjustment Magenta          : 0
Saturation Adjustment Red       : 0
Saturation Adjustment Orange    : 0
Saturation Adjustment Yellow    : 0
Saturation Adjustment Green     : 0
Saturation Adjustment Aqua      : 0
Saturation Adjustment Blue      : 0
Saturation Adjustment Purple    : 0
Saturation Adjustment Magenta   : 0
Luminance Adjustment Red        : 0
Luminance Adjustment Orange     : 0
Luminance Adjustment Yellow     : 0
Luminance Adjustment Green      : 0
Luminance Adjustment Aqua       : 0
Luminance Adjustment Blue       : 0
Luminance Adjustment Purple     : 0
Luminance Adjustment Magenta    : 0
Split Toning Shadow Hue         : 0
Split Toning Shadow Saturation  : 0
Split Toning Highlight Hue      : 0
Split Toning Highlight Saturation: 0
Split Toning Balance            : 0
Color Grade Midtone Hue         : 0
Color Grade Midtone Sat         : 0
Color Grade Shadow Lum          : 0
Color Grade Midtone Lum         : 0
Color Grade Highlight Lum       : 0
Color Grade Blending            : 50
Color Grade Global Hue          : 0
Color Grade Global Sat          : 0
Color Grade Global Lum          : 0
Auto Lateral CA                 : 0
Lens Profile Enable             : 0
Lens Manual Distortion Amount   : 0
Vignette Amount                 : 0
Defringe Purple Amount          : 0
Defringe Purple Hue Lo          : 30
Defringe Purple Hue Hi          : 70
Defringe Green Amount           : 0
Defringe Green Hue Lo           : 40
Defringe Green Hue Hi           : 60
Perspective Upright             : Off
Perspective Vertical            : 0
Perspective Horizontal          : 0
Perspective Rotate              : 0.0
Perspective Aspect              : 0
Perspective Scale               : 100
Perspective X                   : 0.00
Perspective Y                   : 0.00
Grain Amount                    : 0
Post Crop Vignette Amount       : 0
Shadow Tint                     : 0
Red Hue                         : 0
Red Saturation                  : 0
Green Hue                       : 0
Green Saturation                : 0
Blue Hue                        : 0
Blue Saturation                 : 0
HDR Edit Mode                   : 0
Convert To Grayscale            : False
Override Look Vignette          : False
Tone Curve Name 2012            : Linear
Camera Profile                  : Embedded
Camera Profile Digest           : 553F5A29E2BFD14973038D1977616F6F
Has Settings                    : True
Has Crop                        : False
Already Applied                 : False
History Action                  : saved
History Instance ID             : xmp.iid:06a79b8e-04a5-4b52-a580-95f09cfce4aa
History When                    : 2025:04:13 08:42:22-07:00
History Software Agent          : Adobe Photoshop Camera Raw 16.5 (Macintosh)
History Changed                 : /metadata
Tone Curve PV2012               : 0, 0, 255, 255
Tone Curve PV2012 Red           : 0, 0, 255, 255
Tone Curve PV2012 Green         : 0, 0, 255, 255
Tone Curve PV2012 Blue          : 0, 0, 255, 255
Point Colors                    : -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000
Exif Version                    : 0231
Maker Note Unknown Binary       : (Binary data 130 bytes, use -b option to extract)
Exif Image Width                : 4824
Exif Image Height               : 3408
DNG Version                     : 1.4.0.0
DNG Backward Version            : 1.0.0.0
Unique Camera Model             : Plustek OpticFilm 8200i
Color Matrix 1                  : 3.2404542 -1.5371385 -0.4985314 -0.969266 1.8760108 0.041556 0.0556434 -0.2040259 1.0572252
As Shot White XY                : 0.3127 0.329
Maker Note Safety               : Unsafe
Raw Data Unique ID              : E3BB96E86460FD74E93B6EB4A3FA1A53
Raw Image Digest                : 8290177745e2f8f5b8325a732755da1f
Subfile Type                    : Reduced-resolution image
Strip Offsets                   : (Binary data 8186 bytes, use -b option to extract)
Strip Byte Counts               : (Binary data 5144 bytes, use -b option to extract)
Profile CMM Type                : Linotronic
Profile Version                 : 2.1.0
Profile Class                   : Input Device Profile
Color Space Data                : RGB
Profile Connection Space        : XYZ
Profile Date Time               : 2021:05:21 16:05:34
Profile File Signature          : acsp
Primary Platform                : Microsoft Corporation
CMM Flags                       : Not Embedded, Independent
Device Manufacturer             : 
Device Model                    : 
Device Attributes               : Reflective, Glossy, Positive, Color
Rendering Intent                : Perceptual
Connection Space Illuminant     : 0.9642 1 0.82491
Profile Creator                 : 
Profile ID                      : 0
Profile Copyright               : Copyright by LaserSoft Imaging AG
Profile Description             : SFprofT (OpticFilm 8200i)
Media White Point               : 0.96362 1.01057 0.81055
Red Matrix Column               : 0.21921 0.08772 0
Green Matrix Column             : 0.13847 0.57034 0
Blue Matrix Column              : 0.19788 0.01196 0.84502
Red Tone Reproduction Curve     : (Binary data 14 bytes, use -b option to extract)
Green Tone Reproduction Curve   : (Binary data 14 bytes, use -b option to extract)
Blue Tone Reproduction Curve    : (Binary data 14 bytes, use -b option to extract)
A To B0                         : (Binary data 217222 bytes, use -b option to extract)
A To B1                         : (Binary data 217222 bytes, use -b option to extract)
A To B2                         : (Binary data 217222 bytes, use -b option to extract)
Image Size                      : 4824x3408
Megapixels                      : 16.4
Preview TIFF                    : (Binary data 8995734 bytes, use -b option to extract)
Thumbnail TIFF                  : (Binary data 276696 bytes, use -b option to extract)

Sorry, the docker stuff is

Submitted by lexa on

Sorry, the docker stuff is completely out of our scope.

Could you please compile LibRaw from source on the problematic system and try to reproduce the problem?

Also: it is pointless to discuss some problem with specific raw (dng) file without access to specific file (But if we can't reproduce the problem ourselves, we can't help either).

-- Alex Tutubalin @LibRaw LLC

Downloaded, thank you (you

Submitted by lexa on

Downloaded, thank you (you may remove from Dropbox to not extra distribute the file).

(As expected) the file looks normal regardless of decoding path used (with or without Adobe DNG SDK). Not surprising for uncompressed 16-bit linear (3-channel) DNG: there are very few places where things can go wrong.

Could you please use LibRaw's 4channels sample to check if RAW decoding/unpacking is the same on both platforms. 4channels produces four monochrome tiff files (G.tiff, R.tiff, B.tiff, G2.tiff), one per channel.

G2 should be completely black. Other three should match and provide same checksum on two different systems.

It it is so: the problem is in postprocessing stage (LibRaw::dcraw_process), but first, it's worth making sure.

-- Alex Tutubalin @LibRaw LLC

OK, thanks.

Submitted by lexa on

OK, thanks.

Looks like byte order swapped in REMOTE sample.

I'll investigate the problem in depth tomorrow.
At a 1st glance: LibRaw::read_shorts does not work as expected.

It calls libraw_swab() if host byte order does not match file byte order (this is the case: MM/Motorola byte order in file and II/intel byte order on host).

Depending on compile flags, libraw_swab is implemented in two ways:
- own code if LIBRAW_OWN_SWAB defined on compile
- host (libc) swab() function overwise.

Is there any chance to know compile flags used on DO-docker?

-- Alex Tutubalin @LibRaw LLC

Checked with LIBRAW_OWN_SWAB

Submitted by lexa on

Checked with LIBRAW_OWN_SWAB defined (so, own LibRaw swab used): looks OK.

So the only suspect for now: host (libc) swab() function

Recompilation with LIBRAW_OWN_SWAB should solve the issue.

-- Alex Tutubalin @LibRaw LLC

Alex, I'd like to confirm

Submitted by dmitri on

Alex, I'd like to confirm that your suggestion worked and the issue is now fixed.

I used the following flags in my Docker container:

ENV CFLAGS="-O2 -g -DLIBRAW_OWN_SWAB" 
ENV CXXFLAGS="-O2 -g -DLIBRAW_OWN_SWAB"
ENV LDFLAGS="-L/usr/local/lib"

With code tagged 0.21.4 found in the GitHub repo, compiled with make install.

Thank you!

That means that system swab()

Submitted by lexa on

That means that system swab() call is broken in this specific environment. A very strange story...

-- Alex Tutubalin @LibRaw LLC