random Crash with CRW file in memory stream

Submitted by yan on

Hi.
I've found a random crash with MemoryStream and CR2.
The file is downloaded from edsdk api and i use open_buffer to unpack image.
I try to found it. It's like datastream make memory erro whith cr2 file. Crash always appear on memchr in libraw_datastream.h :

int get_ljpeg_byte() {
        if(offt<next_ff) return buf[offt++];
        int ret = buf[offt++];
        if(ret == 0xff) { if(buf[offt]==0x00) offt++; else return 0;}
        // find next 0xff
        unsigned char *p = (unsigned char*) memchr(buf+offt,0xff,size-offt);
        if(p)
            next_ff = p-buf;
        else
            next_ff = size;
        return ret;
    }
 <cpp>

Forums: 

I cannot reproduce the

Submitted by lexa on

I cannot reproduce the problem (using dcraw_emu -mmap *CR2).

Could you please describe your system enviroment (OS, compiler version and so)?

-- Alex Tutubalin @LibRaw LLC

Windows seven 64bits Visual

Submitted by yan on

Windows seven 64bits
Visual studio 2010
libraw 1.13.1 build in 32bits

It's a random crash. Generally it's appeare after 10-30 shoot in RAW.

I'll test to read same image N times in a loop to see if this reproduct crash.

Unfortunately, I've no canon

Submitted by lexa on

Unfortunately, I've no canon SDK on hands, so cannot repeat your case.
I've submit data for Canon Developer account and waiting for approval.

As a workaround you may try to copy the data from ESDK into permanent buffer with enough size allocated at your program start. At least, the problem will become more repeatable.

-- Alex Tutubalin @LibRaw LLC

with this code

Submitted by yan on

with this code (approximatively what i do), in release, i crash(heap corruption) with recycle.
If i remove recycle, doesn't crash.
heap corruption could be in recycle? mabe it's the two reclycle call (me and libraw destructor) wich corrupt heap?

#include "libraw.h"
#include <vector>
#include <iostream>
#include <fstream>
 
int main (int argc, char * argv[])
{
     for(int i =1;true;++i)
    {
        std::cout <<"image "<<i<<std::endl;
        std::vector<char > buff;
        std::ifstream f;
        f.open(argv[1],std::iostream::binary);
 
        long size;
        {
            long begin;
            begin = f.tellg();
            f.seekg (0, std::ios::end);
            size = long(f.tellg()) - begin;
            f.seekg (0, std::ios::beg);
        }
        buff.resize(size);
        f.read(&buff[0],size);
 
 
        LibRaw proc;
        std::cout <<"open buffer : " << proc.open_buffer(&buff[0],size) <<std::endl;
        std::cout <<"unpack : " <<  proc.unpack()<<std::endl;
 
        std::cout <<"recycle"<<std::endl<<std::endl;
        proc.recycle();
 
 
    }
 
    return 0;
}
<cpp>

recycle() is called too in

Submitted by lexa on

recycle() is called too in the thery beginning of the open_datastream() call (open_buffer() translated into it).

It should be no problem on extra recycle() calls because it should do nothing if anything already cleaned.

Anyway, I'll check in deep. It is *possible* that some things done wrong in case of memory buffer.

Thanks for your report!

-- Alex Tutubalin @LibRaw LLC

OK,ok. This sample does not

Submitted by lexa on

OK,ok. This sample does not requires Canon SDK, I'm already looking into similar code....

-- Alex Tutubalin @LibRaw LLC

Because I cannot reproduce

Submitted by lexa on

Because I cannot reproduce the bug, could you please provide more details about bug 'environment'.

If crashes are in memchr() call, this may be caused *ONLY* by 'out of buffer' memchr run.
So, you have the &buff[0] address (in the main) and size. The memchr is called on some internal sub-buffer:
memchr(buf+offt,0xff,size-offt)

So the question are:
1) is buf (buffer begin) is within your buff[...] ?
2) is buf+size (in memchr context) is within your buff[...]

-- Alex Tutubalin @LibRaw LLC

it's a very *bip* bug :/ I ve

Submitted by yan on

it's a very *bip* bug :/

I ve make a lot of test today. With the main have done, crash is very rarely in debug and random in release. I test to add libraw src to not use dll. I've crash only in release.

i'll intall a linux and test with valgrin. Maybe i'll found something.
thanks for your help.

For Windows version the

Submitted by lexa on

For Windows version the context is stored in object. So, it reentrant.

For Unix version, the default is not re-entrant, but libraw_r library is reentrant.

-- Alex Tutubalin @LibRaw LLC

No-no. libraw (.a, .so) is

Submitted by lexa on

No-no.

libraw (.a, .so) is compiled with -DLIBRAW_NOTHREADS, so some internal decoder buffers are static. This is faster, but this library is not reentrant. You may use it in multithreaded environment, but not for decoding several files in parallel

libraw_r is compiled with -pthread and without LIBRAW_NOTHREADS define, so all I/O calls are made via thread-safe version and decoder storage is allocated within LibRaw class. So, extra locks, extra indirection for storage, but thread-safe.

Not reentrant version is several percent faster.

-- Alex Tutubalin @LibRaw LLC

I've run this code in the

Submitted by lexa on

I've run this code in the same enviroment:
Windows7 x64
VC++ 2010 (with SP1)
LibRaw 0.13.1
And no problems in two tests:
* debug build, Canon 30D sample image, about 250 iterations
* release build, Canon 5DmkII sample image, 100+ iterations several times.
I've tried three LibRaw.DLL versions: from 0.13.1 distribution (with and without GPL2/3 packs) and rebuilt from source (without demosaic packs).

So, case is still unconfirmed for me. Is it possible that your computer has some memory problem (becasuse of overclock or overheat or so)?

Also, I've tried to move 'LibRaw proc' declaration outside from loop (within loop the object will be destroyed on each iteration).

-- Alex Tutubalin @LibRaw LLC

Also, I've added buffer-read

Submitted by lexa on

Also, I've added buffer-read ability to dcraw_emu sample (modified version available from GitHub) and test open_buffer() code under Windows.

Again, no errors and no problems.

-- Alex Tutubalin @LibRaw LLC

I've test with valgrind. No

Submitted by yan on

I've test with valgrind. No memory error found ^^

I check your build option and my visual project option. We have a difference .
You use /MT and i use /MD. In this config, on an xp the main example crash on the first unpack.
If i change you option to use /MD it don't crash :D
I found more information about this mix error here :
http://msdn.microsoft.com/en-us/library/2kzt1wy3(v=vs.71).aspx


Caution Do not mix static and dynamic versions of the run-time libraries. Having more than one copy of the run-time libraries in a process can cause problems, because static data in one copy is not shared with the other copy. The linker prevents you from linking with both static and dynamic versions within one .exe file, but you can still end up with two (or more) copies of the run-time libraries. For example, a dynamic-link library linked with the static (non-DLL) versions of the run-time libraries can cause problems when used with an .exe file that was linked with the dynamic (DLL) version of the run-time libraries. (You should also avoid mixing the debug and non-debug versions of the libraries in one process.)

I'll check tomorrow with my application if it's only that.

ps :I think you should propose a debug and a release dll with /MD and /MDd option.

Thanks for info. It looks

Submitted by lexa on

Thanks for info.

It looks like I need to provide four different DLLS:
librawT.dll - /MT
librawTd.dll - /MTd
librawD.dll - /MD
librawDd.dll - /MDd

Oh! Windows make me crazy!

-- Alex Tutubalin @LibRaw LLC

In the best, is that i

Submitted by yan on

In the best, is that i understand.

I thinks if you use pimpl idiom on LibRaw class you can remove this problem.
If i understand correctly the error, your libraw_memmgr implementation is declared in .h and is inlined.
So when you use it in application free and malloc are not the same that use in the dll.

http://support.microsoft.com/kb/140584/en-us

Yes, memory manager is

Submitted by lexa on

Yes, memory manager is exposed to public. But the code that uses memory manager is entirely within DLL, including LibRaw class constructor and all methods.

The problem may occur only with link time code generation, but LibRaw.DLL do not use it.

Anyway, 4 different DLLs should solve the problem.

-- Alex Tutubalin @LibRaw LLC

Like LibRaw expose

Submitted by yan on

Like LibRaw expose libraw_memmgr memmgr; i think memmgr use malloc and free from crt associated with application. Not with crt associated with the dll. Internally that can cause CRT error.

I test with my application. I've always a crash in memchr. But i think it's a multithread error with edsdk and COM initialization.

thanks for help.

Good news. If i use my memory

Submitted by yan on

Good news.
If i use my memory to dowload cr2 file with edsdk, i haven't crash :D

thanks.

Is is possible, that ESDK

Submitted by lexa on

Is is possible, that ESDK memory is protected from outside access and only ESDK call may use it?

Unfortunately, I've no reply from Canon-Europe yet, so cannot look into ESDK.

-- Alex Tutubalin @LibRaw LLC

If i read doc, i can use it's

Submitted by yan on

If i read doc, i can use it's memory stream which received file :/ but random crash appear. It's like stream size is incorrect (and it's possible ^^).

Now it's works perfectly for me when i'm embded my memory to edsdk MemoryStream.

For edsdk, you probably wait a long time before receive an answer...

Memory manager is used (and

Submitted by lexa on

Memory manager is used (and inlined) only within LibRaw::* calls.
These calls (including constructor) are not inlined, so there are no problem with it.

There *was* a problem with LibRaw::make_mem_image() call in previous versions of LibRaw: if you're allocated mem_image within LibRaw and free() it within your app, you're in trouble. The problem is fixed in 0.12 or 0.11 or so by providing extra call ..clear_mem_image()

But the problem was *not* random, it appear every time and within free() call, not in memchr() as in your case.

-- Alex Tutubalin @LibRaw LLC