Different behaviour between git and 0.20.2 with wide char

Submitted by lock on

Hello everyone,
I come with a small question. Indeed, I'm using libraw in my project and everything is fine. However I recently found a small glitch if I use the git version and not the 0.20.2 version.

To handle wide char filenames I use a function to convert them and it was ok until now. It looks like the git version has changed some behaviour.

char* get_real_path(const char *source) {
	HANDLE hFile;
	DWORD maxchar = 2048;
 
	wchar_t *wsource = g_utf8_to_utf16(source, -1, NULL, NULL, NULL);
	if ( wsource == NULL ) {
		return NULL ;
	}
 
	if (!(GetFileAttributesW(wsource) & FILE_ATTRIBUTE_REPARSE_POINT)) { 
		g_free(wsource);
		return NULL;
	}
 
	wchar_t *wFilePath = g_new(wchar_t, maxchar + 1);
	if (!wFilePath) {
		PRINT_ALLOC_ERR;
		g_free(wsource);
		return NULL;
	}
	wFilePath[0] = 0;
 
	hFile = CreateFileW(wsource, GENERIC_READ, FILE_SHARE_READ, NULL,
			OPEN_EXISTING, 0, NULL);
	if (hFile == INVALID_HANDLE_VALUE) {
		g_free(wFilePath);
		g_free(wsource);
		return NULL;
	}
 
	GetFinalPathNameByHandleW(hFile, wFilePath, maxchar, 0);
 
	gchar *gFilePath = g_utf16_to_utf8(wFilePath + 4, -1, NULL, NULL, NULL); // +4 = enleve les 4 caracteres du prefixe "//?/"
	g_free(wsource);
	g_free(wFilePath);
	CloseHandle(hFile);
	return gFilePath;
}

Could you tell me more about that? Is it a bug? The only thing I change is the LibRaw version, nothing else. In one case I can open wide char filename, in other case I can't.

My best regards, and thanks again for your work.

Forums: 

Could you please reformulate

Submitted by lexa on

Could you please reformulate your question? What is the problem and how it is related to code you quote?

-- Alex Tutubalin @LibRaw LLC

Ok Sorry if I'm unclear.

Submitted by lock on

Ok Sorry if I'm unclear.

My question is:

is there a difference between 0.20.2 and git version for the wide char handling ?

Because if I compile my soft with LibRaw <= 0.20.2 I have no problem. If I compile with LibRaw git version I have an error trying to opening CR2 Raw with wide char in paths.

My code was not very useful I must admit, this is just my code used to read wide char path.

It looks like (I'm not sure,

Submitted by lexa on

It looks like (I'm not sure, but the function names indicate this), you're converting filename from UTF-8 back to UTF-8.

If it worked at 0.20 then it was a miracle.

LibRaw has special LibRaw::open_file(wchar_t*) call under Win32, it is THE recommended way to go.

I do not know why UTF-8 filenames worked before. It should not.

-- Alex Tutubalin @LibRaw LLC

Hummm If I well recall,

Submitted by lock on

Hummm If I well recall, LibRaw::open_file(wchar_t*) does not work with msys2.

And my code is compiled through msys2.

Here my code

Submitted by lock on

Here my code

static int siril_libraw_open_file(libraw_data_t* rawdata, const char *name) {
/* libraw_open_wfile is not defined for all windows compilers */
#if defined(_WIN32) && !defined(__MINGW32__) && defined(_MSC_VER) && (_MSC_VER > 1310)
	wchar_t *wname;
 
	wname = g_utf8_to_utf16(name, -1, NULL, NULL, NULL);
	if (wname == NULL) {
		return 1;
	}
 
	int ret = libraw_open_wfile(rawdata, wname);
	g_free(wname);
	return ret;
#elif defined(_WIN32)
	gchar *localefilename = g_win32_locale_filename_from_utf8(name);
	int ret = libraw_open_file(rawdata, localefilename);
	g_free(localefilename);
	return ret;
#else
	return(libraw_open_file(rawdata, name));
#endif
}

As you can see I use it when it is allowed. But on my side I always compile with msys2... So I don't have API.

There is libraw_open_wfile()

Submitted by lexa on

There is libraw_open_wfile() that expects wchar_t filename.

LibRaw/github switched from iostreams-based LibRaw_abstract_datastream implementation to native-calls based implementation (LibRaw_bigfile_buffered_datastream) which is much faster under Win32.

Probably, your msys2's std::filebuf worked right with UTF-8 filenames, but this is miracle, not the expected behaviour.

-- Alex Tutubalin @LibRaw LLC

You may switch back via

Submitted by lexa on

You may switch back via defining LIBRAW_USE_DEPRECATED_IOSTREAMS_DATASTREAM

But this is temp. solution: it will work with LibRaw 0.21, but will be removed in future releases.

Also: LIBRAW_WIN32_UNICODEPATHS is set if:
# elif _GLIBCXX_HAVE__WFOPEN && _GLIBCXX_USE_WCHAR_T

So if such macros are defined for your runtime, unicode paths should work.

-- Alex Tutubalin @LibRaw LLC

I did love the miracle

Submitted by lock on

OK thanks for your kindness.
But with libraw_open_wfile

I get:

undefined reference to `libraw_open_wfile'

because in the LibRaw code there is

#if defined(_WIN32) && !defined(__MINGW32__) && defined(_MSC_VER) &&           \
    (_MSC_VER > 1310)
  int libraw_open_wfile(libraw_data_t *lr, const wchar_t *file)
  {
    if (!lr)
      return EINVAL;
    LibRaw *ip = (LibRaw *)lr->parent_class;
    return ip->open_file(file);
  }

So it can't work under msys2.

Oh thank you.

Submitted by lock on

Oh thank you.

If you fix it, you'll make me happy.
Thanks for your feedback

Please, let me know.
Cheers,

Followup:

Submitted by lexa on

Followup:
I do not have msys2/mingw/similar tools installed, so not tested.

MS Visual Studio (community) is free, so it makes sense to use native vendor compiler for Win32

-- Alex Tutubalin @LibRaw LLC

Thank you :)

Submitted by lock on

I will test it.

In fact our code is cross-compiled with CI pipelines: can't use visual studio. Especially because we build it for linux too.

As a result, you use very

Submitted by lexa on

As a result, you use very untested (and not supported) code path :)

-- Alex Tutubalin @LibRaw LLC

No problem: no risk no fun ;)

Submitted by lock on

No problem: no risk no fun ;).
I will let you know.

Last question. Which version I could consider for your change?

LIBRAW_VERSION > LIBRAW_MAKE_VERSION(0, 21, 0) ?

Yes (if you do not use

Submitted by lexa on

Yes (if you do not use earlier github snapshots)

-- Alex Tutubalin @LibRaw LLC

Thank you !

Submitted by lock on

Looks like your patch works like a charm.

Cheers,