Once again thanks for the instant response. My unix is as rusty as my c...had to look up mmap. Can't believe I didn't think to check the buffer stream for reference, face::palm.
just checked for ifname:
- these 'names' are under #ifdef DCRAW_VERBOSE, or #ifndef LIBRAW_LIBRARY_BUILD. The only exlusion is parse_external_jpeg(), but also after if(!ifp->ifname())... return;
Yes, you may implement LibRaw datastream without filename. Look into LibRaw_buffer_datastream as a sample.
Your datastream->fname() should return NULL in this case. This should work OK for most RAW data with one exception:
- old (not current) 'CHDK hack' (or DIAG RAW hack, cannot remember now) creates RAW file without any metadata, just sensor dump.
- all metadata is stored in filename.JPG (same file, different extension).
dcraw (and than LibRaw) is able to parse filename.raw + filename.JPG pair to get metadata (exposure parameters, etc) from the second file.
Another story is make_jas_stream() (from Jasper JPEG2000 library). This call is needed only if you want to decode Red cinema file. Again, you may implement it as 'return 0'; and all things will work OK (exception LIBRAW_EXCEPTION_DECODE_JPEG2000 raised within LibRaw, then converted to return code in unpack())
It looks like their are quite a few references to file path throughout the code despite having the file stream.
Long story short, will I be able to create a data stream that does not have access to a true file path? I should be able to get a file name, but unfortunately that's the extent. If it's not possible the rest of this post is irrelevant, just analysis of the impact of file path in the code. If it is possible, if you don't mind elaborating on the impact it would help me greatly. Thanks.
1) I noted a few line directives from dcraw.c. Do I need to be concerned about its use of file path or just within libraw?
LibRaw_abstract_datastream::subfile_open(const char *fn)
2) I only note this used in a parse_external_jpeg() which seems to be a hack to get metadata. It appears to be a warning at worse on a filename issue. Correct?
3) What is the significance of this function? It appears to open any file really, but has a thread lock. Correct?
#define ifname ((char*)libraw_internal_data.internal_data.input->fname())
4) This seems to be more of a concern, there are 17 references in dcraw_common, a lot in the same LibRaw_abstract_datastream::subfile_open(const char *fn) function and some related to minolta. There are many more references to its own similar 'ifname' in dcraw.c itself if I need to be concerned with that?
Thanks for the extremely fast response. I completely missed the difference in the subfile_open of the two streams in my first glance. So it should just be a simple exercise of replicating big with fdopen.
If I really want to strive for the speed in stream I could see if Android supports any method of FILE* to fstream. How much of an improvement do you think there is? It's Android, so linux, if the improvement only lies in Win.
LibRaw_file_datastream is based on С++ iostreams, while bigfile_datastream uses old plain FILE*.
Iostreams implementation is slightly faster (under Win32, at least), but there is no way to limit file buffer size. Also, iostreams implementation is limited to 2GB (or 4Gb??) files on some systems because of 32-bit nature.
Most raw files are much smaller than 2/4GB (and usually fit into 50-100M with very few exceptions), so (faster) iostream implementation is used. Bigfile_datastream is used for larger files to avoid too large buffers and 2/4Gb limit. The only files that go over limit are cinema files (Red's R3D files and other multi-frame files).
I'm puzzled again. When trying unpack() followed by subtrack_black() or not,
It makes no difference whether I set '''rawProcess.imgdata.params.use_camera_wb ''' to 0 or to 1 (i set that before unpack()), does this affect the raw_image data i'm using so far for my own processing or does this only affect imgdata.image after a call to raw2image() and/or dcraw_process? I only notice differences if i use imgdata.
The documentation was saying it affects unpack(), hence my asking.
Yes, I use averaged dark frame subtraction for night shots (not deep sky, but landscape lit by moon or milky way). It helps a lot with banding and other pattern noise.
Anyway, 'standard' (black+cblack) and averaged values are close enough to not change white balance much.
Well, in deep sky imaging we always deal with faint signal, we have rather dark pixels all over the place, therefore different "dark" (levels or averaged frame) subtraction methods really makes a different overall look, it's a critical step for a good final result. I'll give it a try with white balancing after dark frame subtraction.
Ok, thanks. Indeed, that's probably why I overlooked the dark / cdark business since I was already doing that on my own with the stacked (averaged) dark frames. However, I have yet to obtain somehow a similar white balancing.
On your opinion, would it still make sense to do my own averaged dark image subtraction (and no black / cblack business) and afterwards, apply the white balance coefficients "as shot"? The idea behind this is that subtracting a dark image (average or not) is, since I do the job "per channel", like a (x,y) dependent black and cblack subtraction, isn't it?
For subtract_black() call, the 'black level value' and 'black level data' are the same. Just different terms for the same thing to maintain 'writing style' (by not repeating same words many times).
Image fixed (you need to check []Display checkbox to display).
LibRaw's and dcraw's images (with same parameters) should be very close (or identical).
For unknown reason you use no_auto_bright for LibRaw and auto-brightening (default) for dcraw. That's why dcraw's histogram is shifted to the right.
I do not see any attached image.
Bayer image contains only one color component per pixel.
image[][4] array has 4 components per pixel (R,G,B,G2 or CMYG or RGBE).
raw2image() copies bayer data into image array
What do you mean for one component out of 4? I actually noticed this, but I don't really understand what's going on.
For testing (it uses extra memory, so not mobile friendly):
should work.
And replace with your own LibRaw_datastream implementation later.
Once again thanks for the instant response. My unix is as rusty as my c...had to look up mmap. Can't believe I didn't think to check the buffer stream for reference, face::palm.
BTW, for debug code try to
- mmap file
- pass mmap to LibRaw_buffer_datastream.
It should work under Linux (Android). It is not working right under Windows because Win32 memory mapping is some strange thing.
Followup:
just checked for ifname:
- these 'names' are under #ifdef DCRAW_VERBOSE, or #ifndef LIBRAW_LIBRARY_BUILD. The only exlusion is parse_external_jpeg(), but also after if(!ifp->ifname())... return;
Yes, you may implement LibRaw datastream without filename. Look into LibRaw_buffer_datastream as a sample.
Your datastream->fname() should return NULL in this case. This should work OK for most RAW data with one exception:
- old (not current) 'CHDK hack' (or DIAG RAW hack, cannot remember now) creates RAW file without any metadata, just sensor dump.
- all metadata is stored in filename.JPG (same file, different extension).
dcraw (and than LibRaw) is able to parse filename.raw + filename.JPG pair to get metadata (exposure parameters, etc) from the second file.
Another story is make_jas_stream() (from Jasper JPEG2000 library). This call is needed only if you want to decode Red cinema file. Again, you may implement it as 'return 0'; and all things will work OK (exception LIBRAW_EXCEPTION_DECODE_JPEG2000 raised within LibRaw, then converted to return code in unpack())
It looks like their are quite a few references to file path throughout the code despite having the file stream.
Long story short, will I be able to create a data stream that does not have access to a true file path? I should be able to get a file name, but unfortunately that's the extent. If it's not possible the rest of this post is irrelevant, just analysis of the impact of file path in the code. If it is possible, if you don't mind elaborating on the impact it would help me greatly. Thanks.
1) I noted a few line directives from dcraw.c. Do I need to be concerned about its use of file path or just within libraw?
LibRaw_abstract_datastream::subfile_open(const char *fn)2) I only note this used in a
parse_external_jpeg()which seems to be a hack to get metadata. It appears to be a warning at worse on a filename issue. Correct?3) What is the significance of this function? It appears to open any file really, but has a thread lock. Correct?
#define ifname ((char*)libraw_internal_data.internal_data.input->fname())
4) This seems to be more of a concern, there are 17 references in dcraw_common, a lot in the same
LibRaw_abstract_datastream::subfile_open(const char *fn)function and some related to minolta. There are many more references to its own similar 'ifname' in dcraw.c itself if I need to be concerned with that?Or use LibRaw::COLOR(row,col) call
Looks like it is actually the duty of imgdata.idata.filters to provide that order.
I'm coming back on that one, I wanted to know how imgdata.idata.cdesc returns the CFA pattern: Is it in clock-wise order or on a per row basis?:
Taking RGBG and respective 1-based indices is it:
or
It seems to me that it is in clock-wise order but I would like confirmation.
Cheers,
Thomas
I do not think you'll see any real difference in speed for compressed RAW files.
Thanks for the extremely fast response. I completely missed the difference in the subfile_open of the two streams in my first glance. So it should just be a simple exercise of replicating big with fdopen.
If I really want to strive for the speed in stream I could see if Android supports any method of FILE* to fstream. How much of an improvement do you think there is? It's Android, so linux, if the improvement only lies in Win.
Thanks again!
LibRaw_file_datastream is based on С++ iostreams, while bigfile_datastream uses old plain FILE*.
Iostreams implementation is slightly faster (under Win32, at least), but there is no way to limit file buffer size. Also, iostreams implementation is limited to 2GB (or 4Gb??) files on some systems because of 32-bit nature.
Most raw files are much smaller than 2/4GB (and usually fit into 50-100M with very few exceptions), so (faster) iostream implementation is used. Bigfile_datastream is used for larger files to avoid too large buffers and 2/4Gb limit. The only files that go over limit are cinema files (Red's R3D files and other multi-frame files).
Yes (use_camera_wb fallbacks to daylight if not camera wb present)
Are these the same white balance multipliers that are used when setting:
RawProcessor.imgdata.params.use_camera_wb=1;
?
subtract_black() works with imgdata.image
rawdata.raw_image is unaffected (to allow several processing /renderings/ of same raw data with different parameters)
I'm puzzled again. When trying unpack() followed by subtrack_black() or not,
It makes no difference whether I set '''rawProcess.imgdata.params.use_camera_wb ''' to 0 or to 1 (i set that before unpack()), does this affect the raw_image data i'm using so far for my own processing or does this only affect imgdata.image after a call to raw2image() and/or dcraw_process? I only notice differences if i use imgdata.
The documentation was saying it affects unpack(), hence my asking.
Yes, I use averaged dark frame subtraction for night shots (not deep sky, but landscape lit by moon or milky way). It helps a lot with banding and other pattern noise.
Anyway, 'standard' (black+cblack) and averaged values are close enough to not change white balance much.
Well, in deep sky imaging we always deal with faint signal, we have rather dark pixels all over the place, therefore different "dark" (levels or averaged frame) subtraction methods really makes a different overall look, it's a critical step for a good final result. I'll give it a try with white balancing after dark frame subtraction.
White balance should be applied after black level subtraction, indeed.
The difference between in-camera stored black level data and averaged black frame will affect only darkest parts of image, but not overall look.
Ok, thanks. Indeed, that's probably why I overlooked the dark / cdark business since I was already doing that on my own with the stacked (averaged) dark frames. However, I have yet to obtain somehow a similar white balancing.
On your opinion, would it still make sense to do my own averaged dark image subtraction (and no black / cblack business) and afterwards, apply the white balance coefficients "as shot"? The idea behind this is that subtracting a dark image (average or not) is, since I do the job "per channel", like a (x,y) dependent black and cblack subtraction, isn't it?
For subtract_black() call, the 'black level value' and 'black level data' are the same. Just different terms for the same thing to maintain 'writing style' (by not repeating same words many times).
In my pseudo code, you need to do this way:
LibRaw imgp; // Image processor class; imgp.open_file(...); imgp.unpack(); for(row=0; row < ..height; row++) for(col=0; col < .. width; col++) { int color_index = imgp.COLOR(row,col); int black_level_for_this_color = imgp.imgdata.color.black + imgp.imgdata.color.cblack[color_index]; int next_pixel_value_adjusted_for_black = VISIBLE_PIXEL(row,col) - black_level_for_this_color; if(next_pixel_value_adjusted_for_black < 0) next_pixel_value_adjusted_for_black = 0; }VISIBLE_PIXEL is defined in my previous message.
But for astrophotography it is better to average several dark frames (and do not use black/cblack[]) and subtract this value.
Pages