> The rendering tools will inevitably use differing sub-components.
> The only way to stop that s to forbid progress in rendering tools.
Once again you are trying to pass on the issue at hand. Problem is not what we want to do, but what DNG forces us to do - to ignore the contained information to assure normal processing.
> The DNG spec has been submitted to the ISO for consideration,
> so there is some discussion happening there, I assume.
Thank you to be so opened about the nature of that discussion. It sounds especially unusual given it comes from a person who founded the Digital Standards and Practices Committee with the American Society of Media Photographers.
> I can tell you that Adobe is most interested in speaking to people who are
> at least willing to acknowledge the good work they have already done
You seem not to realize that Adobe are not the only party here to decide. Somehow I doubt that you gave here an accurate profile to Adobe, too; and that you really understand that it is not just about Adobe doing good job with their applications, but about the future of our archives, and the quality of our conversions being on par to the money invested into the lenses and cameras and studios etc, all other things being equal. If only Adobe ACR/LR would be the best converter in the world, DNG might look much more attractive. But it is not.
> Camera Manufacturers unwillingness to support DNG is a political issue,
> not a technical one
Please try to support your statement with some evidence.
> As to usable, DNGs can be read by most Adobe products, Capture 1, Bibble 5,
> Lightzone, Apple on system level, and Windows on a system level,
> and nearly all DAM products.
To read is not everything. There are at least two questions here, firstly, is the rendition in third-party raw converters based solely on DNG data fields; and secondly, how different that rendition is when compared to the rendition from the original raw data.
> I am curious as to who exactly "we" is.
Once again in your own words - we like people who have done their homework first.
I remember well your writing from the past, September 2005: "While DNG is not guaranteed to be around forever it has a better chance than any particular individual camera format currently available". You continue than: "As more photographers see its benefits, the number of DNG files in existence will dwarf any other single format." That is, you were acknowledging that DNG can go to oblivion yet suggesting to use it as widely as possible. Sound logic and good advice.
The rendering tools will inevitably use differing sub-components. The only way to stop that s to forbid progress in rendering tools.
The DNG spec has been submitted to the ISO for consideration, so there is some discussion happening there, I assume.
Other people can have input, but I can tell you that Adobe is most interested in speaking to people who are at least willing to acknowledge the good work they have already done, and who seem to have a good grasp on the issues.
When you lead with a discussion of backward compatibility that, if implemented, could only serve to cripple development, that is an issue. Your discussion also seems to be unaware or inexplicably dismissive of the modifications done to the DNG spec that were made in the interest of third parties.
Camera Manufacturers unwillingness to support DNG is a political issue, not a technical one, as far as anyone has ever been able to show me. It would be nice if Adobe could order them to support DNG, but that's not possible.
As to usable, DNGs can be read by most Adobe products, Capture 1, Bibble 5, Lightzone, Apple on system level, and Windows on a system level, and nearly all DAM products. Calling it "unusable" is probably not a good way to start a discussion, and is, I would suggest, not accurate.
Keep in mind, that I do not speak for Adobe in any way. They may be happy to talk to you, but in general, they like people who have done their homework first.
> It's unavoidable that different applications will use different tools
> to render from raw.
There is a huge difference between "will" and "must".
Currently, one can't store just DNG. At the very least the original RAW should be embedded into the DNG, and thus the amount of storage and hence reliability of the archives in diminished.
At the bare minimum, a converter renders from the original RAW the same as from derived DNG; but in certain cases the rendition from the original RAW is better (we will discuss why in a due time). Some converters, important ones from camera manufactures being the prime concern to the end-users, do not acknowledge DNG files; and no roadmap to include those is announced. Those converters that can't render from the original RAW but can process derived DNG - I do not know of any worth the effort, the results from such converters are sub-par. This all means the DNG is very far from being accepted and needs a lot of work. The only way we see to improve DNG to the point of being usable and commonly accepted by the industry is through joint effort and wide discussion, as outlined in the article. Currently I do not see any blog or other Internet resource, not even a mailing list that is dedicated to such a purpose. All I see is special interests.
As far as "inaccuracies" that cause you needless work - you have yet to show any. It takes real work, experimentation and effort to understand RAW conversion. Such work can't be considered needless. Slogans are not going to work, same as printing press is not going to cure credit crisis.
It's unavoidable that different applications will use different tools to render from raw. No matter how the information is saved, there will be tags attached to images that other applications can't make use of. Get used to it. The most important thing is that data is preserved and accessible.
As to what is the use of DNG. I'd say the following:
1. An openly structured raw data container that can be used to hold the image data, as well as other stuff described below. Apple can currently make use of this to render raw filetypes it has never seen before.
2. A way to attach all the rendering instructions and color profiles needed to replicate a user's intention for image rendering. This is particularly important as users will begin to render images with custom profiles to create distinctive looks. If the profiles don't travel with the files, the rendering cannot be replicated, even on software of the same version. This is something you need to get a better understanding of.
3. A way to attach one or more fixed renderings to a raw file container that is not application-specific in any way, and lets the user employ multiple raw converters in an archive, and bring the renderings along. It also allows the use of third-party asset management software that can correctly render the files. Lastly, it enables the user to leave the program entirely and take the renderings with him.
4. A raw data container that metadata can be written to in a safe and documented manner, unlike proprietary raw files.
5,. A format to save rendered filetypes in when they are edited parametrically. This is starting to be a large problem, and is going to grow larger fast.
6. A way to make image files self-validating in a way that can survive subsequent changes to the metadata or image settings. No other graphic file format offers anything like this.
I consider these to be extremely important facets of image archiving.
It is not, and never has been, intended as a way to standardize the raw rendering output.
Again, any input on what specifically needs changing in the spec, or in the applications that create DNG is most welcome. If there is specific information you need from a DNG that you can't get, that would be helpful to know. It would also be good if you corrected the inaccuracies in your article before it spreads too far around the internets. Like I say, battling these misconceptions makes a lot of needless work for me.
If we need to override data supplied in one of the DNG fields using hard-wired data or specific algorithms, all derived from original raw, - what is the place for DNG than? Each DNG interpreter will do the interpretation in its own way, just as it happens with original raw.
When you say "the DNG proper" are you referring to the XMP space? Or to the Adobe-generated tags? Isn't the EXIF data available to the third-party raw converter from the Adobe-created DNG? How specifically would you suggest modifying the spec, and/or modifying the behavior of the DNG converter?
Is there anything else you can put your finger on that is concrete? I hope we can agree that it is impossible for a live parametric rendering to be identical from program to program, and that the DNG presents the best current possibility for attaching universally accessible renderings to raw image data. I hope we can agree that the 1.2 extension made the format considerably more friendly to third parties. I also hope we can agree that full backwards compatibility of rendering engines is not going to happen. I hope we can agree that forward compatibility is the most important goal, and that there are some really imaginative solutions included in the DNG that are not found anywhere else.
I also hope we can agree that most of the other points you've identified are ones that are coming from the camera manufacturer side, such as the difficulty of parsing some encoded or encrypted data, and that camera manufacturers don't publish the spectral response of their cameras.
I'd say that if we are on the same page, a more appropriate title for the article, regarding DNG, would be "A few small suggestions for the next DNG spec" rather than "Path to Nowhere." I'll spend a lot of time undoing misconceptions that are spread with analysis like this one.
A converter can't rely on just the information as contained in DNG proper, like the black level in the example above. Interpretation of the DNG data in a raw converter often depends on the specific make/model and some other EXIF tags; and worse, converters that can render quality images from DNG need to ignore certain DNG data substituting it with hard-wired data of their own, derived from hacking the original raw format.
>> We would appreciate if instead of simply ruling out the results of our experiment
>> as purposely skewed you would do those yourself and post the results
>> proving your point.
> The premise of what you are asking for is impossible and/or undesirable.
In the first experiment, you say to alter the EXIF data, and then seem surprised that it changes the output. What is unexpected here?
In the second example, are you saying that Adobe changes an EXIF value or improperly calculates the Black Level Tag? If so, that sounds like an implementation issue, not one of the spec. (Unless you see something in the spec that indicates that the right data is being tossed out by design.
Mostly what I think you're saying is that if you pull stuff out form DNG's as Adobe makes them, they don't render predictably. Again, that does not seem to be a huge surprise.
What is the problem one would encounter if he were not trying to intentionally damage the file?
Peter
- take any .CR2 from Canon camera
- convert it to .DNG by Adobe (i.e. 'right') converter
- replace Make/Model fields by Exiftool:
exiftool -Make=xxx -Model=yyy canon_eos_50d_01-02.dng
- try to render this file with some Non-Adobe DNG-enabled applications. I've used ACDSee Pro 2.5 and shadows are _very_ different. I'm sure, that LightZone (example of another non-Adobe DNG-enabled app) will show the same problem.
Updated: ACDSee and Lightzone show *right* rendering for Make/model preserved and incorrect one if Make/Model changed in Exif.
> I have no idea what you're driving at here. Are you saying that some DNG-aware
> applications can make use of EXIF
Sure!
I've done simple experiment in two minutes
- image from Canon 50D converted to DNG with Adobe 5.2.0.65 converter
Results
* Black Level tag value: 1023 1023 1023 1023
* Black level repeat dim: 2 2
* White level tag: 13200 (i.e. data is unscaled, 14-bit preserved as is)
But the real Black Level for this image is 511, not 1023.
Right RAW processor should:
- think: "this is Canon! we should calculate black level by left side of black mask"
- ignore black level tag from DNG
- calculate black level by averaging black (masked pixels)
If you strip all vendor-specific metadata from this file you will get unacceptable nonlinearity in shadows.
>Please, in your own words, take some time to try and understand what the article is about.
I'm pretty sure I understand where you're going with it. I just think the fundamental underlying assumptions are flawed. On top of that, you don't clearly outline what is an issue with the format, and what is an implementation issue.
>In doing so you will find the answers to many of your questions right in the article, including specific suggestions and the archival value of allowing more proprietary tags like colour decoding profiles of un-specified structure.
Well, the DNG spec does currently provide for anyone to embed profiles that describe the color any way they wish. If a manufacturer wanted to create his own tags and embed, or make profiles available, it could do that, as I understand it.
I think the far more important and challenging issue is how to deal with tags for further decoding and output of the files. As imaging technology moves from simply trying to render the files "correctly" to making more creative interpretations from the captured data, the challenge becomes attaching the custom profiles that can control that output, and keeping them available as the file gets passed from user to user. In this regard , the new 2.5-D profiles that the DNG allows is a really ingenious solution. I suggest checking them out.
Charting the spectral response of the camera's chip is far less important than enabling the user to create a portable parametric rendering. I
Once again, all parametric renderings will be engine-specific, and are likely to change somewhat over time, as applications are improved. (And those are implementation decisions made by the application's creator, and not an issue of format.) In fact, the rendering does not even *exist* until the raw data is loaded into the imaging pipeline along with the instructions. That's why the ability to attach a fixed rendering is so important. It is the *only* possible solution to create a totally predictable rendering, unless we stop software from developing any further.
Likewise, *backwards* compatibility of parametric rendering engines simply cannot work the way you outline unless:
1. The engines are not allowed to change and improve. or
2. A software manufacturer is obligated to update all old versions with new capability whenever it's developed.
Neither one of these will happen.
>By the way, in your opinion, shall we consider that the history of DNG starts with v. 1.2 and ACR v. 4.5?
I'd say 1.1, rather than 1.2. That was when all image data,a swell as all private makernotes was first transferred. Prior to that, the format definitely left some stuff behind.
>In that case, it is not a bad start. But what to do with the users invested into DNG at v.1.0 stage? Were they misled?
Sure, if that's the way you want to look at it. The first implementation had problems. It would be great if all 1.0 implementations were problem-free.
>We would appreciate if instead of simply ruling out the results of our experiment as purposely skewed you would do those yourself and post the results proving your point.
Did you understand my criticisms? The premise of what you are asking for is impossible and/or undesirable.
>As one of the methods of verification archival properties, we suggest you make a DNG out of a RAW file, and then delete from a copy of that DNG all vendor-specific EXIF fields to evaluate the difference in processing results between the "pure" DNG stripped of those vendor-specific EXIF fields and more complete data.
I have no idea what you're driving at here. Are you saying that some DNG-aware applications can make use of EXIF?
When you talk about "archival" what are you really talking about? Are you talking about access to the metadata, access to a particular rendering, or the survivability of the format.
The first of these is really the province of the camera maker, who encode the data, and could make it more discoverable. The second one can *only* be addressed by an embedded rendering. And the third is an area that you glossed over entirely in your discussion of the spec. The data validation tags in DNG 1.2 are a huge leap forward in archive stability, as they offer a way to determine the file integrity automatically. I suggest you check out this part of the spec.
>We would also love to see your priorities being sorted out. You begin your second paragraph with "most important", and continue with "most importantly" in the paragraph next to your last one. Such phrasing indicates that you were rather sloppy, again, to use a word of yours, while writing your comment.
Okay, fine, it was sloppy. I should have said that all of those things are important. It's important to read and understand something before you send out a purportedly comprehensive discussion of it. And it's also important to understand what is the problem with the spec, and what is the problem with any specific implementation. And it's important to understand what's possible, and what's not.
>It is our feeling that the readers of your comment would benefit from a disclosure more detailed than just The DAM Book author; like mentioning your special relations with Adobe in the status of Adobe Alpha Tester.
Okay, yes, I am an unpaid alpha tester for Adobe, which gives me access to the development teams. And I helped Adobe improve the DNG spec, with particular emphasis on addressing the issues raised by other software vendors, such as Bibble (ask Eric - he brought specific problems to light, and they were addressed in 1.2). I also focused on the ability to facilitate predictable rendering, which can only be achieved by use of embedded rendered data. In addition, Adobe occasionally pays me (poorly) to speak. Adobe also paid me to write a white paper on this subject.
Please, in your own words, take some time to try and understand what the article is about. In doing so you will find the answers to many of your questions right in the article, including specific suggestions and the archival value of allowing more proprietary tags like colour decoding profiles of un-specified structure.
By the way, in your opinion, shall we consider that the history of DNG starts with v. 1.2 and ACR v. 4.5? In that case, it is not a bad start. But what to do with the users invested into DNG at v.1.0 stage? Were they misled?
We would appreciate if instead of simply ruling out the results of our experiment as purposely skewed you would do those yourself and post the results proving your point. As one of the methods of verification archival properties, we suggest you make a DNG out of a RAW file, and then delete from a copy of that DNG all vendor-specific EXIF fields to evaluate the difference in processing results between the "pure" DNG stripped of those vendor-specific EXIF fields and more complete data.
We would also love to see your priorities being sorted out. You begin your second paragraph with "most important", and continue with "most importantly" in the paragraph next to your last one. Such phrasing indicates that you were rather sloppy, again, to use a word of yours, while writing your comment.
It is our feeling that the readers of your comment would benefit from a disclosure more detailed than just The DAM Book author; like mentioning your special relations with Adobe in the status of Adobe Alpha Tester.
1. The most important is that the author does not seem to have looked at the DNG spec 1.2 very closely. There were several important changes incorporated in that document that make the format considerably more friendly to third parties. These include:
a. Provisions for any software to attach its color decoding profiles into the DNG, even if they are entirely different than those used by Adobe.
b. A tag was added to indicate which application built the embedded rendering. This was added specifically because users will have DNG files created by different applications, and need to send them for output and get predictable results.
c. The licensing terms for Adobe's color matrix was updated to address concerns of third-party software, and licensing terms were added to enable third-party software to protect or distribute its own rendering profiles, if they so choose.
2. The article makes two claims about the rendering differences produced by DNG files processed by Adobe software. In the first, the author uses "an old version of the DNG converter." It would be helpful to know which one. Is it the first implementation, or a later one? There were acknowledged problems with the very first version that have been rectified in later versions. And what does "amplified" mean, exactly?
In the second claim, the author states that the renderings produced by different versions of Camera Raw are, in fact, somewhat different. This is pretty unsurprising, since the current software has had further engineering, and refinements to the rendering engine. For the comparison to be of real value, the author should document how he has created and applied the parametric rendering instructions for the comparison of the files. It's quite likely that the first image is taking advantage of improvements in sharpening and shadow rendering that were built into the later version but not available to version 3.x (he does not document which versions he used to test with). The 5.x version is also highly likely to be using a different profile to render, although he could choose to use the old one if he wishes.
Either the author tested in the manner he did to purposely skew the results, or he does not understand rendering and forward compatibility very well. For instance, it would be much more relevant to know if a DNG made in ACR 3.6 looked the same when rendered out of 5.2, and what rendering instructions were present. The method he chose will clearly employ the improvements in the current version when rendering the file, including sharpening, vibrance, new profiles and other rendering tools. These will not be available in all past versions, just as some TIFF layer adjustments from Photoshop CS3 can't be understood by Photoshop CS, and so a different result will be produced. At minimum, this needs to be understood.
Most importantly, the author's criticism's here seems to be confused by the capabilities of the specification, and Adobe's implementation of the spec. He also does not seem to understand that the universality of DNG is that of a container, not a particular rendering. The *only* way to get a universal static rendering of the file it to embed a non engine-specific one: this has also been greatly enhanced by the tools in the 1.2 spec, which the author should take some time to try and understand. This kind of unclear thinking, along with sloppy research from the primary documents, does not move the discussion forward.
I would be interested in hearing specific suggestions regarding these issues, if they are informed by clear thinking about what is the job of the specification, and what is the job of the software, and how to handle forward compatibility in a parametric image editing environment. While there are surely a number of important issues to address - I can think of at least a half a dozen - very little in this article approaches that.
Peter Krogh
Author, The DAM Book, Digital Asset Management for Photographers
As you can see in RAW data histograms, Green and Red are balanced perfectly (without any filters) in incadescent bulb light. You don't want to use any filters in this low-light condition, anyway.
Also, I don't think that 'gamut' is a good word in describing digital camera. Camera is sensitive to any visible light (to infrared - too), so camera gamut is equal (or broader) to human gamut.
We should describe camera behavior in terms of metamerism: some colors cannot be separated (i.e. same RGB(G) response for different colors)
This is a real dilemma - they could make two sensor types, one with narrow-bandpass colour filters, one with broad spectrum RGB. You would end up with a low ISO sensor offering maximum colour discrimination and Delta-E accuracy (but of course the potential for greater Delta-E errors if software conversion was not precisely matched to the transmission curves); and a high ISO sensor where the overlapping curves could reduce colour discrimination, relying on software to restore it.
I always felt that the RGB filters used on the KM7D/5D were narrower in band than the filters used on the same sensor by Nikon (D70 etc), allowing the lower ISO 100 setting and also giving the distinctive colour of the 7D/5D. I have also always thought that Canon's (early) RGB filters must be weak, helping reduce noise but giving a colour quality I never liked as much. The wide gamut captured by Canon models compared to the 7D/5D also points to this - narrow cut filters will, of course, limit the overall gamut but improve discrimination and accuracy within that gamut.
Did you mean green and blue are perfectly balanced in tungsten light, not green and red?
I have a large stock of ex-Minolta 85B (daylight to tungsten) conversion filters here, never been able to sell them. I gave away 200 40.5mm ones last week (40.5mm is not much use!). I may try the 85B in daylight, using the camera's tungsten preset, to see if it has some benefits.
Sure, pure CC30-40M (for different cameras) balances Green with Blue. Red channel remains half-stop underexposed.
Combination of something like СС30M plus CC15-20R should result in better channel balance in daylight. The cost is extra filter with extra glare (and one more slot in filter holder).
I'll try this combination when Moscow weather become more sunny :)
Filters on sensor is result of some compromise:
* For shooting in tungsten light (espessially, in 'home' bulbs, not high-temperature halogen ones) you need to lower sensitivity of red. If not, then skin tones will be overexposed. Look at the last chapter of my 'White balance problems' article ( http://www.libraw.org/articles/white-balance-in-digital-cameras.html ): green and red (without filters) are near-perfectly balanced under incadescent bulb.
* Also, there is 'High ISO race' now: last Canon/Nikon models have ISO 25600 sensitivity. To lower the signal/noise level, manufactures need to widen filter response curves. And, surely, color characteristics on Canon 5D-2 or Nikon D3 is much worse than on previous generation.
Hope, that 'high color quality' race will start sometime later.....
I realise that, but the blue channel will not be proportionately affected. A mix of CCR and CCM would be needed to equalise the balance, just wondered in view of the serious effect of red channel noise on blue skies whether CCR might not be a better option.
Some time ago (2-3 years) I did some tests using 80B filters for tungsten, and found that between 1 and 1.5 stops of highlight headroom was recovered by shooting with 80B (based on ACR conversions of the time, KM 7D). Iliah's findings show that maybe ACR was pre-boosting the red channel anyway. A different filter (not 80B) might have enabled equal channel gain in raw conversion. I worked in the 1990s with Leaf and other scanning backs, researching hot mirror filters and light balance filters to recover better colour information.
It surprises me that this kind of work (I'm just a photographer, my wife is a colour scientist) does not seem to have influenced the design of RGB filters for Bayer-type sensors. The sensor itself, unfiltered, is highly sensitive to red. But gradation in the red channel is determined by the blue component more than any other. The work of Dr Andrew Stevens in the early 1990s has also been ignored - he showed very effectively how only two coloured filters were required to produce full colour, not three. Andrew was a colour scientist with Kodak but I don't know where he is now. He was able to produce pseudo-isocolour from a panchromatic film and an orange filter, plus an exposure level - nothing more needed (two exposures)!
LibRaw is for software developers. Many users of digiKam, Krita and some other RAW processing software are happy with these programs (which, in turn, uses LibRaw via libkdcraw).
Anyway, LibRaw distribution includes several command-line tools. Take a look to half_mt utility which is similar to 'dcraw -h', but much faster in batch processing on multi-CPU/multicore machines
A magenta filter will crosstalk more than a red filter, and CC40R or 30R would be easily obtained. Have you tried to make up a filter pack which will equalize the highlight clip position for all three channels in daylight?
Second question - to what extent do you think the IR-cut filters used are attenuating the far end of the red response, and might this be in part responsible for the low red sensitivity?
Yes, It works now with LX3 files. For first time I am seeing my true raw as it should be seen.
I will update you once I have a decent UI that uses libraw and it's up on sourceforge.
I've tried LX3 samples from photographyblog and process them without any problems on LibRaw 0.6.1.
Anyway, there is some strange problems when processing Panasonic files (in my tests - files from FZ28) on multi-threaded LibRaw version, so I'll check it.
> The rendering tools will inevitably use differing sub-components.
> The only way to stop that s to forbid progress in rendering tools.
Once again you are trying to pass on the issue at hand. Problem is not what we want to do, but what DNG forces us to do - to ignore the contained information to assure normal processing.
> The DNG spec has been submitted to the ISO for consideration,
> so there is some discussion happening there, I assume.
Thank you to be so opened about the nature of that discussion. It sounds especially unusual given it comes from a person who founded the Digital Standards and Practices Committee with the American Society of Media Photographers.
> I can tell you that Adobe is most interested in speaking to people who are
> at least willing to acknowledge the good work they have already done
You seem not to realize that Adobe are not the only party here to decide. Somehow I doubt that you gave here an accurate profile to Adobe, too; and that you really understand that it is not just about Adobe doing good job with their applications, but about the future of our archives, and the quality of our conversions being on par to the money invested into the lenses and cameras and studios etc, all other things being equal. If only Adobe ACR/LR would be the best converter in the world, DNG might look much more attractive. But it is not.
> Camera Manufacturers unwillingness to support DNG is a political issue,
> not a technical one
Please try to support your statement with some evidence.
> As to usable, DNGs can be read by most Adobe products, Capture 1, Bibble 5,
> Lightzone, Apple on system level, and Windows on a system level,
> and nearly all DAM products.
To read is not everything. There are at least two questions here, firstly, is the rendition in third-party raw converters based solely on DNG data fields; and secondly, how different that rendition is when compared to the rendition from the original raw data.
> I am curious as to who exactly "we" is.
Once again in your own words - we like people who have done their homework first.
I remember well your writing from the past, September 2005: "While DNG is not guaranteed to be around forever it has a better chance than any particular individual camera format currently available". You continue than: "As more photographers see its benefits, the number of DNG files in existence will dwarf any other single format." That is, you were acknowledging that DNG can go to oblivion yet suggesting to use it as widely as possible. Sound logic and good advice.
The rendering tools will inevitably use differing sub-components. The only way to stop that s to forbid progress in rendering tools.
The DNG spec has been submitted to the ISO for consideration, so there is some discussion happening there, I assume.
Other people can have input, but I can tell you that Adobe is most interested in speaking to people who are at least willing to acknowledge the good work they have already done, and who seem to have a good grasp on the issues.
When you lead with a discussion of backward compatibility that, if implemented, could only serve to cripple development, that is an issue. Your discussion also seems to be unaware or inexplicably dismissive of the modifications done to the DNG spec that were made in the interest of third parties.
Camera Manufacturers unwillingness to support DNG is a political issue, not a technical one, as far as anyone has ever been able to show me. It would be nice if Adobe could order them to support DNG, but that's not possible.
As to usable, DNGs can be read by most Adobe products, Capture 1, Bibble 5, Lightzone, Apple on system level, and Windows on a system level, and nearly all DAM products. Calling it "unusable" is probably not a good way to start a discussion, and is, I would suggest, not accurate.
Keep in mind, that I do not speak for Adobe in any way. They may be happy to talk to you, but in general, they like people who have done their homework first.
I am curious as to who exactly "we" is.
Peter
> It's unavoidable that different applications will use different tools
> to render from raw.
There is a huge difference between "will" and "must".
Currently, one can't store just DNG. At the very least the original RAW should be embedded into the DNG, and thus the amount of storage and hence reliability of the archives in diminished.
At the bare minimum, a converter renders from the original RAW the same as from derived DNG; but in certain cases the rendition from the original RAW is better (we will discuss why in a due time). Some converters, important ones from camera manufactures being the prime concern to the end-users, do not acknowledge DNG files; and no roadmap to include those is announced. Those converters that can't render from the original RAW but can process derived DNG - I do not know of any worth the effort, the results from such converters are sub-par. This all means the DNG is very far from being accepted and needs a lot of work. The only way we see to improve DNG to the point of being usable and commonly accepted by the industry is through joint effort and wide discussion, as outlined in the article. Currently I do not see any blog or other Internet resource, not even a mailing list that is dedicated to such a purpose. All I see is special interests.
As far as "inaccuracies" that cause you needless work - you have yet to show any. It takes real work, experimentation and effort to understand RAW conversion. Such work can't be considered needless. Slogans are not going to work, same as printing press is not going to cure credit crisis.
It's unavoidable that different applications will use different tools to render from raw. No matter how the information is saved, there will be tags attached to images that other applications can't make use of. Get used to it. The most important thing is that data is preserved and accessible.
As to what is the use of DNG. I'd say the following:
1. An openly structured raw data container that can be used to hold the image data, as well as other stuff described below. Apple can currently make use of this to render raw filetypes it has never seen before.
2. A way to attach all the rendering instructions and color profiles needed to replicate a user's intention for image rendering. This is particularly important as users will begin to render images with custom profiles to create distinctive looks. If the profiles don't travel with the files, the rendering cannot be replicated, even on software of the same version. This is something you need to get a better understanding of.
3. A way to attach one or more fixed renderings to a raw file container that is not application-specific in any way, and lets the user employ multiple raw converters in an archive, and bring the renderings along. It also allows the use of third-party asset management software that can correctly render the files. Lastly, it enables the user to leave the program entirely and take the renderings with him.
4. A raw data container that metadata can be written to in a safe and documented manner, unlike proprietary raw files.
5,. A format to save rendered filetypes in when they are edited parametrically. This is starting to be a large problem, and is going to grow larger fast.
6. A way to make image files self-validating in a way that can survive subsequent changes to the metadata or image settings. No other graphic file format offers anything like this.
I consider these to be extremely important facets of image archiving.
It is not, and never has been, intended as a way to standardize the raw rendering output.
Again, any input on what specifically needs changing in the spec, or in the applications that create DNG is most welcome. If there is specific information you need from a DNG that you can't get, that would be helpful to know. It would also be good if you corrected the inaccuracies in your article before it spreads too far around the internets. Like I say, battling these misconceptions makes a lot of needless work for me.
;-)
Peter
If we need to override data supplied in one of the DNG fields using hard-wired data or specific algorithms, all derived from original raw, - what is the place for DNG than? Each DNG interpreter will do the interpretation in its own way, just as it happens with original raw.
When you say "the DNG proper" are you referring to the XMP space? Or to the Adobe-generated tags? Isn't the EXIF data available to the third-party raw converter from the Adobe-created DNG? How specifically would you suggest modifying the spec, and/or modifying the behavior of the DNG converter?
Is there anything else you can put your finger on that is concrete? I hope we can agree that it is impossible for a live parametric rendering to be identical from program to program, and that the DNG presents the best current possibility for attaching universally accessible renderings to raw image data. I hope we can agree that the 1.2 extension made the format considerably more friendly to third parties. I also hope we can agree that full backwards compatibility of rendering engines is not going to happen. I hope we can agree that forward compatibility is the most important goal, and that there are some really imaginative solutions included in the DNG that are not found anywhere else.
I also hope we can agree that most of the other points you've identified are ones that are coming from the camera manufacturer side, such as the difficulty of parsing some encoded or encrypted data, and that camera manufacturers don't publish the spectral response of their cameras.
I'd say that if we are on the same page, a more appropriate title for the article, regarding DNG, would be "A few small suggestions for the next DNG spec" rather than "Path to Nowhere." I'll spend a lot of time undoing misconceptions that are spread with analysis like this one.
Peter
> I'm still not sure what you're driving at.
A converter can't rely on just the information as contained in DNG proper, like the black level in the example above. Interpretation of the DNG data in a raw converter often depends on the specific make/model and some other EXIF tags; and worse, converters that can render quality images from DNG need to ignore certain DNG data substituting it with hard-wired data of their own, derived from hacking the original raw format.
>> We would appreciate if instead of simply ruling out the results of our experiment
>> as purposely skewed you would do those yourself and post the results
>> proving your point.
> The premise of what you are asking for is impossible and/or undesirable.
Please explain why.
I'm still not sure what you're driving at.
In the first experiment, you say to alter the EXIF data, and then seem surprised that it changes the output. What is unexpected here?
In the second example, are you saying that Adobe changes an EXIF value or improperly calculates the Black Level Tag? If so, that sounds like an implementation issue, not one of the spec. (Unless you see something in the spec that indicates that the right data is being tossed out by design.
Mostly what I think you're saying is that if you pull stuff out form DNG's as Adobe makes them, they don't render predictably. Again, that does not seem to be a huge surprise.
What is the problem one would encounter if he were not trying to intentionally damage the file?
Peter
- take any .CR2 from Canon camera
- convert it to .DNG by Adobe (i.e. 'right') converter
- replace Make/Model fields by Exiftool:
exiftool -Make=xxx -Model=yyy canon_eos_50d_01-02.dng
- try to render this file with some Non-Adobe DNG-enabled applications. I've used ACDSee Pro 2.5 and shadows are _very_ different. I'm sure, that LightZone (example of another non-Adobe DNG-enabled app) will show the same problem.
Updated: ACDSee and Lightzone show *right* rendering for Make/model preserved and incorrect one if Make/Model changed in Exif.
> I have no idea what you're driving at here. Are you saying that some DNG-aware
> applications can make use of EXIF
Sure!
I've done simple experiment in two minutes
- image from Canon 50D converted to DNG with Adobe 5.2.0.65 converter
Results
* Black Level tag value: 1023 1023 1023 1023
* Black level repeat dim: 2 2
* White level tag: 13200 (i.e. data is unscaled, 14-bit preserved as is)
But the real Black Level for this image is 511, not 1023.
Right RAW processor should:
- think: "this is Canon! we should calculate black level by left side of black mask"
- ignore black level tag from DNG
- calculate black level by averaging black (masked pixels)
If you strip all vendor-specific metadata from this file you will get unacceptable nonlinearity in shadows.
>Please, in your own words, take some time to try and understand what the article is about.
I'm pretty sure I understand where you're going with it. I just think the fundamental underlying assumptions are flawed. On top of that, you don't clearly outline what is an issue with the format, and what is an implementation issue.
>In doing so you will find the answers to many of your questions right in the article, including specific suggestions and the archival value of allowing more proprietary tags like colour decoding profiles of un-specified structure.
Well, the DNG spec does currently provide for anyone to embed profiles that describe the color any way they wish. If a manufacturer wanted to create his own tags and embed, or make profiles available, it could do that, as I understand it.
I think the far more important and challenging issue is how to deal with tags for further decoding and output of the files. As imaging technology moves from simply trying to render the files "correctly" to making more creative interpretations from the captured data, the challenge becomes attaching the custom profiles that can control that output, and keeping them available as the file gets passed from user to user. In this regard , the new 2.5-D profiles that the DNG allows is a really ingenious solution. I suggest checking them out.
Charting the spectral response of the camera's chip is far less important than enabling the user to create a portable parametric rendering. I
Once again, all parametric renderings will be engine-specific, and are likely to change somewhat over time, as applications are improved. (And those are implementation decisions made by the application's creator, and not an issue of format.) In fact, the rendering does not even *exist* until the raw data is loaded into the imaging pipeline along with the instructions. That's why the ability to attach a fixed rendering is so important. It is the *only* possible solution to create a totally predictable rendering, unless we stop software from developing any further.
Likewise, *backwards* compatibility of parametric rendering engines simply cannot work the way you outline unless:
1. The engines are not allowed to change and improve. or
2. A software manufacturer is obligated to update all old versions with new capability whenever it's developed.
Neither one of these will happen.
>By the way, in your opinion, shall we consider that the history of DNG starts with v. 1.2 and ACR v. 4.5?
I'd say 1.1, rather than 1.2. That was when all image data,a swell as all private makernotes was first transferred. Prior to that, the format definitely left some stuff behind.
>In that case, it is not a bad start. But what to do with the users invested into DNG at v.1.0 stage? Were they misled?
Sure, if that's the way you want to look at it. The first implementation had problems. It would be great if all 1.0 implementations were problem-free.
>We would appreciate if instead of simply ruling out the results of our experiment as purposely skewed you would do those yourself and post the results proving your point.
Did you understand my criticisms? The premise of what you are asking for is impossible and/or undesirable.
>As one of the methods of verification archival properties, we suggest you make a DNG out of a RAW file, and then delete from a copy of that DNG all vendor-specific EXIF fields to evaluate the difference in processing results between the "pure" DNG stripped of those vendor-specific EXIF fields and more complete data.
I have no idea what you're driving at here. Are you saying that some DNG-aware applications can make use of EXIF?
When you talk about "archival" what are you really talking about? Are you talking about access to the metadata, access to a particular rendering, or the survivability of the format.
The first of these is really the province of the camera maker, who encode the data, and could make it more discoverable. The second one can *only* be addressed by an embedded rendering. And the third is an area that you glossed over entirely in your discussion of the spec. The data validation tags in DNG 1.2 are a huge leap forward in archive stability, as they offer a way to determine the file integrity automatically. I suggest you check out this part of the spec.
>We would also love to see your priorities being sorted out. You begin your second paragraph with "most important", and continue with "most importantly" in the paragraph next to your last one. Such phrasing indicates that you were rather sloppy, again, to use a word of yours, while writing your comment.
Okay, fine, it was sloppy. I should have said that all of those things are important. It's important to read and understand something before you send out a purportedly comprehensive discussion of it. And it's also important to understand what is the problem with the spec, and what is the problem with any specific implementation. And it's important to understand what's possible, and what's not.
>It is our feeling that the readers of your comment would benefit from a disclosure more detailed than just The DAM Book author; like mentioning your special relations with Adobe in the status of Adobe Alpha Tester.
Okay, yes, I am an unpaid alpha tester for Adobe, which gives me access to the development teams. And I helped Adobe improve the DNG spec, with particular emphasis on addressing the issues raised by other software vendors, such as Bibble (ask Eric - he brought specific problems to light, and they were addressed in 1.2). I also focused on the ability to facilitate predictable rendering, which can only be achieved by use of embedded rendered data. In addition, Adobe occasionally pays me (poorly) to speak. Adobe also paid me to write a white paper on this subject.
http://www.adobe.com/digitalimag/ps_pro_primers.html
I have also licensed photographs to Adobe in my capacity as a professional photographer.
Peter Krogh
Dear Peter,
Please, in your own words, take some time to try and understand what the article is about. In doing so you will find the answers to many of your questions right in the article, including specific suggestions and the archival value of allowing more proprietary tags like colour decoding profiles of un-specified structure.
By the way, in your opinion, shall we consider that the history of DNG starts with v. 1.2 and ACR v. 4.5? In that case, it is not a bad start. But what to do with the users invested into DNG at v.1.0 stage? Were they misled?
We would appreciate if instead of simply ruling out the results of our experiment as purposely skewed you would do those yourself and post the results proving your point. As one of the methods of verification archival properties, we suggest you make a DNG out of a RAW file, and then delete from a copy of that DNG all vendor-specific EXIF fields to evaluate the difference in processing results between the "pure" DNG stripped of those vendor-specific EXIF fields and more complete data.
We would also love to see your priorities being sorted out. You begin your second paragraph with "most important", and continue with "most importantly" in the paragraph next to your last one. Such phrasing indicates that you were rather sloppy, again, to use a word of yours, while writing your comment.
It is our feeling that the readers of your comment would benefit from a disclosure more detailed than just The DAM Book author; like mentioning your special relations with Adobe in the status of Adobe Alpha Tester.
I see a few fundamentals issues with the article.
1. The most important is that the author does not seem to have looked at the DNG spec 1.2 very closely. There were several important changes incorporated in that document that make the format considerably more friendly to third parties. These include:
a. Provisions for any software to attach its color decoding profiles into the DNG, even if they are entirely different than those used by Adobe.
b. A tag was added to indicate which application built the embedded rendering. This was added specifically because users will have DNG files created by different applications, and need to send them for output and get predictable results.
c. The licensing terms for Adobe's color matrix was updated to address concerns of third-party software, and licensing terms were added to enable third-party software to protect or distribute its own rendering profiles, if they so choose.
2. The article makes two claims about the rendering differences produced by DNG files processed by Adobe software. In the first, the author uses "an old version of the DNG converter." It would be helpful to know which one. Is it the first implementation, or a later one? There were acknowledged problems with the very first version that have been rectified in later versions. And what does "amplified" mean, exactly?
In the second claim, the author states that the renderings produced by different versions of Camera Raw are, in fact, somewhat different. This is pretty unsurprising, since the current software has had further engineering, and refinements to the rendering engine. For the comparison to be of real value, the author should document how he has created and applied the parametric rendering instructions for the comparison of the files. It's quite likely that the first image is taking advantage of improvements in sharpening and shadow rendering that were built into the later version but not available to version 3.x (he does not document which versions he used to test with). The 5.x version is also highly likely to be using a different profile to render, although he could choose to use the old one if he wishes.
Either the author tested in the manner he did to purposely skew the results, or he does not understand rendering and forward compatibility very well. For instance, it would be much more relevant to know if a DNG made in ACR 3.6 looked the same when rendered out of 5.2, and what rendering instructions were present. The method he chose will clearly employ the improvements in the current version when rendering the file, including sharpening, vibrance, new profiles and other rendering tools. These will not be available in all past versions, just as some TIFF layer adjustments from Photoshop CS3 can't be understood by Photoshop CS, and so a different result will be produced. At minimum, this needs to be understood.
Most importantly, the author's criticism's here seems to be confused by the capabilities of the specification, and Adobe's implementation of the spec. He also does not seem to understand that the universality of DNG is that of a container, not a particular rendering. The *only* way to get a universal static rendering of the file it to embed a non engine-specific one: this has also been greatly enhanced by the tools in the 1.2 spec, which the author should take some time to try and understand. This kind of unclear thinking, along with sloppy research from the primary documents, does not move the discussion forward.
I would be interested in hearing specific suggestions regarding these issues, if they are informed by clear thinking about what is the job of the specification, and what is the job of the software, and how to handle forward compatibility in a parametric image editing environment. While there are surely a number of important issues to address - I can think of at least a half a dozen - very little in this article approaches that.
Peter Krogh
Author, The DAM Book, Digital Asset Management for Photographers
I now have a D90, which is supported, so can use rawanalyze without conversion.
A good and interesting article in general. Interesting comparisons of different processors.
Noise reduction before debayering is clearly the way to go, especially if read noise can be estimated and removed from masked pixels at the same time.
Constructing a noise profile, per camera model and per ISO setting, would be ideal. (Commercial programs like Noise Ninja do this, for example).
I notice once again that ETTL is being described as ETTR, but that doesn't impact the main point of the article.
As you can see in RAW data histograms, Green and Red are balanced perfectly (without any filters) in incadescent bulb light. You don't want to use any filters in this low-light condition, anyway.
Also, I don't think that 'gamut' is a good word in describing digital camera. Camera is sensitive to any visible light (to infrared - too), so camera gamut is equal (or broader) to human gamut.
We should describe camera behavior in terms of metamerism: some colors cannot be separated (i.e. same RGB(G) response for different colors)
This is a real dilemma - they could make two sensor types, one with narrow-bandpass colour filters, one with broad spectrum RGB. You would end up with a low ISO sensor offering maximum colour discrimination and Delta-E accuracy (but of course the potential for greater Delta-E errors if software conversion was not precisely matched to the transmission curves); and a high ISO sensor where the overlapping curves could reduce colour discrimination, relying on software to restore it.
I always felt that the RGB filters used on the KM7D/5D were narrower in band than the filters used on the same sensor by Nikon (D70 etc), allowing the lower ISO 100 setting and also giving the distinctive colour of the 7D/5D. I have also always thought that Canon's (early) RGB filters must be weak, helping reduce noise but giving a colour quality I never liked as much. The wide gamut captured by Canon models compared to the 7D/5D also points to this - narrow cut filters will, of course, limit the overall gamut but improve discrimination and accuracy within that gamut.
Did you mean green and blue are perfectly balanced in tungsten light, not green and red?
I have a large stock of ex-Minolta 85B (daylight to tungsten) conversion filters here, never been able to sell them. I gave away 200 40.5mm ones last week (40.5mm is not much use!). I may try the 85B in daylight, using the camera's tungsten preset, to see if it has some benefits.
David
Sure, pure CC30-40M (for different cameras) balances Green with Blue. Red channel remains half-stop underexposed.
Combination of something like СС30M plus CC15-20R should result in better channel balance in daylight. The cost is extra filter with extra glare (and one more slot in filter holder).
I'll try this combination when Moscow weather become more sunny :)
Filters on sensor is result of some compromise:
* For shooting in tungsten light (espessially, in 'home' bulbs, not high-temperature halogen ones) you need to lower sensitivity of red. If not, then skin tones will be overexposed. Look at the last chapter of my 'White balance problems' article ( http://www.libraw.org/articles/white-balance-in-digital-cameras.html ): green and red (without filters) are near-perfectly balanced under incadescent bulb.
* Also, there is 'High ISO race' now: last Canon/Nikon models have ISO 25600 sensitivity. To lower the signal/noise level, manufactures need to widen filter response curves. And, surely, color characteristics on Canon 5D-2 or Nikon D3 is much worse than on previous generation.
Hope, that 'high color quality' race will start sometime later.....
I realise that, but the blue channel will not be proportionately affected. A mix of CCR and CCM would be needed to equalise the balance, just wondered in view of the serious effect of red channel noise on blue skies whether CCR might not be a better option.
Some time ago (2-3 years) I did some tests using 80B filters for tungsten, and found that between 1 and 1.5 stops of highlight headroom was recovered by shooting with 80B (based on ACR conversions of the time, KM 7D). Iliah's findings show that maybe ACR was pre-boosting the red channel anyway. A different filter (not 80B) might have enabled equal channel gain in raw conversion. I worked in the 1990s with Leaf and other scanning backs, researching hot mirror filters and light balance filters to recover better colour information.
It surprises me that this kind of work (I'm just a photographer, my wife is a colour scientist) does not seem to have influenced the design of RGB filters for Bayer-type sensors. The sensor itself, unfiltered, is highly sensitive to red. But gradation in the red channel is determined by the blue component more than any other. The work of Dr Andrew Stevens in the early 1990s has also been ignored - he showed very effectively how only two coloured filters were required to produce full colour, not three. Andrew was a colour scientist with Kodak but I don't know where he is now. He was able to produce pseudo-isocolour from a panchromatic film and an orange filter, plus an exposure level - nothing more needed (two exposures)!
David
LibRaw is for software developers. Many users of digiKam, Krita and some other RAW processing software are happy with these programs (which, in turn, uses LibRaw via libkdcraw).
Anyway, LibRaw distribution includes several command-line tools. Take a look to half_mt utility which is similar to 'dcraw -h', but much faster in batch processing on multi-CPU/multicore machines
Magenta is 'Minus Green filter', but Red is 'Minus Green and Minus Blue'.
In daylight situation CC30M-CC40M produces near perfect sensitivity balance on my cameras.
A magenta filter will crosstalk more than a red filter, and CC40R or 30R would be easily obtained. Have you tried to make up a filter pack which will equalize the highlight clip position for all three channels in daylight?
Second question - to what extent do you think the IR-cut filters used are attenuating the far end of the red response, and might this be in part responsible for the low red sensitivity?
Yes, It works now with LX3 files. For first time I am seeing my true raw as it should be seen.
I will update you once I have a decent UI that uses libraw and it's up on sourceforge.
Thanks,
We've found and fixed bug in Panasonic .RW2 files processing. This bug affects only thread-safe version of library.
Could you please test LibRaw 0.6.2 on your files?
I've tried LX3 samples from photographyblog and process them without any problems on LibRaw 0.6.1.
Anyway, there is some strange problems when processing Panasonic files (in my tests - files from FZ28) on multi-threaded LibRaw version, so I'll check it.
You also may check new LibRaw release (sorry, source code only): http://www.libraw.org/blog/libraw-062-beta1.html
Pages