Do I lose anything converting to DNG?

Question

I'd like to convert all my ARW (Sony) files to DNG for several reasons, but the only thing keeping me back from doing it is the fear of losing useful metadata during the conversion. I know that the image it's self is completely safe during the conversion but what about proprietary maker noteS? Will those stay too? And even if they do, are they actually useful and can be actually used by something other then Sony's own RAW converter?

P.S. I'm using Ubuntu so I'd prefer to use KipiPlugins' DNG converter because from what I understand it does everything the Adobe converter does, but if I'm wrong I can use Adobe's instead.

EDIT:

It looks like the LensID does carry over! With the ARW and the DNG files converted by Kipiplugins' built in converter, info is stored as "Exif.Sony.0x___". With Adobe's official converter, it moves the info to more logically named fields in the XMP, such as "LensID" and "Lens".

Now that I've figured it out, I think I might use Adobe's Official converter, just because I prefer how it reorganizes the Exif Data.

I'm not fully convinced though and I'd like to hear others take on all this.

That would not answer his questions, as (missing/additional) metadata tends to be invisible until seen with the right program :) — Leonidas, Dec 30 '10 at 12:39
@Leonidas - Presumably he has something to look at the EXIF or he wouldn't care in the first place... :) — Joanne C, Dec 30 '10 at 14:38
I just looked at both a ARW file and a DNG file's EXIF, and they both have the same "Exif.Sony.0x____" fields with the same data. So does this mean everything is being carried over or just that my Exif view doesn't see the proprietary Exif stuff? — RPG Master, Dec 30 '10 at 23:44
Also, if only Sony's Raw converter can see or use the info, is it really worth keeping anyway? — RPG Master, Dec 30 '10 at 23:45
@Rowland Shaw Sorry I didn't reply sooner. My main reason for converter is for DNG's support of XMP metadata. I'd like for all my tags and comments to stick to the files themselves instead being stuck to digiKam's database. Also, if I choose not to embed the original file, their is a decent shave off the file size. — RPG Master, Dec 31 '10 at 03:30
Besides some EXIF, do you actually lose any image detail in DNG file (vs native RAW file by canon)? — rvpals, Oct 10 '11 at 19:10

score 14 · Accepted Answer · answered Dec 30 '10 at 15:08

14

Your answer can be found at this forum site, but the short is, you will lose some EXIF information, the lens id in particular, but the normal EXIF will be there (IE, aperture, focal length, exposure time, flash firing).

answered Dec 30 '10 at 15:08

PearsonArtPhoto

10,941
10
60
93

1

I just looked at both a ARW file and a DNG file's EXIF, and they both have the same "Exif.Sony.0x____" fields with the same data. So does this mean everything is being carried over or just that my Exif view doesn't see the proprietary Exif stuff? – RPG Master Dec 30 '10 at 23:44
1

This answer would be much better if you didn't "just" link to a forum that might disappear. – Håkon K. Olafsen Aug 05 '14 at 07:53
1

It's not "Just" a link, I include the key points from the link. It just so happens there isn't a lot there, but one could read the form if one is interested in more detail than I included. – PearsonArtPhoto Aug 05 '14 at 12:00
I down-voted this answer because the discussion in the linked thread contradicts the very brief summary in this answer. It's incorrect and doesn't match the linked thread or the edit in the OP's question. Lens ID does carry over to the DNG file. As another commenter said, the answer would be much better if it actually answered the question rather than pointing at a discussion where you must infer the answer from multiple contradicting posts in that forum. – juggler Jan 08 '18 at 00:38

score 11 · Answer 2 · answered Dec 30 '10 at 16:39

DNG can be a rather complex beast. The file format is similar to TIFF, in that it is not specifically an image format itself, but more of a container. A "normal" DNG image will store metadata, the primary image in TIFF format, and possibly a thumbnail image. Depending on how DNG is used by any given program, the reality may differ. It is possible to store the original RAW image data in its native format inside of a DNG, and include the XMP sidecar as another file in the DNG container. Some programs store the original RAW, a TIFF version, and a JPEG thumbnail, along with some metadata.

The story is not particularly simple when it comes to DNG. Generally speaking, for compatibility purposes, DNG images store primary image data in TIFF format. As such, they are not truly RAW images, as original image data must be processed to create a common, interchangeable format in TIFF. Some cameras these days output their sensor data directly into DNG format, and such manufacturers prefer to call that "true" raw format, but the simple fact of the matter is that their raw sensor data must still be transformed into an RGB format that can be used by various image editing programs.

If you want the benefit of RAW, you need to use your camera's native RAW format. The key difference between RAW images and their RGB counterparts is in the pixel data...a RAW image contains Bayer sensor array pixel data, rather than computer screen RGB triplet data. Bayer sensor pixels and computer screen pixels are not the same thing, and should be treated as distinct types of information to produce the maximum amount of quality when processing digital photos.

Actually, it's TIFF/EP which is specific to raw and includes color filter (bayer filter) information for interpreting the information. It differs from TIFF in that it's an ISO standard not an Adobe administered one. So, I'm not sure that last assertion in your second paragraph is entirely correct, the DNG should contain the raw sensor data. Anyways, as a Pentax shooter, I get to choose either DNG or PEF, so I've chosen DNG and to date no Pentax shooter I'm aware of has been able to show a meaningful difference in the outcome between the two other than DNG is more widely supported. — Joanne C, Dec 30 '10 at 17:07
From what I have read, DNG supports TIFF/EP, however in somewhat of a limited capacity. Every time I have read about DNG and TIFF/EP, there was always a catch clause that made it sound like DNG did not fully support everything that TIFF/EP does. Even if we do assume that DNG does fully support all of TIFF/EP, there is still the simple fact that bayer array information has to be translated into a TIFF/EP structure and data types, rather than its native structure. The only way to really, truly get "RAW" data is to use "RAW" data...direct, unprocessed, strait off the sensor. — jrista, Dec 30 '10 at 17:22
The specification is here: http://www.adobe.com/products/dng/pdfs/dng_spec.pdf and one of the more interesting aspects is that DNG has a section for converted raw formats to be stored compressed in the new DNG file. In any case, DNG is based on TIFF/EP, it's not actually TIFF/EP, so my point is, I think DNG is getting the raw data off the sensor and it is not being converted for TIFF storage. — Joanne C, Dec 30 '10 at 18:33

score 5 · Answer 3 · answered Dec 30 '10 at 16:35

5

RAW is RAW. If you convert it to anything else, it is no longer RAW. Sure, you have more bit-depth than converting to an 8-bit format and you don't have an image yet (not all color channels are present at all pixels) but if you really want to keep your originals, you should keep your originals. Sorry if this goes against common lore but it can't be any other way.

DNG has been welcomed by many like a second coming when really it is just another proprietary file-format. It belongs to Adobe rather than a camera manufacturer. The only real benefit is that it is publicly documented.

The downsides are many because it is removed from the camera. Sure, it could (but in fact it does not) contain all the data from the camera sensor but it does know how the camera works. It has information on color primaries and special header fields to tell whether the sensor as a Fuji-style pixel alignment (pixels not arranged in a grid) but imagine all the present and future possibilities. The camera itself knows how pixels are arranged, how they respond to light (some may be more or less sensitive based on their position, some may be hot/dead, etc.).

Honestly, I'd rather see cameras produce a losslessly compressed high bit-depth image (say 16-bit PNG) in the camera than anything else. The output could then be unambiguously interpreted everywhere and would have been processed with all the camera knows about itself.

answered Dec 30 '10 at 16:35

Itai

102,570
12
191
423

2

DNG is a raw format, the TIFF/EP standard on which it is based is specific to raw image capture. – Joanne C Dec 30 '10 at 17:09
1

@John - That's what the propaganda says ;) Well, yes, it is specific. That is why it does not include all color channels at each pixels but things are much more complicated. The process of creating an image involves data and process. DNG is a way to store that data and defines a process (it has to, otherwise it could not be viewed) but being standard across most cameras (DNG makes an exception for Fuji SuperCCDs) the data has to either be massaged to be compatible with the process or suffer from excluding what a camera requires which differs from the process. – Itai Dec 30 '10 at 18:45
3

I realize it is hard to explain, so let's try an example: Imagine a camera sensor is less sensitive towards the edge of the sensor, what goes in the DNG? Option 1) The RAW data exactly read from the sensor which results in an image that looks dark along the edges in DNG viewers. Option 2) The RAW data is corrected for the fall-off and stored in the DNG file. In this case, the file looks OK but you don't have the original data in the pure sense. Most cameras do the latter because there are things that depend on that particular camera (not the model, THAT exact camera) like hot/dead pixels. – Itai Dec 30 '10 at 18:53
@Itai - That's not entirely true, I linked the DNG standard above. I think there's a lot of misinformation about DNG out there, but Adobe has made the specification and SDK source freely available without encumbrance, so if it is anything less than raw, then this can be established. By the way, most proprietary raw formats are based on TIFF. – Joanne C Dec 30 '10 at 19:00
3

@John - Indeed, it has been established, DNG is less than RAW. Please read through my answer and comments carefully. The present version of DNG can store RAW data from any sensor, I agree. However, this is not sufficient because it separating data from the process of understanding that data. Vendors either have to produce data which will be correctly interpreted by DNG readers by transforming the RAW data OR give the RAW data and face inconsistent output, including from camera to camera because of sensor-corrections built into cameras as part of the QA process. – Itai Dec 30 '10 at 21:21
@Itai - I did, I just don't agree with the assertion. Here's a very comprehensive set of information on DNG: http://www.barrypearson.co.uk/articles/dng/ In a nutshell, DNG enables camera specific details for conversion though not all DNG readers may be able to make of use of them (ACR generally will) and so will fall back to certain forms of defaults. This, for example, allowed me to use ACR with the K-5 before Adobe released the update that directly supported my camera. – Joanne C Dec 30 '10 at 22:05
2

@John - There lies the problem, DNG can contain anything based on its ability to evolve, but someone has to keep updating the process (aka software) to ensure proper interpretation. Camera manufacturers do that with their own RAW formats but can do it at their own pace without submitting to a committee about how their next camera will work. – Itai Dec 30 '10 at 22:29
@Itai - DNGPrivateData allows vendors to do this without changes to the DNG specification. – Joanne C Dec 31 '10 at 00:15
2

@John - It really does not matter because THE process does not change by whatever is put in the private data. If the camera puts there that it has triangular pixels or even worst additional date per pixel (say overflow info, base-level, etc), no one will know about it and all DNG viewers will show the wrong thing! That's why they had to put additional fields for Fuji and they will have to keep doing it. Private data only serves the vendor's own software because no one know what to do with it. – Itai Dec 31 '10 at 02:08
1

@Itai - There are always exceptions, but to argue that DNG loses something in the vast majority of the cases is disingenuous, it doesn't. It's not perfect, mind you, but the crazy scattering of raw formats is hardly perfect either. Some of us would prefer not to have to do software upgrades every time we buy a camera. – Joanne C Dec 31 '10 at 02:11
2

@John - We've actually diverged from the original question. What happens is that because manufacturers cannot control the process and still want DNG viewers to show the right thing, they actually convert from the internal RAW to the DNG version of RAW which is designed to satisfy how DNG viewers work. Converting to DNG can therefore be a lossy operation and just because some converters decide not to include certain metadata but because even the pixel data can be altered to take into account particulars of the camera. – Itai Dec 31 '10 at 02:12
Anyways, my initial point is that DNG is raw, so conversion to it isn't converting away from raw and, more to the point, the DNG can contain the original raw file as well, so you lose nothing at all in the process, but you have an archival format that has more chance of surviving over time. – Joanne C Dec 31 '10 at 02:14
@Itai - It isn't lossy in ACR. Anyways, by standarizing certain pieces of information you potentially turn the undisplayable into something that can be displayed and, then as needed, adjusted. But yeah, we got away from the original question, though a good debate nonetheless. – Joanne C Dec 31 '10 at 02:17
1

DNG is good enough that it's the one and only "RAW" format for Leica's $30K S2 medium format (and definitely professional grade) SLR (as well as the digital Ms, which I'll concede can be seen as shiny toys for big boys). The problems with DNG are in individual implementations, not in the format itself. – Feb 08 '11 at 16:19
2

@stan - This only means that Leica choose to use the format in a way that fits with the interpretation process implied by DNG. It is highly possible, and it has happened in the past with Fuji (DNG has extra fields to trigger a different process for Fuji SuperCCDs), that a camera produces an output which cannot be correctly interpreted by the implied DNG process. The way manufacturers often go around this is to convert their internal RAW representation to a representation which fits the way DNGs are meant to be interpreted. – Itai Feb 08 '11 at 19:07
There is zero data missing from a Leica DNG as compared to anyone else's RAW (and they use the full 16 bits of color depth per channel, not 12 or 14). DNG is not a limited format in any way (well, except for that whole 16-bit color depth thing -- it'll need a revision if anyone decides to go 24 bits per channel) -- its broken and short-circuited implementations that show problems. – Feb 08 '11 at 19:15
1

@stan - You oversimplify. A long blog post may be needed ;) Example: A perfect sensor has every photosite arranged in a perfect grid and all have exactly the same sensitivity. This is perfect for the way DNGs are specified. After testing the factory realizes that there are a few photosites that are less sensitive to light. Instead of replacing the sensor, they adjust the firmware in-camera to compensate by simply boosting the read-out levels of those when outputting DNGs. If they did not do that, the DNGs would look strange. So you do not have the exact RAW data anymore. – Itai Feb 08 '11 at 19:29
There's room for that in DNG too, along with noise subtraction image data, CB adjustments not made to the raw sensor data and everything else that goes into anyone's RAW image format. Don't let broken or short-circuited implementations cloud your judgement -- the full DNG spec is a complete package. – Feb 08 '11 at 19:32
I am familiar with the spec, there is room for a lot of things to be added but reading files with private data correctly is all but impossible except by dedicated software that knows how to interpret it. That is why it is more often done to fudge the pixel data rather than use those. – Itai Feb 08 '11 at 19:45
3

Then there is the issue of sensor construction. Some sensors are designed quite differently and will produce data that cannot be interpreted properly by the same process as other DNGs. That is why the fields are there for SuperCCDs which use octagonal sensors which are not lined up in a grid. Then you have the SR variety which used two type of photosites or gets two samples per photosites (depending on the generation). EXR sensors can also read half their pixels partway through an exposure. If you put that data in a DNG without compensating only proprietary software will read it! – Itai Feb 08 '11 at 19:48
1

@Itai, is your argument basically that camera manufacturers are in the best position to interpret RAW sensor data, and that DNG introduces another conversion that may or may not get in the way of resolving to an accurate image? Can you point to an example image where this is evident, or yours more of a hypothetical position? (I'm interested because i have avoided DNG for the same basic reason.) Conversely, in what cases can DNG be better than RAW? – b w Oct 10 '11 at 20:19
6

Judging from the length of this thread, I'm having trouble explaining it. The main point is that even DNG software needs to be updated to support each camera because they have to be able to interpret the data placed in DNG relative to its origin. There is a lot of generic stuff but as soon as data is needed above that, you need new DNG software. The whole thing masks RAW decoding issues by saying DNG format is universal so there is no need for proprietary software. I do not see a case where DNG can be better. – Itai Oct 11 '11 at 17:29

score 3 · Answer 4 · answered Dec 11 '13 at 03:25

Converting an image file from the manufacturer's raw format to .dng will strip all of the information in the maker notes section of the EXIF data. Since all Adobe products ignore the maker notes to begin with, if you only use Adobe products you will not see a difference in this respect.

There are additional things that the conversion strips as well. For example, data from masked pixels used to determine black point are not carried over into the .dng file. Instead black point is computed and 'baked in' during the conversion process. As with all raw convertors that do not use the manufacturers own proprietary and often encrypted algorithms, there is no guarantee that the conversion by the third party software will be the same as conversions that use the manufacturer's algorithms.

Since each sensor design is different, the output from the sensor must be interpreted based in the design of that sensor. As new cameras are released with new sensor designs, updates to the DNG convertor must be made to properly convert the output from the new sensor. Not all Bayer masks, for example, use the same exact colors for each of the R,G, and B filters. Some, such as newer designs from Fuji, even alter the pattern of which pixels are filtered by R, which by G, and which by B. Without the specific information of the sensor's unique design, the convertor will misinterpret the data from the sensor.

Do I lose anything converting to DNG?

4 Answers4

Linked

Related