11

Is there some way of saving (basic) non-destructive RAW developing/adjustments that is not limited to a particular app ecosystem (eg Adobe)? Understandably some transformations are proprietary (eg. some cool denoise algorithm, or a spot removal tool) but others are rather generic that I think could easily be standardized (eg. crop, exposure, white balance) and be read by other applications.

At first I thought sidecar xmp files could be useful in that respect, since Camera RAW/Lightroom and darktable save their adjustments inside such files, but seems that adjustments created by Camera RAW/Lightroom can not be previewed/edited by darktable and vice versa.

Any solution that has worked for you?

This question is mostly about creating a futureproof way of storing the original camera data, but also the edit history (adjustments in a non-destructive way)

mattdm
  • 143,140
  • 52
  • 417
  • 741
FotisK
  • 343
  • 1
  • 2
  • 13

3 Answers3

3

They're all proprietary. Each application uses their own algorithms to convert the raw data. To the best of my knowledge none of the adjustments made with one application will translate identically in another application unless both applications use the same raw conversion engine "under the hood" (e.g. Adobe Lightroom and Adobe Photoshop both use Adobe Camera Raw to do the actual raw conversion). So you have to choose only one application to do raw conversion.

If you then want to use other tools to further edit the image you must export the results of the raw conversion in a standardized raster image format that can be edited by the other tools. Probably the most common way to deal with this is to do the raw conversion and then export the images as 16-bit tiff files.

Of course converting to tiff locks in a lot of the decisions made in raw conversion: black point, white point, gamma correction, etc. and reduces the amount of further adjustment to things such as color temperature/white balance, etc. But the 16-bit tiff has a lot more information than, say, an 8-bit jpeg. The maximum number of gradations in an 8-bit color value are 256. The maximum number of gradations in a 16-bit color value are 65,536. In addition to the lower bit depth there is also the consideration of compression. So the tiff file will allow a lot more adjustment before things start to fall apart than a jpeg will. The biggest disadvantage of 16-bit tiff files is their size: A raw file from a 20MP camera will run around 24-30MB. The exact size depends on content. Most raw files are compressed using lossless compression so the more variation there is in the scene the larger the file size will be. The same 20MP image converted to a 16-bit tiff file will run around 100MB uncompressed.

From the comments:

After posting the question last night, I discovered that darktable supposedly (and to some extent) can read the adjustments inside adobe's xmp files - for what it's worth, here and here are the links. I haven't managed to make it work thus far though.

Even when the instructions can be read properly the question still remains, is darktable applying the exact same algorithm to get the exact same result? Or is it just using the darktable algorithms to approximate the results from a certain setting using ACR? My hunch is that you could get the exact same results by editing in darktable to begin with as you could get by translating the instructions from ACR using darktable. You just have to learn how to use the different GUI to get there.

Indeed @MichaelClark, one could use LR, Br or darktable without needing to use another! I was more interested in finding a workflow, that is futureproof - in the sense that I won't be locked down in one particular application/ecosystem/OS.

In that case always save your original raw files. That's about as future proof as it gets. People like to make a lot of noise about a "standard" raw format, but it's just a bunch of smoke and mirrors. DNG is no more future proof than .cr2 or .nef. And more applications let you convert .cr2 and .nef files than let you work with .dng files.

Nothing is future proof. Absolutely nothing.

Michael C
  • 175,039
  • 10
  • 209
  • 561
  • I'd be interested in a justification in the claim that those various properties are "locked in" when converting from raw to tiff. Raw isn't magic, it's just another raster format. The only thing that gets locked in is the demosaicing, imho. All the information after that step should equally well be preserved in tiff. – ths Jan 13 '17 at 07:41
  • 1
    Do you understand what black point is? All values in the sensor readout data beneath the selected black point are normalized at zero. There is no longer any distinction between those values that were once different. Thus, that information is irretrievably gone. Same thing with white point. All values above the white point are normalized at the maximum. Any distinction between them in the sensor readout is thrown away. – Michael C Jan 13 '17 at 07:47
  • 1
    Fully in agreement, but just some additional opinion - as I'm still pretty new to this & can't provide a full second answer, but I've found my 'best' method is to do my initial colour editing in Nikon's own software, then export tiff & do the rest in photoshop. Opening directly in Adobe is, imo, nowhere near as good a result. I do find, however, that the tiff file sizes are more like 180MB than 100, for a 24mp image – Tetsujin Jan 13 '17 at 08:17
  • yes, if you throw data away before saving as tiff, you loose something. like if you convert to b/w or sepia, you don't have colour information anymore. that's self-explanatory. but if you save it without cropping the ends (with a well behaved histogram, so to speak), everything should remain. – ths Jan 13 '17 at 09:07
  • 2
    Thank you @Michael Clark, and everybody for the comments! TIFF is what I would call a "baked" format. Beyond baked-in parameters like exposure/clipping/demosaicing and noise-reduction that depend a lot on the raw-developer-software interpretation, there are also more drastic transformations, like Masks, rotation and cropping that are being destructively applied. While I would definitely use tiff for archival purposes of a final photograph, I don't feel using it for anything else. – FotisK Jan 13 '17 at 13:37
  • @Michael Clark, after posting the question last night, I discovered that darktable supposedly (and to some extent) can read the adjustments inside adobe's xmp files - for what it's worth, here and here are the links. I haven't managed to make it work thus far though. – FotisK Jan 13 '17 at 13:43
  • 1
    @FotisK Even when the instructions can be read properly the question still remains, is darktable applying the exact same algorithm to get the exact same result? Or is it just using the darktable algorithms to approximate the results from a certain setting using ACR? – Michael C Jan 13 '17 at 17:02
  • that is a valid point @MichaelClark! I doubt they can be identical. I trust some basic transformations (like crop/rotate) ought to be visually identical and others like exposure/saturation/white balance, close enough that they evoke the same impression (when not viewed side by side). Sometimes this approximation can be enough though. In my case, I keep "crudely" developed copies in my library for previewing purposes; enough to inspire me someday to re-visit the photo and create something out of it – FotisK Jan 13 '17 at 22:46
  • 1
    @FotisK My hunch is that you could get the exact same results by editing in darktable to begin with as you could get by translating the instructions from ACR using darktable. You just have to learn how to use the different GUI to get there. – Michael C Jan 14 '17 at 00:36
  • @ths If you're going to do that why not just use the second application to do the raw conversion? Even if you don't "...crop the ends..." of the histogram you're still baking in a lot. – Michael C Jan 14 '17 at 00:40
  • Indeed @MichaelClark, one could use LR, Br or darktable without needing to use another! I was more interested in finding a workflow, that is futureproof - in the sense that I won't be locked down in one particular application/ecosystem/OS – FotisK Jan 14 '17 at 02:20
  • 2
    @FotisK In that case always save your original raw files. That's about as future proof as it gets. People like to make a lot of noise about a "standard" raw format, but it's a bunch of smoke and mirrors. DNG is no more future proof than .cr2 or .nef. And more applications let you convert .cr2 and .nef files than let you work with .dng files. – Michael C Jan 14 '17 at 02:24
  • Let us continue this discussion in chat. – FotisK Jan 14 '17 at 02:48
2

After a few days of research, I've concluded that at the time of writing there is no holy grail. But as for futureproofing I'd go the following route:

  • keep the original RAW file
  • work with Adobe products for the adjustments (either inside Lightroom or with sidecar xmp files produced by adobe camera RAW)

This probably is as futureproof and cross-platform as it gets. Which is, not a lot, but not totally bleak either.

As a second layer of future-proofing, I would retain

  • a 16-bit TIFF of the original RAW (in case a day comes that nobody wants to create a RAW decoder for my particular camera)
  • a high quality jpg export of every variant/version of a particular image (in case I need reference to reproduce this exact look)
  • and a 16bit TIFF of all the final products (eg. Edits that made it in print, exhibitions, or in general were used someplace beyond my light table)

DNG Files? No thanks

While I have not looked deeply into DNG files (so please correct me if I'm mistaken somewhere) I understand that

  1. they cannot replace RAW files, because

    a. there is a level of interpretation of the sensor data, which means they are already slightly away from the original RAW

    b. It's not one universal file format; it's a container. Which means that in the distant future someone will have to write software that reads the particular flavor of DNG that your camera files got converted into. Not much different fate to that of RAW files, is it?

  2. DNG files are probably not any better at future-proofing the adjustments:

    a. the xmp adjustments embedded in the DNG file, are not any more universal than the adjustments inside the sidecar xmp. They are still software-specific. On the contrary, I've seen a few 3rd party programs (see below) that read ACR/Lightroom adjustments from .xmp files, but I'm not aware of any that can read them out of DNG files.

Reasoning behind choosing Lightroom and Camera Raw

Futureproofing

Adobe products have become pretty much the industry standard and as such:

  1. Competitors will have to lure existing Adobe customers into their own ecosystems. A few are already attempting reproducing (subset of) the adjustments that ACR stores inside the .xmp files. For instance the open source project Darktable is claiming accurate reproduction of crop,rotation,flip and tags, and mostly accurate reproduction of exposure/blacks, grain, tone curve (only lightness supported), color zones local contrast - as seen here and here. Others imply that you can migrate whole lightroom libraries including the adjustments (Capture One Pro claims that it "allows import of other 3rd party application catalogs to make migration easier than ever, maintaining folder structures and most adjustments on your images" accompanied by a picture showing Aperture, Lightroom and Media Pro as seen here. On1 Photo RAW supposedly will be adding support for that in March 2017 - for more read here, here and the timeline here). While these attempts can never be 100% accurate, they are accurate enough to give a starting point for further image manipulation and/or at the very least underline an interest by 3rd parties to figuratively-speaking reverse engineer Lightroom's and ACRs ways of storing adjustments.
  2. For better or for worse, Adobe is rather successful so this means that there is no reason in the foreseeable future to go out of business
  3. Even then, due to the large Adobe user base, it's quite possible that future engineers will probably be interested in recovering Lightroom and ACR generated data.

Cross-platform

While adobe products do not run on Linux, they run on macOS, Windows, Android and iOS. As of July 2017 Lightroom mobile, supports raw editing (read here). For linux, one could use darkroom which however is a one way trip since Darkroom does not sync it's own adjustments back in adobe's .xmp format.

FotisK
  • 343
  • 1
  • 2
  • 13
  • "It's not one universal file format; it's a container. Which means that in the distant future someone will have to write software that reads the particular flavor of DNG that your camera files got converted into. Not much different fate to that of RAW files, is it?" Do you have a source for this? Pretty much kills the main point for why I was considering DNGs in the first place. – Mateen Ulhaq Dec 31 '19 at 09:19
1

The DNG file format will allow saving adjustment information directly in the file, rather than a sidecar file, and the DNG format is a documented specification that is free for anyone to use. Some cameras support DNG natively as a RAW format, for others, you can use the free Adobe DNG converter. Within the Adobe ecosystem, images edited with the version of ACR provided in even an old version Photoshop Elements (7) will tranfer to the latest Lightroom (CC) with all the adjustments intact.

Unfortunately, I've tested DNG files edited with Lightroom, in Corel Aftershot, and Raw Therapee and neither appear to be able to recognise the adjustments, which is not surprising given that the adjustment parameters the different programs use are quite different, even if they do similar things.

I haven't tested On1 Photo RAW or Cature One Pro, but I'd imagine the results will be similar.

vkiwi
  • 241
  • 2
  • 2