2

Recently I came across a debate about whether it's fair to call RAW 'unprocessed' or 'unaltered' sensor data of a DSLR. As far as I know, the analog sensor data is processed into a digital RAW file using the ISO setting, a process during which the Bayer-sensor data is demosaiced as well. However, those are just the necessary steps for saving a digital file while maintaining maximum quality.

Are those assumptions correct? If so, are there any additional steps that need to be performed between the sensor receiving light and the saving of the RAW file? And can those be reasonably be described as an 'alteration' of the sensor data, or is it fair to say that a RAW file holds the unaltered sensor data?

MoritzLost
  • 1,002
  • 1
  • 12
  • 30
  • 1
    The discussion of RAW data versus images on this site seems a bit like an insistence that exposed film contains a suspension of silver-halide crystals but not images. To me, there's a latent image [or philosophically a continuum of possible latent images] in both cases and Manual of Photography 10th Edition at times uses "RAW image". Anyway, I'm clearly more a fan of looking at how terms are used than trying to standardize terms that have no technical standard. And from that perspective (the practical one), RAW is whatever a camera will produce with a minimum of arbitrary interpretation. –  Jun 03 '16 at 19:31
  • @benrudgers That is also my stance on the topic. Still, I am trying to understand better what happens between the sensor and the RAW file, and in what ways this process could theoretically be changed (the latter isn't part of my original question though) ... that is, if the steps I described in my question are correct and whether there are any more of importance for the generating of the RAW file – MoritzLost Jun 03 '16 at 20:03
  • 2
    Demosaicing is not performed during that operation, at least not for any camera that I've owned. – junkyardsparkle Jun 03 '16 at 22:57
  • The addition of metadata to the quantized sensor measurements is what allows algorithmic possessing the latent image in the data when passed into the pixel pipeline. It's the metadata that indicate whether the latent image requires demosiacing. –  Jun 04 '16 at 02:53
  • @benrudgers and every image from a camera with a Bayer mask (the overwhelming majority of cameras used for photography as defined here) will have metadata that indicate it requires demosaicing. – Michael C Jun 04 '16 at 04:23
  • @MichaelClark I agree that Bayer filter arrays are the most common. For demosiacing, "It is a Bayer filter array" is not enough metadata, of course. The color of the first filter in the array element must be known; whether the data are recorded row by row or column by column or some other way; how the data is compressed; the bit depth of the quantization; etc. Fortunately, this is available by reference and placing the camera model and similar high level descriptions in the metadata allows lookup and algorithmic processing of the latent RAW image. –  Jun 04 '16 at 04:55
  • No one said none of that information was not needed for demosaicing. No one said that cameras with Bayer masks don't provide that info. What we said was any image created by a camera with a Bayer mask will need to be demosaiced. – Michael C Jun 04 '16 at 06:11
  • I agree we said that. I apologize for my expansion upon the necessary metadata related to demosaicing implying that someone had said something that nobody said no how. Anyways, I in no way meant anything like that something. –  Jun 04 '16 at 22:27

2 Answers2

5

It varies highly from camera to camera. Some designs do a minimum of processing on the image sensor itself, others do a little more. Those that do more do so mainly in the area of noise reduction either before or after sending the analog data to be converted to digital data. One method is the relative amplification of the signal from pixels masked for red, green, or blue (which is done for reasons related to the different noise characteristics of pixels filtered for the different colors of the Bayer Mask). Another method used after analog-to-digital conversion is to average pixels with a much higher luminance value than their neighbors to a value much closer to the surrounding pixels.

Even different camera models that share the same sensor design may apply different processing to the output from the sensor, either before or after it is converted to digital information, prior to it being saved as a raw data file. Information about the conditions under which the data was obtained (camera model, sensor characteristics, ISO, WB, etc.) will be appended to the file so the application eventually converting the data to a viewable image will (hopefully) know how to convert it. Demosaicing is not normally done to sensor data before it is saved as a raw file. That is done when the raw data is converted to something else, such as an image displayed on a monitor by a raw conversion application, or converted as output as a jpeg or tiff.

Michael C
  • 175,039
  • 10
  • 209
  • 561
  • There are some thoughts on the internet that some cameras apply some lens correction directly to the data instead of metadata. Some cameras also apply WB preconditioning where they digitally scale some of the color channels – MirekE Jun 05 '16 at 17:59
  • @MirekE That's all done when converting the raw data to jpeg in camera, either for the jpeg image preview embedded in the raw file, or for saving as a jpeg image. Raw data essentially has no color until it is demosaiced. – Michael C Jun 06 '16 at 00:24
  • It can be done on non-demosaiced raw files, this is how apparently CornerFix works. https://sites.google.com/site/cornerfix/ I spent some time researching this when I had a camera that was apparently affected by this (Leica M9), besides Leica there were also discussions that this is possibly done on a FF fixed lens Sony camera, but this was several years ago and I don't have any links anymore to share besides those that I am mentioning under Birfl's reply. – MirekE Jun 06 '16 at 00:46
  • So can Digital Lens Optimization for Canon raw files which results in a raw file double the size of the original. But none of that is being done in camera by the camera's processing pipeline, which is what this question is about. – Michael C Jun 06 '16 at 00:49
  • I have no info about the Canon thing - it may be applied as lens calibration metadata. What I was talking about was direct manipulation with the raw data. – MirekE Jun 06 '16 at 00:53
  • Since converting .cr2 files to .dng results in white point and black point being "baked in" and the information from masked pixels being thrown away along with the maker notes in the metadata, I don't consider such a file a 'raw" file. It's something like a high-bit tiff - somewhere between raw and jpeg/png/etc. – Michael C Jun 06 '16 at 00:53
  • @MirekE But none of the examples you cite are being done in camera by the camera, which is what is stated in the question. They're being done by post-camera applications that convert data from an already existing "raw" file produced by the camera. Any result of that manipulation, even if saved in the same .cr2, .nef, etc. file format is no longer a true raw file. – Michael C Jun 06 '16 at 00:56
  • Again, I am talking about direct conversion of the RGBG data. I don't have time to reverse engineer Leica M9 raw files, so if you say that Leica M8/9 positively do not modify raw data in camera with lens corrections (namely vignetting), then I will trust you. – MirekE Jun 06 '16 at 01:08
  • @MirekE Other than what you call color preconditioning, which some cameras do when applying amplification to the electrical signal from pixels filtered by different colors by slightly different amounts and was pretty much already covered in the answer. But I've edited to make it more obvious. – Michael C Jun 06 '16 at 01:13
  • Where does my answer say any such thing as "...Leica M8/9 positively do not modify raw data in camera with lens corrections (namely vignetting)..."? You need to read the opening two sentences more carefully! Words such as "varies highly" and "Some do... minimum..., others do ... more..." don't mean what you seem to think they mean. – Michael C Jun 06 '16 at 01:28
  • "Mainly" does not mean the same thing as "only" any more than "many" or "most" mean the same thing as "all"! – Michael C Jun 06 '16 at 01:30
3

Back in the 1980s, Craig Anderton wrote a book called Home Recording for Musicians in which he observed that a recording studio isn't a capturer of reality, it's a processor of reality. You should think of your camera the same way. The light making up an image enters the lens, which changes it, reaches the sensor, which changes it, is passed through various bits of analog circuitry, which change it, and finally becomes digital data by way of an analog-to-digital converter (ADC) which, as you might have guessed, changes it.

Any change is an alteration, and it might be reasonable to call alteration done with a purpose processing. There are undoubtedly some things done on purpose in the analog stages of your camera. ISO adjustment is the elephant in that room because analog amplification is necessary to get the most out of the ADC.

You're not getting information straight from the sensor because it's an analog signal, and analog signals are notoriously difficult to store and reproduce well. Whether you call what happens between there and when you have digital information alteration or processing is debating the trivial. If you're going to split those hairs, the answer is that it's probably some of both.

The ideal place for capturing data for raw files is the output of the ADC, because anything that happens to it beyond that is algorithmic and can be done off-camera. If you find a better way to do something (demosaicing, application of curves, etc.), it's much more economical to update a raw converter that runs on a computer than it is to create, test and distribute new camera firmware. Providing raw data is also something of an acknowledgment that the decisions made by the manufacturer aren't going to please everyone. Doing as little as possible to the data post-ADC gives the photographer the same flexibility he'd have had in a darkroom.

Whether or not the contents of a raw file is straight from the ADC or somehow adulterated is known only to the manufacturers. One implies that it doesn't, and my own inclination is to believe that to be true for reasons that go beyond the scope of answering this question.

As Michael Clark points out in his answer, everything in a raw file that isn't image data is bonus information that can be used in making decisions about how to process it.

Blrfl
  • 5,532
  • 21
  • 27
  • Unless you're designing your sensor and processing pipeline with the primary intent of scoring as highly as possible on DxO Mark. Then you apply some noise reduction before storing the "raw" data. By reducing the noise floor before the data is tested, it appears to improve the SNR and thus DR. So what if point sources of light such as dim stars disappear along with the noise? It's a better sensor, right? DxO says so! – Michael C Jun 04 '16 at 22:14
  • 1
    @MichaelClark: Good thing Volkswagen doesn't build cameras. ;-) – Blrfl Jun 04 '16 at 22:45
  • If one company uses WB precondition, second company noise reduction and lossy compression and third applies lens corrections directly to the data, I could not agree with your statement that "odds are very good that what you get in a raw file is straight from the ADC". It is not in all camera models, but many people got caught by a surprise... – MirekE Jun 05 '16 at 18:10
  • @MirekE: Without proof that's actually happening, neither of us can say, so I've updated my answer accordingly. – Blrfl Jun 05 '16 at 18:52
  • I thought comments are not the right place to put an evidence, but here we go: http://www.dxomark.com/Reviews/Half-cooked-RAW/Noise-reduction, http://www.rawdigger.com/howtouse/sony-craw-arw2-posterization-detection, https://helpx.adobe.com/x-productkb/multi/lens-profile-support.html, http://www.rawdigger.com/howtouse/rawdigger-histograms-overexposure-shapes – MirekE Jun 05 '16 at 19:02
  • @MirekE: (1) The kind of noise reduction that would produce what DXO observed is entirely within the realm of possibility on contemporary sensors, and they make no claim about which side of the ADC they believe it happened. (2) Whether compression artifacts count as purposeful alteration is debatable. (3) All the Adobe page points out is that many cameras apply lens corrections to non-raw formats and that some others embed the lens profile as metadata which is then applied by Adobe, not the camera. (4) Again, very likely happening in the analog domain. – Blrfl Jun 05 '16 at 21:12
  • Comments are really not a good place for detailed debate. I will leave your comments unanswered and leave it up to the reader to decide. – MirekE Jun 05 '16 at 21:47