magyar változat

RAW Conversion

 
Music Photo Instruments Modelling Transport Writings Publications Sound x


L I N K S

RAW Conversion (2008)

Update: may, 2012


Friends, inexpert in digital imaging, often ask: why to use raw format? Some of them try the converter, they got with their camera, and become confused, when don't find any difference compared to the camera-made JPG on their (uncalibrated) display. So I decided to run a test of some popular RAW converters, (including the one I have in my camera, and that I recived on the CD), to see clear, what's the real difference. I'll focus just to the image quality, and don't take any interest for their additional features.

First, some theoretical overview for those, who are beginners in digital imaging.

Digital image in most cases contains three color channels (R -red, G -green, B -blue), and 8 bit of information in each. That means, it can display 256 shades of each prime colors. It is considered to be fine enough to be seen smooth, and ungraduated by the human eye. Mixing that 256 shades of prime colors, we can get 16 millions of colors as total. Of course, the range of visible colors is much wider, while that of printable ones is a bit narrower.

Imagine a scene with heavy contrast! Now we're talking only about tones, but of course, it can be escalated by saturated colors.

Example: Sun-lit white marble wall, with a deep shaded gateway, where a black-skinned man stands in black clothes, and black sunglasses. Difficult task, almost even impossible to make the details of the marble visible, and the man recognizable, using either analogue or digital technique. (Don't speak about HDR now!) If we'd used the available 256 tones for the man in the shadow, we'll not have any for the marble. It's darkest point will give the value of 256, like all others will, because there is no value higher then 256 in an 8-bit system. In the same way, if we gave the 256 values to the differently lit points of the marble, the whole shaded zone will get the 0 value, so the details - including the man - will be lost. Finally, if we'd share the 256 values to all of the range, from the darkest point of the shaded gateway to the brightest detail of the marble, getting an unnaturally low contrast image, we won't have enough tones in each (shaded and sunlit) range, if we'd later decide to print a picture of a marble, or the man with natural contrast. They'll have graduated (posterised) tones. The electric sign, produced by the sensors, is however digitised by more then 8 bits. Cameras usually use 12 bits (newer ones 14 bits). More then this is not physically recognised by the sensor's hardware itself. 12 bits means 4096 levels per color channels, giving an acceptable level of freedom to make the final 256 of it.

That's not so nice however, becouse these 4096 are levels of a linear scale, while those 256 of a logarithmic one, matching the behaviour of the human perception. Maths-geeks already see the problem, others won't be in a fever reading more maths formulas and equations. Be satisfied with this: the tonal separation of the dark tones is much poor, then that of the light tones.

We can however make the best of this freedom only if we really have the ability of decision. This is what RAW for. JPG is an 8-bit format (of course 8 bits per color channel) furthermore with lossy compression. The advantage of it is that they're small files, so more can be saved on a card. Furthermore we directly get a format, supported by all browsers, mailing and viewer applications. The essence of lossy compression is (just like in the case of MP3), that it looses information, we could not perceive. But this is only true, until we start to modify our image. Maybe we want to find out exactly what has been deemed to be useless by the software. As the file is an 8-bit format, we have no more freedom, it had been decided, what is important: the man in black, or the marble. It had been decided by the machine instead of ourselves! Making photos in JPG is just like we would’ve not bring the negative from the lab, only the 10x12 (OK, 13x18) cm minilab prints. To illustrate the differences in the abilities of past-processing, let me give this intentionally extreme example. The supplied image is 4 or 5 stops underexposed, cause my flash did not fire. (It was not charged). The camera was set to save raw and jpg separately, so we can find out, what can we bring out of them subsequently .

-

Of course, we cannot expect the wonder, but perhaps you agree, that the image rendered from RAW is less bad. By the way, please remember, what had been bring out of such an underexposed negative! (Practically transparent celluloid!)

The Racers

Look at our RAW-converters!

I'll compare six converters:

RRawShooter plays with ColorEngine profile pack. Unfortunately RawShooter had been discontinued years ago, bought up by Adobe, so the knowledge accumulated in it should be used in newer Adobe products.
CaptureOne offers several profiles for EOS-10D, and even profiles for other cameras' can be used. Furthermore - however doesn't offer automatic calibration - it has a sophisticated profile editor, giving the ability to change the colors of the image in a wide range. It is far more, than what I want to test, so I used the "Generic EOS-10D" profile.
A calibration plugin for ACR is available by a third-party developer. I tried it.
RawTherapee also supports third-party profiles, namely in standard ICC format. It's also been tested (one of the images, and the calibration targets).
DPP appears only one time, as I tried it on a test chart.

All images were converted to sRGB, directly by the RAW-converter, cause you will see them on the web.

First let' see the targets, IT8, and Gretag-Macbeth Color Checker, then some "live situation"!

White Balance

Right now, we have an interesting problem. Needless to say, acceptable quality subsequent white-balance correction can only made of a RAW file.

Let's suppose to take shoots in bulb-light, but the white balance had been set on 'cloudy' setting! Take one shot in JPG, and one in RAW! What could be brought out of one, and the other?

JPG shot is just like this::

jpg

Histogram doesn't promise any good: red channel is blown out (vertical line on the right), blue is burn. ("Run against the wall" on the left). By the way, nothing would be better, changing the exposition: increasing will increase blowing of reds, and decreasing will increase burning of blues. Additionally the green channel - what is now correct - can be blown or burned.

After 10-15 minutes of hacking:

utolagos

All it is done with these curves:

gorbe

Bad! Grays are more-or-less correct, also whites, although there is some red or yellow cast on the top of the camera, and in the shadow, dropped on the white board.

Needless to say: all this is just one click in a RAW-converter. Use the white-balance pipette, and click on third from the right gray patch of Color Checker!

konverter

No blowing-out, no burning-in! White is white, gray is gray, black is black! Colored patches of Color Chacker are near to the original, but a bit dun-colored, because I lightened the middle tones to increase shadow detail. I made it in Photoshop, so that is why the histogram is ragged. In the sake of fair play, I used Canon File Viewer Utility, what is practically equal to the EOS 10D's onboard processor. Noise reduction and sharpening had not taken, or rather, the factory presets of both camera and software had not been modified.
However the Devil is in the details! Take a closer look at the second and third image in 200% size. It's an important thing, if one is about to make prints.

Modified JPG:

utolag

RAW:

raw

I think, the blue-green and yellow cotton-wool-like cast speaks for itself, not to mention that red stripe, well known by Canon users, which is rather like the color code of a mobile telecom company on the upper image.

Moreover, setting of white balance in a RAW converter is not a real modification, as the image data in the file is exactly the same, what ever value of white balance is set on the camera. Just a few figures are being written in the file, what can be used, or ignored. Also the final JPG is rendered using this value, by the onboard processor of camera , but - as we've seen - this file can not be modified so simply.

Back to the test!

Camera was set to "Daylight", and I used "As shot" in all converters. Interesting, how much different numerical data can be read out from the same file, while the color of them differs much less. After this I wanted to know, what figures we get, setting that slightly yellow-casted test shot to neutral, using pipette tool. (Using the same patch as reference). Results can be seen in the last two rows. Who understands this?

 

Adobe

CaptureOne

RawShooter

RawTherapee

Kelvin

5450

4950

4950

5502

Tint

+2

-2

-7

-1,079

Kelvin (neutral)

4950

4650

4200

5085

Tint (neutral)

-8

-6

-12

-1,173

Color cast is legal, as I couldn't produce a correctly balanced lighting.

szurke

How many different interpretations of gray tones, and contrast! And we're not talking about colors yet, just gray tones!

Test Charts

sRGB Color Space

Of course, the charts contain several colors, that can't be displayed in sRGB. Let's talk about this first!

How shell I show you, what you cannot see? Gray-shaded zones contain colors, which cannot be displayed in sRGB.


srgb

Everyone who saves images in sRGB, relinquish these colors voluntarily. Either shots in JPG, or converts RAW files in sRGB by the RAW-converter. Of course, not irrecoverably in the second case, as the original information stays untouched in the RAW file, so it can be rerendered whenever needed.

- 'What's the use of this, since they are neither displayable, nor printable?' - Some may ask
- Deception! Although the original IT8 chart is printed on photographic paper, this accuracy cannot be expected in everyday print stores. But even the average gives more then sRGB! Gamut of three print-technologies are demonstrated on the charts above: ink-jet - better then average, but not high-end -, minilab, using chemically developed photographic paper, and offset print. The way, you can bring it out of a minilab or inkjet printer is not simple, but not impossible. It depends on the control over profiles, drivers, and others. Details are in an other article.

- 'In the real world, nobody shots test chart, everyday scenes don’t contain such colors' - Of course, the real world is much more colorful then any chart. Just one example:

eredeti sRGB

These colors, hanging out of sRGB gamut, were converted by Photoshop's profile converter, using 'relative colorimetric' intent and black point correction. As you see, this is not ideal, but this difference is negligible in this aspect.

For the sake of comparability, I present the rendered test patches together with reference patches, simulated by software, according to the factory specifications of the charts. The image of our chart should have looked so, if everything would be ideal, from the sensor of the camera to the RAW converter. Of course, if we printed this ideal image using an ideal printer, we could not differ the printed one from the original chart.

IT-8


IT8

In this case, reference image stayed unchanged, and I tended to correct the contrast of each image to match it, according to the gray patches. It was mainly made by the converter, but some fine tuning has done subsequently by Photoshop. (Of course, the camera-made JPG had been modified completely in PS). Horizontal greyscale is used just for matching contrast curve, white balance should be checked using gray patches of column 16.

Gretag-MacBeth Color Checker


Color Checker

In this case, reference image stayed unchanged, and I tended to correct the contrast of each image to match it, according to the gray patches. It was mainly made by the converter, but some fine tuning has done subsequently by Photoshop. (Of course, the camera-made JPG had been modified completely in PS). Disks represent the reference colors.

I merged the patches of Gretag-MacBeth Color Checker, rendered by different converters, to one image. For the sake of comparability, I made some swindle. As it can be seen on the grayscales above, and on curves below, default contrast curves of the converters, and hence average density of images differ so much, that comparison of colors would be impossible. Therefore I tried to change contrast and density of images, using the sliders of converters, to match each other. Opposite to the 'clickable' images above, in this case I used the gray patches of camera-made JPG as reference. In this way, also the modification of reference image, according to camera JPG, was necessary. These modified reference colors give narrow bars between test patches. Test images had been processed by PS's 'Gaussian blur' plugin, because texture of test chart would make comparison more difficult.

Color Checker szinek

I won't comment these results. Just a few notes: notice, how different is the color rendition of several converters in the case of light and dark, prime- and mixed colors! Notice, that the rendition of grey tones is a completely different story! Watch, that - for example - dark blue and green is rendered most accurately by RawShooter, while yellow by CaptuerOne, red by the on-board processor, and cyan and olive by ACR. Look at the prime colors by CaptureOne and ACR! While brightness of green is nearly equal - only hue differs a bit - blue of ACR is significantly dark, and red is light. Just the opposite as those of Capture One

Of course, in the real world - except some geeks like me - no one shots test charts, so it's almost useless to derive far-reaching conclusions of it. What is to be advantage or disadvantage among the features above, it depends on the actual scene, image, work-flow, and of course your taste. In most cases - excluding fine-art reproduction and some cases of ...... - the goal is not the perfect similarity to the original colors, rather to be nice! As it's not certainly the best Hifi speaker for listening to music, which has the best tests of intermodulation distortion, and Burst-response.

Films

OK, but what about film? What these charts look like shooting them on film? Whether the color rendition of those legendary films was accurate? Or it's a deception, what is however better to see, then the reality? Some folks - mostly those who are unskilled in IT - often say: 'That good old analogue was true! Slides have been just developed, as they dropped out of camera, and they were ready to project. No tricks, no cracking in Photoshop' Perhaps they say: 'Digital is acceptable just until we use the JPG directly from the camera. But slides have nicer colors'. In my opinion, the photographer, that uses films, also makes this tricks and hacks, just not subsequently, but previously, while decides which film to use. The tricks and hacks are made in the factories by chemists and technicians.

.............. Perhaps you'll agree, that IMAGE is created in most cases this way: the author sees something, what makes associations, connecting to his earlier experiences. This way an inner-image arises in his brain. He sees this image in a framed 50x70 print, just before the moment, of releasing the shutter. Since this moment, until he brings that print to the framer, he tries to make this inner-image to physically exist. It's nobody else's business, how much is done before and after the shot, What is emotional and what is conscious, what is done in darkroom, and what is on the computer, what is by handcrafts and what is by softwares.

Look just three examples! One professional slide (Fuji Provia-100), one professional negative (Fuji NPH-400), and one commerce negative (Agfa Vista-100). All three were scanned by Epson V750, using SilverFast AI. Although the scanner is calibrated, it only gives perfect solution for slides. Removing orange-cast of negatives, always contains some subjective aspects.


srgb

Contrast of scanned negatives was set according to the test chart, while the slide remained untouched. Any corrections should’ve been useless, because through the markedly higher contrast, gray patches, darker then No 18 have burned in. In the 1/2 stop overexposed frame, however, the highlights started to blow out. Thus exposure range of Provia is just a little bit higher, than 5 stops.
Interesting, that among RAW converters, colors of CaptureOne are most similar to that of films. Yes, to which one? Since colors of films differs from each other, at least as much as that of RAW converters. (Or even more!)

Curves

Rendition of an even gray-scale is important property of converters (as well, as film-developer combinations). Gradation curves, derived of this were important parts of product catalogues and datasheets. Usually several curves were published, informing the user about expectable changes in density, contrast, (Gamma), base-fog, and tonal range, as the function of developing time. Tried to simulate this, plotting "gradation curves" of RAW converters. X-axis represents the exposition, marking the corresponding zone values. Of course, Y-axis does not represents density, which cannot be defined in the digital domain, rather 8-bit values (0-255) in sRGB color space. (Theoretically zone 5 should be 128)

Black curve is derived from factory data of IT8 chart. Further curves illustrate the changes in parameters of applications, on the model of changes of developing parameters in the past. Unlike the ones of chemical processes - these parameters can be combined free, creating almost ever curve shape. What is visible at first view, that every curves have a characteristic 'S' shape. A more or less linear section is impacted between two curved ones. Just like the portions of classical film characteristic curves: toe, straight line, and shoulder! This similarity cannot be accidental, since the curve, derived clearly of sensor readout data, by physical, mathematical, and statistic formulas, should be linear. (What is broadly matched by black curve, although it's toe is slightly curved). This way RAW-converters simulate the behaviour of films - exactly: the analogue process, from exposed film to printed copy - increasing contrast of midtones, and compressing it in shadow and highlight areas.

Factory Settings


IT8

One black, and one green curve appear in each graphs. Black is the 'gradation curve' of IT8 chart, while green is of Provia, of course rendered from the scanned image. The default curves of each applications are shown here. Some settings and corrections are presented in the next diagram. (Of course, peak of vertical axis is 255 instead of 300, sorry for not redrawing it!)

Specific Sliders


IT8

Examples


CRW_1826

For high resolution details, click in the white frame!


For high resolution details, click in the numbered frames!

-

This time films turn up again, namely Velvia-50 slide, and Ilford Pan-50 black and white. Both are in 6x6 format. Slide have been scanned by two scanners: Nikon 8000 ED, and Epson V750.
Tripod was not used, so the viewpoint is not exactly the same, furthermore Sun was moving a bit, during the process.

For comparison, 6 MPixel RAW files, and scanned images were interpolated to cca 48 MPixel. After that, images taken from different viewpoints were fitted, paying attention for each of them to be resampled and rotated.

Nikon scanner was used at 4000 dpi, while Epson at 3200, then all files were resampled to 5000 dpi. All scanned images were subsequently sharpened, (USM) but with different parameters. (Inspace of scanners is discussed in my other article).


Take a snapshot to the images at 100% size! Differences in sharpening and demosaic algorithms can be observed. Of course, default settings were used, while each softwares provide some facilities to change them. I tried two of them: CaptureOne's 'standard' and 'soft' rendering, and different demosaic algorithms in RawTherapee: EAD, HPHD, and VNG-4. All other parameters of sharpening are default.


-

From the aspect of moiré, the worst case is to plot parallel lines, just a few degrees rotating from raster of sensor. This can be seen on the left, markedly sweating the racers. For better visibility the crops were resized to 200%. Slightly different aspects of camera-JPG ('10D') are caused by that, it was taken from different viewpoint.

Test-shot was taken using Canon EF 17-40/4L lens at 40mm. Slide was scanned by Nikon 8000 ED at 4000 dpi.

Although discussion of film is a side track in this article, please let me comment these results, before someone should proclaim the triumph of analogue technique against digital. Or just the fall of it.

Some maths again! Interference (moiré, aliasing) occurs during digitalisation, if the frequency of digitised analogue sign is higher then the half of sample rate in digital domain. (Nyquist’s law). It’s valid not just for temporal frequencies, (sounds), but also for spatial ones, like grids.
Translation: appearing of moiré on the above figure means, that image, optically drawn by lens – analogue sign – contains higher spatial frequencies (finer details), then half of frequency of the grid, formed by the pixels of sensor. Well-seen, that an image of a grid can be represented as a grid, until at least one row of pixels forms it’s lighter, and one it’s darker line. (One for the rod, an one for the hole.) That means, that two pixels are needed for each period of grid. Moiré doesn’t appear on film, as it’s grains are placed randomly, not in a raster. Spot, melted to gray in the centre of Siemens-star is however nearly the same sized as in the case of 6MP APS sensor. It means, that resolution is not limited by the lens, rather sensor, and film. Therefore resolution of Provia and 6 MP sensor is approximately equal.

Supplement:

Above mentioned snap is further detailed by resolution of scanner, and different behaviour of analogue and digital sign. While response of digital sign “ends sharply” at a given frequency (see above), that of analogue sign plots a special curve (MTF), what is characteristic of the device. For more about it see in my article about resolution (Only in Hungarian yet) and links on the left.

Further notes:

Provia’s sharpness is not the peak of all the films, nor EOS 10D’s among digital sensors.
Sharpness of films is somewhat modified during scanning process. Of course, in practice, lens of projector or enlarger also decrease it, maybe more.
If not dots per mm should be compared, but dots per image frame, we should found that film is better. (1,6 times more dots). If FullFrame sensor should used instead of APS, but the same pixel/mm ratio, results should be equal again, and sensor should won by higher pix/mm.

We could used medium-format, signing, that an acceptable medium-format analogue set can got for the price of a medium-quality DSLR set, while medium format digital set for the price of a medium-quality car. In the above test, subjective evaluation is effected by contrast, gamma, and noise of scanning and RAW conversion. Grain of film can be considered as fine details, mystifying our eyes. Conclusions, should only be derived comparing MTF curves, drawn by matching all of these parameters.


Here should be the final conclusion

Great Hungarian comedian, György SÁNDOR said in his unforgettable 'Salutatory' number: 'Don't forget our Alma Mater, and if you'll ever got any trouble in your life, then solve it yourselves!' So all draw the lesson for themselves! Everyone needs to run tests by his own taste, style, workflow, etc.

By myself, I use RawTherapee for reprography, and RawShooter for anything else, in general.

Of course, experimentation, tests, and calibration takes a lot of time, and even a lot of money! However not so much, as one can think it first! OK, test charts are needed, and they are available mainly on the net. Few years ago I took useless efforts for finding a dealer in Hungary. Nothing found. Even softwares are not cheap, but there are some other options. Of course, I don't think about file sharing! Many scanners come with some profiling options, including test charts, and many profiling applications can be downloaded free. Finally, some of the RAW converters, tested here, are open-source, which also run under Linux.


Writing of Hungarian names on this site: LAST NAME(,) First name
Everythhing on this site is VITÁNYI, Iván jr's intellectual property, unless otherwise noted

emil