An image sensor noise test

How much noise there is in the raw image file is a key image quality factor. It is not the only one, but it has large impact and is the easiest to measure and has less subjective element to it than many other image quality factors such as color rendition.

The image quality of medium format systems are often described as far superior to the smaller formats, higher resolution (obvious), better color rendition, finer tonality and higher dynamic range are often claimed. Tonality and dynamic range is very much about noise which is the factor we are going to look into here.

I do this test for the following reasons:

• Despite frequent claims of the medium format superior image quality, few actual tests look into noise performance for real, and those that do are often poorly made with misleading results.
• Many seem to have missed the huge progress in noise performance in DSLRs that came with the introduction of the Sony Exmor sensor technology used in for example Nikon D800 and D7000.
• Second hand medium format is today a fairly affordable option for serious amateur photographers so it is relevant to test how these perform compared to modern alternatives.
• The popular website DxOMark that does provide high quality scientifically made measurements of many sensors including medium format presents them only in graph form which is hard to interpret, here the purpose is to show results visually and discuss the practical meaning of them.

Unfortunately I did not have access to all those systems so this test is on the following cameras:

• Nikon D7000. An APS-C camera which has a Sony Exmor sensor, which presents a break-through in dynamic range. Pixel size is almost exactly the same as the new full-frame D800 and other well-made measurements show that per pixel it performs almost exactly the same. Thus we can consider the 100% crops from D7000 as a good representation of 100% D800 crops.
• Canon 5D mark 2. A very popular full-frame 135 camera from Canon. Canon is currently known for not having as good dynamic range in their sensors as the competitors, but since this camera up to very recently was one of the most popular "high resolution" DSLR cameras most "MF vs DSLR" tests were made against this. Unfortunately for Canon users the recently released 5D mark 3 only provides a minor improvement in dynamic range at base ISO.
• Leaf Aptus 75. Medium format digital back 36x48mm sensor. Released late 2005 this 33 megapixel discontinued digital back is one good alternative for an amateur that wants to invest in second hand medium format gear. Although not the latest digital back it still has comparable DR to many current backs (not as good as IQ180 though!). In the test it has been mounted to a Linhof Techno with a 90mm lens. Lens is long enough to have negligible color cast.

The test

My goal is not to come to an actual exact measurement value, but to visually compare these cameras against eachother which I think is more relevant for practical use.

The upper row shows the original in-camera JPEG preview of 5Dmk2 test shots, with optimal ETTR to the left and then 1 - 9 stops underexposed. The bottom row shows the same files neutrally developed in RawTherapee and then exposure compensated 1 - 9 stops to get equal brightness. Obviously noise gets worse with more stops compensated.

The test scene

Test scene with color checker and a black object in the form of a camera lens, lit with a daylight lamp.

I did not have lenses with the same field of view to all cameras so the perspective is a bit different. The ETTR exposure has been calibrated after the brightest white patch on the color checker. The small symbols on the colorchecker are even brighter and are thus overexposed.

I have not cared much about focusing, so don't try to evaluate sharpness from these pictures.

The images have been developed in RawTherapee 4 with neutral settings, that is just demosaicing, white balancing according to a default color matrix, and a default gamma curve. The underexposed images have then been pushed with the exposure slider (which in this program behaves linearly when highlight recovery is disabled).

The look of the images from 5Dmk2 and Aptus75 is very similar, while D7000 is a bit more saturated. This is due to the default color matrix for the camera, no conclusions should be drawn from that. We do this test to evaluate noise, not color.

The brightness of the Aptus75 series of images varies slightly, this is due to that a technical camera with a mechanical copal shutter has been used. These shutters are not as precise as electronic shutters so there can be a 1/4 stop variation or so.

Noise in mid-range areas of an ETTR exposure

In the color checker color patches all cameras have very low noise as shown in Figure 1. It is by eye impossible to distinguish noise levels between the different cameras.

Figure 1. Colors from the color checker in the ETTR exposure. 100% crops, from left to right: Aptus 75, 5Dmk2 and D7000. The brown color is about 4 stops below saturation.

Noise in dark shadows of an ETTR exposure

On a calibrated screen the noise is visible, at least in the 5Dmk2 crop. Most would not consider it disturbing though.

Figure 2. Detail from the black lens in the ETTR exposure. 100% crops, from left to right: Aptus 75, 5Dmk2 and D7000. The black color between the two green numbers "2" and "3" is about 8.5 stops below saturation. Note that the D7000 image may appear brighter (especially the numbers) due to that it is more in focus.

Figure 3. Same as figure 2 but the images has been pushed 3 stops to make noise more visible.

Figure 4. Same as figure 3 but the images have all been downscaled to 8 megapixel images. From left to right: Aptus 75, 5Dmk2 and D7000.

Number of noise-free stops

Figure 5. The darkest colors we can have before noise start to become apparent on the best camera. Dark color patches from 3 stop underexposed image, then brightened to show noise, original brightness shown in bottom left corner. From left to right: Aptus 75, 5Dmk2 (bottom) and D7000. The blue patch (darkest) is about 7 stops from saturation, green -5.7, red -6.5. The dark gray forming the grid between patches is at -7.3 stops.

Reaching the limit

Figure 6. The darkest colors we can have before noise becomes uncontrolled (very significant noise increase to next stop darker), level chosen individual for each camera to as close as possible match eachothers' noise levels. The original brightess shown in small patch. From left to right: Aptus 75, 5Dmk2 (bottom) and D7000. D7000 green patch at about -10 stops, Aptus at -9 and 5Dmk2 at -8.

Figure 7. One stop darker than figure 6, here the noise is considered "uncontrolled". From left to right: Aptus 75, 5Dmk2 (bottom) and D7000. Same spacing as before, D7000 green patch at about -11 stops, Aptus at -10 and 5Dmk2 at -9. Not seen in this crops, but in full size image the Aptus show clear noise uneveness due to dark noise, and the D7000 has some mild horizontal streaks. The 5Dmk2 has had patterned noise from the start.

Backlit scene shadow pushing

In a typical sunset example the dark foreground will be 7-8 stops from saturation. Figure 8 shows the quality of these areas after being pushed 2 stops up to -6. If you want a "grunge HDR" look you would need to push much more and then the result would clearly be noisy.

Figure 8. Typically backlit scene shadow pushing scenorio, -8 stop shadowed areas push up two stops to -6. From left to right: Aptus 75, 5Dmk2 (bottom) and D7000.

I would say that the noise in the D7000 is acceptable, and also on the Aptus 75 but it is closer to the limit (typically there are structures in a real picture that reduces the impact of the random noise). The D7000 does not have a wide margin either though, so you may still want to do multiple exposures or use gradient filters.

Even if the sensor noise would be zero the darkest parts of an image would be noisy due to that the signal itself is noisy -- few photons means high shot noise. This may be the limit for the D7000. To solve this problem the sensor needs higher full well capacity so longer shutter speeds and can be used and more photons be captured. This would put the larger medium format sensor at an advantage, but as we see in this test that advantage does not compensate for its higher sensor noise.

Summary of observations

Figure 9. Summary of observations presented in a scale of color checkers from the original test files in their original brightness. Approximate light level from saturation shown below the images.

• Down to about 4.5 stops from saturation the cameras are hard to differ from eachother, all of them have low noise.
• Compared to the others the 5Dmk2 has evident noise problems in darker areas.
• The Aptus and D7000 is relatively close, the D7000 has an advantage due to lower noise from the chip while the Aptus has an advantage due to higher full well capacity and thus lower photon shot noise.
• Both the Aptus and the D7000 has random noise (except very close to the noise floor), while the 5Dmk2 has an evident pattern noise which makes the noise more disturbing and reduces subjective dynamic range further.
• The Aptus is very slightly better than the D7000 down to about 7 stops from saturation (max 1/3 stop better if even that), then read/dark noise becomes a factor and the D7000 wins with about 2/3 stops towards the noise floor.
• If scaled for resolution the Aptus performs better related to the D7000 but is still not as good in the darkest shadows.
• The 5Dmk2 does not perform that bad in the brighter areas, probably even better than the D7000 if close to saturation, but from about 6-7 stops from saturation noise becomes a real problem, especially due to the pattern. In the darkest shadows its noise is roughly 2 stops higher than the D7000.
• If an image is not brightened in post-processing both Aptus and D7000 has controlled noise down to almost pitch black, while 5Dmk2 noise becomes uncontrolled just before, which makes noise visible in 100% crops on a calibrated screen but rarely in a print.
• For the use case of brightening shadows in an underexposed backlit scene the D7000 and Aptus have a significant advantage to the 5Dmk2 because the common case is that it can be done with acceptable results with the D7000/Aptus but not with the 5Dmk2. The D7000 is a little better than the Aptus in this use case but not significantly so.

Discussion

That the "old" Leaf Aptus 75 fares so well is not a surprising result. The pace of CCD development has been slower, and the same sensor as used in Aptus 75 is still used in current backs today. This cannot be said about CMOS sensors used in DSLRs, back at the time of Aptus 75 was launched the CMOS noise performance at base ISO was not that good. This has changed though with the recent Sony Exmor sensors.

The Canon sensors are nowadays well-known for not having that great noise performance at base ISO, and we can clearly see that. The 5D mark 2 is far behind the other two. It is no surprise that past "MF vs DSLR" tests when Canon cameras were used showed a huge dynamic range advantage of the medium format sensors.

What do these noise levels mean in practice?

In the backlit scene scenario using the Aptus or D7000 may however provide a significant change over the 5D mark 2 - if you needed multiple exposure or gradient filters before you can possibly drop them with the other two cameras, D7000 having the greatest advantage. However, noise is not completely invisible and exposures out in the field are not always perfectly ETTR, so if you are a perfectionist you probably will want to use gradient filters or multiple exposures for all these cameras.

To truly make gradient filters obsolete I would say that you need the noise performance of the D7000 pixels plus two more stops up to saturation, so you can push with less photon shot noise.

So where is the medium format advantage?

I did observe a lower noise level of the Aptus at better lit areas, thanks to lower photon shot noise. This should lead to finer tonal transitions which also is often claimed, but the difference in this case would be very small and hard to detect.

Still the claims of superior dynamic range has not disappeared. A quite common claim is that medium format has better ability to recover highlights, "more highlight dynamic range". This is probably due to differently tuned auto-exposure in the cameras and differently tuned histograms tools, that is DSLRs tend to expose brighter (closer to clipping) than MF systems. But when you expose manually and learn how the histograms work this difference disappears.

The most likely advantage to actually exist and be significant (except resolution!) is color rendition, but this test doesn't look into that. The theory goes that the MF CCDs have color filters tuned for best color at the expense of high ISO performance (which will be bad anyway due to the CCD), while the DSLR CMOSes have filters better suited to keep good sensitivity at high ISO. I do not know if this is true and if it significantly affects performance, but I have seen side-by-side tests where skin color was clearly more accurate out-of-the-box for medium format. It is an interesting aspect that could be investigated further.

It is also clear that when you work with short depth of field the lenses used on the medium format systems will produce a different look than DSLR lenses, and that look may be important to you. The effect is probably strongest at mid-range depth of fields (short but not the shortest possible) and maybe this is what produces the enhanced "3D look" that some claim to see. Just like the color rendition aspect this is something that needs further investigation.

Exposing in practice

For manual still-life and landscape photography the "highlight blinkies" is the typical tool used, apart from the histogram. Overexpose slightly, re-adjust until highlights stop blinking. The most exact implementation would only start to blink if any raw channel gets clipped. However, the Canon and Nikon cameras analyze the embedded JPEG preview. The Leaf looks at the raw data, but checks only luminance.

In a high contrast scene with daylight and white highlights the blinking starts about 1/3 stop before optimal ETTR for all the tested cameras. This is good. In low contrast scenes the Canon/Nikon may push the jpeg and thus cause underexposure, but that does not matter much since it is a low contrast scene.

The worst case is with colored highlights which through highlight reconstruction may be darkened to fit the sRGB or AdobeRGB color space and may not even blink at all in the JPEG preview despite clipped raw data. This can happen on the Canon, unfortunately I did not have time to test that on the Nikon. Clipping may be seen in one of the individual RGB histograms though. To reduce these errors it is better to set the camera to the larger AdobeRGB color space (closer to what the camera can capture in raw) which then is used in the JPEG previews.

The Leaf Aptus raw-based histogram is much more reliable, but unfortunately the blinkies is only luminance-based. A technique that can be used on this back is to start with a slight under-exposure, point at the highlight in the preview and get a readout of exposure, and adjust the shutter speed to something that puts it just under clipping. It should be noted that on technical cameras with mechanical copal shutters there are only full stop shutter speeds, if one wants to adjust in 1/3 stops the aperture must be used.

Long exposures

Appendix

DxOMark dynamic range vs photographic useful dynamic range

Photographic useful dynamic range is subjective and is considerably less than the scientifically measured. It is not as bad as one may think though because the noise appears where signal is weak that is in the shadows, and since shadows are dark noise is more acceptable there. How much noise that can be accepted will also be highly dependent on the actual scene, noise in a large single-color patch will be more disturbing than a section of a distant forest with lots of branches and other "random" small details.

The nature of the noise is also important, some cameras produce distinctive patterns in the noise (like the tested Canon 5D mark 2) while others have almost completely random noise. Random noise is much less disturbing to the eye.

As an example we can see how the Aptus 75 (75s on DxOMark), D7000 and 5D mark 2 compare pixel per pixel in the DxOMark dynamic range diagram (set to "Screen" mode to get pixel comparison). Nikon D7000 has 13.35 stops, Aptus 75s 11.43 and 5D mark 2 11.16 stops. Rounding a bit one could say that per pixel D7000 is 2 stops better than Aptus 75s which is 1/3 stop better than 5D mark 2. These are high quality scientific measurements of S/N ratio so there is no doubt that they are correct.

However, when S/N approaches 1 at the bottom stops the noise level is so high it is hard for the eye to differ (or care) between differences. Pattern noise also affects the subjective impact. The result of the visual test made here is Nikon D7000 2/3 stops better in dark shadows (considerable less than 2 stops!) and 5D mark 2 is about 1 1/3 stop worse than Aptus 75 (considerable more than 1/3 stops). This shows that an S/N=1 dynamic range measurement is not a good indication of actual performance in practice.

DxOMark also presents a measurement of S/N ratio for 18% gray, that is about 3 stops down from saturation. In the test done here this is the reference middle light level in the ETTR exposure and we can conclude that cameras of any decent quality are then impossible to differ, and indeed DxOMark shows also only very small differences, generally an advantage to cameras with better full well capacity (less shot noise). Only when throwing in a small compact camera (finger nail size sensors) one can see a significant difference at this light level.

When looking only at measurements for one camera (not in comparison mode), DxOMark provides "Full SNR" a complete graph of signal to noise ratio at all gray levels. This way you can see how noisy a sensor is for any suitable light level you choose. Since this is not available in comparison mode few DxOMark users use it or even know about it though.

At -8 stops (0.39%) the Aptus 75s gets 18.9 dB, D7000 20.8 dB and 5Dmk2 18 dB, that is about 2/3 stop advantage for the D7000 to the Aptus and yet 1/3 down to the 5Dmk2. This is similar to what I see here visually, but since an SNR graph does not take pattern noise into account the 5Dmk2 gets a full stop closer than in my test.

To sum up I would say that DxOMark does not provide any really good measurement for practical dynamic range when in comparison mode, the dynamic range diagram digs too deep into the noise, and the SNR 18% is too bright light level to show any significant differences. The full SNR graph available when viewing a single camera provides more information, but must also be used with care since it will not take patterned noise, blotchiness or other noise quality aspects into account.

Common mistakes in DR testing

• In-camera histograms, raw converter histograms, auto-exposure or similar is used to find optimal ETTR exposure rather than using proper raw diagnostics. The result is slightly under- or overexposed files which may put one system at an unfair advantage.
• Raw converter exposure slider are thought to work linearly, that is +1 stop on the slider means +1 stop in the file, but very few raw converters are linear due to highlight reconstruction, default contrast increase and other features. The result is wrong estimations of available dynamic range.

If the scene contains the sun or other very strong highlight one have to clip some, but to simplify let us assume we have a normally bright highlight which we do not need to clip, as in the test scene. To expose optimally we want to gather as much light as possible and thus put that highlight as close to clipping as possible - "expose to the right", that is the histogram should be as far right as possible without clipping.

Problem is that almost all cameras have slightly misleading histograms, Canon and Nikon cameras show the histograms for the embedded JPEG preview, not the raw data. The Aptus used in this test has raw histograms (as far as I know) but show highlight clipping before actual clipping occurs. In practical use the histograms are generally ok, but for exact exposure for testing one will have to analyze the RAW file with proper software and find optimal exposure through trial and error.

Highlight clipping caveats

Typically the green pixels are most sensitive and clip first when a highlight is overexposed. The property that one channel at a time is clipped means that discolourings can occur in clipped areas, which raw converter highlight reconstruction algorithms try to hide, and succeed in most cases.

Another aspect is that to get proper white balance the RGB channels are re-balanced, which may mean that clipping can be introduced in the raw converter in a near-saturated highlight despite that it is not clipped in the raw file.

Due to demosaicing, highlight reconstruction and white balancing it is in most software hard to actually see if a highlight is clipped or not if only one channel is clipped in the raw file.

Raw conversion for diagnostics

dcraw -v -r 1 1 1 1 -o 0 -H 0 -T -W -g 1 1 file

To also skip demosaicing you can run dcraw in "document" mode, like this:

dcraw -v -d -o 0 -H 0 -T -W -g 1 1 file