An image sensor noise testHere I present a simple test of three camera sensors to compare noise between them at base ISO. The test is oriented towards users interested in high resolution photography with high dynamic range, that is typically landscape photography. I do not evaluate high ISO settings at all.
How much noise there is in the raw image file is a key image quality factor. It is not the only one, but it has large impact and is the easiest to measure and has less subjective element to it than many other image quality factors such as color rendition.
The image quality of medium format systems are often described as far superior to the smaller formats, higher resolution (obvious), better color rendition, finer tonality and higher dynamic range are often claimed. Tonality and dynamic range is very much about noise which is the factor we are going to look into here.
I do this test for the following reasons:
Unfortunately I did not have access to all those systems so this test is on the following cameras:
The testThe test procedure is very simple. I have set up a stable test scene and made an optimal ETTR exposure (verified by raw diagnostics), and from that underexposed the scene -1 to -9 stops with one stop decrements. These files are then developed in RawTherapee with the underexposure compensated for and compared visually to come to conclusions.
My goal is not to come to an actual exact measurement value, but to visually compare these cameras against eachother which I think is more relevant for practical use.
The test scene
I did not have lenses with the same field of view to all cameras so the perspective is a bit different. The ETTR exposure has been calibrated after the brightest white patch on the color checker. The small symbols on the colorchecker are even brighter and are thus overexposed.
I have not cared much about focusing, so don't try to evaluate sharpness from these pictures.
The images have been developed in RawTherapee 4 with neutral settings, that is just demosaicing, white balancing according to a default color matrix, and a default gamma curve. The underexposed images have then been pushed with the exposure slider (which in this program behaves linearly when highlight recovery is disabled).
The look of the images from 5Dmk2 and Aptus75 is very similar, while D7000 is a bit more saturated. This is due to the default color matrix for the camera, no conclusions should be drawn from that. We do this test to evaluate noise, not color.
The brightness of the Aptus75 series of images varies slightly, this is due to that a technical camera with a mechanical copal shutter has been used. These shutters are not as precise as electronic shutters so there can be a 1/4 stop variation or so.
Noise in mid-range areas of an ETTR exposureThe first reference ETTR exposure represents a perfectly exposed scene. If there is no noise issues in this exposure one can assume that the camera is "good enough" for practical use.
In the color checker color patches all cameras have very low noise as shown in Figure 1. It is by eye impossible to distinguish noise levels between the different cameras.
Noise in dark shadows of an ETTR exposureIn figure 2 we look at the black object (the lens) and then we look at light levels about 8-9 stops below saturation. In the final picture this would be rendered very dark, but since a neutral setting has been used when the test images were developed from the raw files there is no contrast-enhancing S-curve applied so it is rendered a bit brighter. Note that if this would have been an actual photograph of that lens we would have used more light!
On a calibrated screen the noise is visible, at least in the 5Dmk2 crop. Most would not consider it disturbing though.
Number of noise-free stopsHow many stops are "virtually noise free" in these images? This is too hard to see in the ETTR image (I did not have a transmission wedge), but by looking at the pushed underexposed test images we can see that the Aptus controls the noise well down to about 7 stops from saturation, as shown in figure 5.
Reaching the limitSo how dark can we go before the noise is starting to become really uncontrolled? This is shown in figure 6, this is as dark as it gets before noise is taking over.
Backlit scene shadow pushingThe obvious use of high dynamic range for a landscape photographer is to capture a backlit scene (say a sunset) without the use of a gradient filter or multiple exposures. This would mean that we underexpose the foreground with 2-3 stops, and then push it in post-processing.
In a typical sunset example the dark foreground will be 7-8 stops from saturation. Figure 8 shows the quality of these areas after being pushed 2 stops up to -6. If you want a "grunge HDR" look you would need to push much more and then the result would clearly be noisy.
I would say that the noise in the D7000 is acceptable, and also on the Aptus 75 but it is closer to the limit (typically there are structures in a real picture that reduces the impact of the random noise). The D7000 does not have a wide margin either though, so you may still want to do multiple exposures or use gradient filters.
Even if the sensor noise would be zero the darkest parts of an image would be noisy due to that the signal itself is noisy -- few photons means high shot noise. This may be the limit for the D7000. To solve this problem the sensor needs higher full well capacity so longer shutter speeds and can be used and more photons be captured. This would put the larger medium format sensor at an advantage, but as we see in this test that advantage does not compensate for its higher sensor noise.
Summary of observations
DiscussionThe Aptus Dalsa sensor is 48x36mm in size, while the D7000 Sony Exmor sensor is 24x16mm that is only 22% of the Aptus. Despite that noise performance in the top 7 stops is very similar and towards the noise floor it clearly exceeds the Aptus - that is truly impressive! On the other hand the Aptus is 5 year older than the D7000 and shows that even old medium format digital backs have very good noise performance. As discussed in the introduction the D7000 100% crops performs almost exactly the same as crops from the new full-frame D800.
That the "old" Leaf Aptus 75 fares so well is not a surprising result. The pace of CCD development has been slower, and the same sensor as used in Aptus 75 is still used in current backs today. This cannot be said about CMOS sensors used in DSLRs, back at the time of Aptus 75 was launched the CMOS noise performance at base ISO was not that good. This has changed though with the recent Sony Exmor sensors.
The Canon sensors are nowadays well-known for not having that great noise performance at base ISO, and we can clearly see that. The 5D mark 2 is far behind the other two. It is no surprise that past "MF vs DSLR" tests when Canon cameras were used showed a huge dynamic range advantage of the medium format sensors.
What do these noise levels mean in practice?So how low noise levels do you actually need? Assuming you expose well it will depend on what you do in post-processing. If you do not push shadows, but rather apply a standard contrast-increasing S-curve which will darken shadows further then 5D mark 2 is adequate. So for many use cases one can argue that you don't really need more range than the 5D mark 2 provides.
In the backlit scene scenario using the Aptus or D7000 may however provide a significant change over the 5D mark 2 - if you needed multiple exposure or gradient filters before you can possibly drop them with the other two cameras, D7000 having the greatest advantage. However, noise is not completely invisible and exposures out in the field are not always perfectly ETTR, so if you are a perfectionist you probably will want to use gradient filters or multiple exposures for all these cameras.
To truly make gradient filters obsolete I would say that you need the noise performance of the D7000 pixels plus two more stops up to saturation, so you can push with less photon shot noise.
So where is the medium format advantage?In these tests I cannot say that medium format shows any particular strong advantage over the 4.5 times smaller APS-C sensor, except resolution of course. When compared to the best DSLR sensors the "superior dynamic range" claim is no longer true when looking at 100% crops. Clearly a D800 which has D7000 pixels and the resolution of the Aptus 75 would be the stronger overall performer in terms of noise.
I did observe a lower noise level of the Aptus at better lit areas, thanks to lower photon shot noise. This should lead to finer tonal transitions which also is often claimed, but the difference in this case would be very small and hard to detect.
Still the claims of superior dynamic range has not disappeared. A quite common claim is that medium format has better ability to recover highlights, "more highlight dynamic range". This is probably due to differently tuned auto-exposure in the cameras and differently tuned histograms tools, that is DSLRs tend to expose brighter (closer to clipping) than MF systems. But when you expose manually and learn how the histograms work this difference disappears.
The most likely advantage to actually exist and be significant (except resolution!) is color rendition, but this test doesn't look into that. The theory goes that the MF CCDs have color filters tuned for best color at the expense of high ISO performance (which will be bad anyway due to the CCD), while the DSLR CMOSes have filters better suited to keep good sensitivity at high ISO. I do not know if this is true and if it significantly affects performance, but I have seen side-by-side tests where skin color was clearly more accurate out-of-the-box for medium format. It is an interesting aspect that could be investigated further.
It is also clear that when you work with short depth of field the lenses used on the medium format systems will produce a different look than DSLR lenses, and that look may be important to you. The effect is probably strongest at mid-range depth of fields (short but not the shortest possible) and maybe this is what produces the enhanced "3D look" that some claim to see. Just like the color rendition aspect this is something that needs further investigation.
Exposing in practiceFine-tuning the exposure to just under clipping through raw diagnostics as done in this test is of course not practical in the field. So how far from optimal will these cameras expose if you use the tools at hand?
For manual still-life and landscape photography the "highlight blinkies" is the typical tool used, apart from the histogram. Overexpose slightly, re-adjust until highlights stop blinking. The most exact implementation would only start to blink if any raw channel gets clipped. However, the Canon and Nikon cameras analyze the embedded JPEG preview. The Leaf looks at the raw data, but checks only luminance.
In a high contrast scene with daylight and white highlights the blinking starts about 1/3 stop before optimal ETTR for all the tested cameras. This is good. In low contrast scenes the Canon/Nikon may push the jpeg and thus cause underexposure, but that does not matter much since it is a low contrast scene.
The worst case is with colored highlights which through highlight reconstruction may be darkened to fit the sRGB or AdobeRGB color space and may not even blink at all in the JPEG preview despite clipped raw data. This can happen on the Canon, unfortunately I did not have time to test that on the Nikon. Clipping may be seen in one of the individual RGB histograms though. To reduce these errors it is better to set the camera to the larger AdobeRGB color space (closer to what the camera can capture in raw) which then is used in the JPEG previews.
The Leaf Aptus raw-based histogram is much more reliable, but unfortunately the blinkies is only luminance-based. A technique that can be used on this back is to start with a slight under-exposure, point at the highlight in the preview and get a readout of exposure, and adjust the shutter speed to something that puts it just under clipping. It should be noted that on technical cameras with mechanical copal shutters there are only full stop shutter speeds, if one wants to adjust in 1/3 stops the aperture must be used.
Long exposuresThe noise properties change greatly when exposing for tens of seconds. This is not tested here, but perhaps in a future test. Worth noting is that the Aptus is limited to 30 second exposures, since the sensor has too much dark noise (thermal noise) to allow longer. Result at 30 seconds should generally be good though if hot pixels are cleaned.
DxOMark dynamic range vs photographic useful dynamic rangeThe dynamic range measurement on DxOMark is a straight-forward traditional scientific signal-to-noise measurement which shows how many stops down from saturation one can go before noise reaches equal level as the signal (S/N ratio = 1). On modern camera sensors this is somewhere between 11 to 14 stops. Many thus believe that modern cameras have as much as 14 stops of useful dynamic range. However, the stops get noisier and noiser the farther from saturation it is down to the last stop where the S/N ratio is 1 and then the image of course looks absolutely horrible if it would be brightened. So this measurement does not really say how many stops that are useful to the photographer (which I have tried to answer to some extent in this test here).
Photographic useful dynamic range is subjective and is considerably less than the scientifically measured. It is not as bad as one may think though because the noise appears where signal is weak that is in the shadows, and since shadows are dark noise is more acceptable there. How much noise that can be accepted will also be highly dependent on the actual scene, noise in a large single-color patch will be more disturbing than a section of a distant forest with lots of branches and other "random" small details.
The nature of the noise is also important, some cameras produce distinctive patterns in the noise (like the tested Canon 5D mark 2) while others have almost completely random noise. Random noise is much less disturbing to the eye.
As an example we can see how the Aptus 75 (75s on DxOMark), D7000 and 5D mark 2 compare pixel per pixel in the DxOMark dynamic range diagram (set to "Screen" mode to get pixel comparison). Nikon D7000 has 13.35 stops, Aptus 75s 11.43 and 5D mark 2 11.16 stops. Rounding a bit one could say that per pixel D7000 is 2 stops better than Aptus 75s which is 1/3 stop better than 5D mark 2. These are high quality scientific measurements of S/N ratio so there is no doubt that they are correct.
However, when S/N approaches 1 at the bottom stops the noise level is so high it is hard for the eye to differ (or care) between differences. Pattern noise also affects the subjective impact. The result of the visual test made here is Nikon D7000 2/3 stops better in dark shadows (considerable less than 2 stops!) and 5D mark 2 is about 1 1/3 stop worse than Aptus 75 (considerable more than 1/3 stops). This shows that an S/N=1 dynamic range measurement is not a good indication of actual performance in practice.
DxOMark also presents a measurement of S/N ratio for 18% gray, that is about 3 stops down from saturation. In the test done here this is the reference middle light level in the ETTR exposure and we can conclude that cameras of any decent quality are then impossible to differ, and indeed DxOMark shows also only very small differences, generally an advantage to cameras with better full well capacity (less shot noise). Only when throwing in a small compact camera (finger nail size sensors) one can see a significant difference at this light level.
When looking only at measurements for one camera (not in comparison mode), DxOMark provides "Full SNR" a complete graph of signal to noise ratio at all gray levels. This way you can see how noisy a sensor is for any suitable light level you choose. Since this is not available in comparison mode few DxOMark users use it or even know about it though.
At -8 stops (0.39%) the Aptus 75s gets 18.9 dB, D7000 20.8 dB and 5Dmk2 18 dB, that is about 2/3 stop advantage for the D7000 to the Aptus and yet 1/3 down to the 5Dmk2. This is similar to what I see here visually, but since an SNR graph does not take pattern noise into account the 5Dmk2 gets a full stop closer than in my test.
To sum up I would say that DxOMark does not provide any really good measurement for practical dynamic range when in comparison mode, the dynamic range diagram digs too deep into the noise, and the SNR 18% is too bright light level to show any significant differences. The full SNR graph available when viewing a single camera provides more information, but must also be used with care since it will not take patterned noise, blotchiness or other noise quality aspects into account.
Common mistakes in DR testingDynamic range tests MF vs DSLR etc similar to my test here has been done several times before but often contain errors. These are common ones:
If the scene contains the sun or other very strong highlight one have to clip some, but to simplify let us assume we have a normally bright highlight which we do not need to clip, as in the test scene. To expose optimally we want to gather as much light as possible and thus put that highlight as close to clipping as possible - "expose to the right", that is the histogram should be as far right as possible without clipping.
Problem is that almost all cameras have slightly misleading histograms, Canon and Nikon cameras show the histograms for the embedded JPEG preview, not the raw data. The Aptus used in this test has raw histograms (as far as I know) but show highlight clipping before actual clipping occurs. In practical use the histograms are generally ok, but for exact exposure for testing one will have to analyze the RAW file with proper software and find optimal exposure through trial and error.
Highlight clipping caveatsMost sensors have three color channels - red, green and blue. Half of the pixels are green, 1/4 red and 1/4 blue, and to form the final picture which needs complete RGB values for each pixel the colors are interpolated through a demosaicing algorithm.
Typically the green pixels are most sensitive and clip first when a highlight is overexposed. The property that one channel at a time is clipped means that discolourings can occur in clipped areas, which raw converter highlight reconstruction algorithms try to hide, and succeed in most cases.
Another aspect is that to get proper white balance the RGB channels are re-balanced, which may mean that clipping can be introduced in the raw converter in a near-saturated highlight despite that it is not clipped in the raw file.
Due to demosaicing, highlight reconstruction and white balancing it is in most software hard to actually see if a highlight is clipped or not if only one channel is clipped in the raw file.
Raw conversion for diagnosticsAvailable for free and on many platforms is the dcraw RAW converter. This can be used to make a no-fuzz raw conversion to find out if an area clips or not.
To also skip demosaicing you can run dcraw in "document" mode, like this: