Saturday, February 16, 2008

First K20D Production Camera Formal Test

PopPhoto has completed testing a production unit (yes, they mention it is!) of the K20D and the full test report has been published:-

http://www.popphoto.com/cameras/5112/camera-test-pentax-k20d.html

So, for such a formal test, of course our main concern is on the various measurement results which are obtained under their controlled lab environment, in their professionally set-up lab. Here is the direct link to the K20D lab test results page:-

http://www.popphoto.com/cameras/5112/camera-test-pentax-k20d-rugged-body-smart-controls-page3.html

In order to make the obtained results more meaningful, we need to interpret the results by comparing the results of other DSLRs in the "same class", i.e., the targeted market segment. As the K20D is priced somewhere near the Canon 40D, Nikon D300 and the Sony A700 (said also in the PopPhoto's article) and all are the latest offers from the four brands, it's just trivial and fair enough to compare them.

Specifications wise, putting aside the old 16-segment multi-pattern metering system in the K20D (which has the least number of segments in the industry) and that all other DSLRs has a more sophisticated AF system on paper, the K20D do have a similar specification close to the 40D and the D300.

In order to do a more systematic comparison, we need to have more controlled tested data of the other DSLRs above. Very fortunately, PopPhoto have already tested all of them in their previous issues, the direct links of the corresponding results pages are as follows:-

Canon 40D:
http://www.popphoto.com/cameras/4668/camera-test-canon-eos-40d-specifications-certified-test-results-page3.html

Nikon D300:
http://www.popphoto.com/cameras/4911/camera-test-nikon-d300-vital-statistics-test-results-page3.html

Sony A700:
http://www.popphoto.com/cameras/4724/camera-test-sony-alpha-700-vital-statistics-page3.html

Now, we can go on to compare the results aspect by aspect. I summarise the key performance aspect results above in the following tables for easy reading and our quick comparison:-

Table 1:- Noise

CameraISO 100200400800160032006400
Pentax K20D1.22 (Very Low)1.49 (Very Low)1.76 (Low)1.64 (Low)2.6 (Moderate)3.8 (Unaccepatable)6.04 (Unaccepatable)
Canon 40D0.8 (Extremely Low)0.95 (Extremely Low)1.15 (Very Low)1.49 (Very Low)2.0 (Moderately Low)2.9 (Moderate)N/A
Nikon D300N/A1.1 (Extremely Low)1.2 (Very Low)1.1 (Extremely Low)1.4 (Very Low)1.95 (Low)2.83 (Moderate)
Sony A7001.25 (Very Low)1.3 (Very Low)1.6 (Low)1.7 (Low)2.5 (Moderate)2.7 (Moderate)3.1 (Unaccepatable)

As we can see, noise wise the K20D is just loser here. At lower ISO speeds, it by no means can match the 40D and the D300 for their superior noise performance. At higher ISO speeds, it just always loses, even to the Sony A700 (which also loses to the Canon and Nikon).

Table 2:- AF Speed (in Second)

CameraEV 12
(at ISO 100)
10864210-1-2-3
Pentax K20D*0.370.44
0.43
0.54
0.97
1.171.221.672.33
--
Canon 40D0.380.400.460.490.540.580.610.840.981.20-
Nikon
D300
0.320.330.410.510.670.780.900.930.950.98-
Sony A7000.240.250.310.360.360.890.851.181.16-1.37

* Update (Feb 23): The timing figures are actually K10D ones and I have also corrected the figures above as PopPhoto have already updated their chart in the K20D test report. In conjunction, I have also made two minor changes in the second paragraph below accordingly. For more about the story, see the post made by Jack (the PopPhoto forum adminstrator) in the thread below:-

http://forums.popphoto.com/camera/board/message?board.id=11&thread.id=16705

Their main argument for using the K10D data is just that as far as they learnt from Pentax, the K10D and K20D share the same AF system and in order to avoid further "confusion", they opted to publish the K10D timing results after carried out some tests for verification. Anyway, I myself am not convinced with this "reasoning", though.


Well, as many of the Pentax DSLR users, especially for those who have compared their DSLRs of other brands side by side when they were using their Pentax DSLRs at lower light environments (including me), the lower light AF performance is just pathetic for *any* Pentax DSLRs, the K20D is of no exception, as it has been verified from the above same lab environment, again.

It should be noted that at EV -1, the K20D was simply nearly unable to focus at a rather lengthy time of 2.33s which is in fact impractical to use for the AF. Actually, no other DSLRs would ever require more than 1.37s to focus even at EV -2 and EV -3 where at such a dim lighting level there is simply no figure obtainable for the K20D as it is very probably that the K20D had given up already to do the AF job at EV -2.

Not even to say actually the AF systems of Canon, Nikon and Sony all are designed to have the f/2.8 AF sensors which cater for faster lenses for better AF accuracy when used with those lenses, mostly primes, which Pentax simply lacks such a design.

Table 3:- Other Measurements

CameraResolution (lines)Color Accuracy (Average Delta E)Highlight/Shadow DetailsContrast
Pentax K20D2350 at ISO 100-4007.98 (Excellent)HighNormal
Canon 40D2100 at ISO 100-16007.7 (Excellent)Very HighSlightly Low
Nikon D3002350 at ISO 200-64007.19 (Excellent)Extremely HighSlightly High
Sony A7002280 at ISO 1008.9 (Extremely High)Very HighSlightly High

The K20D seems to have high resolution on a par with the D300. However, at higher ISO, it fails to keep the high resolution figures which the D300 has been able to maintain. But since the K20D has more pixels than the D300, the results are somehow unexpected.

As for the color accuracy, the 40D and D300 both do better than the K20D whilst the A700 is worst.

However, for the preservation of highlight and shadows details (also commonly referred to the Dynamic Range or simply DR), the K20D comes last. The K20D has the most accurate contrast, though. Do note that normally a lower contrast would normally result for a wider DR. It is therefore the Nikon and the Sony should have the potential to yield an even wider DR range if the contrast is tuned down a bit.

Table 4:- Overall Image Quality (IQ) Grading

CameraIQ at ISO 100IQ at ISO 200-800IQ at 1600IQ at 3200IQ at 6400
Pentax K20DExcellentExcellentExcellentExtremely HighNot Rated
Canon 40DExcellentExcellentExcellentExtremely HighNot Applicable
Nikon D300Not ApplicableExcellentExcellentExcellentExtremely High
Sony A700
Excellent
ExcellentExtremely HighExtremely HighNot Applicable

Well, I think the last grading is redundant, after we have gone through the results of all the above key performance aspects measured. :-)

Anyway, after reading all these, there is yet only one question comes into my mind: Could and should Pentax do better? Could they?? Actually, I really just start to puzzle.. ( but I still know that *Pentax* MZ cameras' AF are better - at least faster at low light! :-( )

21 Comments:

Anonymous said...

Hmmm, the tests and comments on the K20D seemed positive to me. However, your interpretation is that the K20D is a poor camera or one that need much improvement. I have a difficult time reconciling your statements to the review.

RiceHigh said...

If you read and digested figures rather than on words solely, you would have learnt the truth.

Julie said...

Why is the last grading redundant? Overall image quality is the concern, not each individual factor. Who gives a shit if a camera measures low noise but absolutely annhilates any detail? Its the final product that counts, not single digit variances in unknown testing.

nlx said...

as always you're not 100% wrong but you concentrate only on the negative side and highlight it, here is what the site conclude :

"Will all of these new capabilities, combined with the well-proven aspects of the K10D, be enough for the K20D to challenge the Canon EOS 40D? We think so, except at the upper ISO limits and in extremely low-light AF challenges. The price is competitive with -- or even below -- DSLRs in the nearby 12MP class, and it's built to outlast most lower-cost models."

From what i can understand its more something like 'its a good competitor with some area for improvments" more than "its crap"

What i would like to see here is real test of the K20d from you, it shouldn't be so hard to get a K20 for a day or two for someone so pasionate about Pentax :)

Btw, is there any other Pentax centred blog around ?

Peter Fang said...

Hi Michael,

It's been a while that I've come to your blog but I'd like to point out something quite interesting regarding the PopPhoto K20D review. If you take a look at their K10D review more than a year ago:

http://www.popphoto.com/cameras/3512/camera-test-pentax-k10d.html

You can see that in the AF speed section, the numbers they published for the K10D is 100% identical to that of the K20D in this March '08 issue. As PopPhoto also tested the DA* 16-50 lens which has SDM, I really don't think they tested the K20D's AF speed with an SDM-capable lens and came away with same numbers as a screw-driven AF lens on the K10D, in fact I suspect that they thought since the K20D's AF system is also called SAFOX VIII, they probably just copied the K10D numbers and pasted it over! If that's not the case, I would truly admire their testing methodology which has such consistent results across two different bodies more than a year later with several firmware revisions and new AF code for SDM...

Anonymous said...

His "passion" is for bashing Pentax. He seizes on any iota that can be construed in a negative way and uses it to continue his harangue.

nlx said...

very interesting Peter Fang, thanks,

maybe someone should ask them about it ?

RiceHigh said...

Peter, thanks for pointing that irregularity out. I think you've made a valid and crucial point here.

Even though, the K10D and K20D AF timings are not 100% identical indeed - there is a not-so-obvious error (if our suspect is true), the K20D timings shift one EV higher except at EV12, so making the K20D figures look even worse!

Afterall, I think it is just not an interesting thing for the observation - it is in fact a very serious thing, I think. It makes the PopPhoto loses creditability if they just simply didn't measure the timings for a new camera before they published the results, plus, there is an additional error when they "copied and pasted" the data. I hope this is actually caused by some kind of human errors if it is really the case.

Nonetheless, regarding the actual AF performance of the K20D. What improvements should we expect for the K20D in the AF and AE metering departments if everything just appears to be the same - as all the previous Pentax DSLRs - which are infamous for poor low light AF performance and accuracy as well as inaccurate metering with huge underexposure tendency? Should these "high expectation" and "firm believe" be just called "wishful thinking" in reality? Just think about it.

RiceHigh said...

> nlx said...
> as always you're not 100% wrong but you concentrate only on the negative side and highlight it

Yes, that's what a "negative" blog is for! ;-)

Btw, whilst it can be clearly seen that Pentax has been staying at where they were at (many) years ago, e.g., the original film *ist SLR's AF and AE metering system, versus other competitors have already vastly improved their products and have made various significant improvements, I don't think the present overall (over-) "positivity" shown by many Pentax fans is anything that is actually beneficial to anyone afterall.

Anonymous said...

Let me point out a few points that lead me to conclude that this is an extremely biased writeup with regards to the K20D.

I mean come on - every positive test result in terms of figures is put down by him, and every negative test result drummed upon. Was there any mention that despite marginally poorer performance at higher ISOs for noise compared to Sony A700, the K20D is marginally better at low ISOs?

Good resolution, comparable to D300 is nitpicked at, claiming that at higher ISOs, the D300 can upkeep this while the K20D cannot. Oh, so what happened to the claims that you are doing a "systematic comparison"? It does seem like you have pigeon-holed everything into "against K20D" and are flogging it for all it's worth.

Lastly, the dynamic range/contrast paragraph just makes my day. If I were as close-minded as you, I might retort that we could use HDR to overcome limited dynamic range, since you say that tweaks to contrast would increase dynamic range, blah blah blah.

Oh, rest it already. If you are biased against Pentax digital SLRs, just say it out loud, and do not hide under a pretense of making a fair comparsion. You make me laugh.

Peter Fang said...

hi nlx,

I posted the question regarding the K10D/K20D AF speed numbers on PopPhoto's forum and let's see what they say about this...

I replied to this thread:

http://forums.popphoto.com/camera/board/message?board.id=11&thread.id=16666

No response so far but it's only Monday morning in the U.S...

Peter Fang said...

Hi Michael,

I didn't have high expectations for K20D's AF system as long as it's still called SAFOX VIII and uses the same sensor. They could tweak the algorithm and control code a bit but the low-light performance can't be improved that much by firmware tweaks. It is interesting to note that people that have tried the K20D said it has an option to turn off the AF double-check which in current Pentax models will cause the AF system to do minor movements upon first reaching to the point near the focus. This makes it possible to AF similar to Canon models. I don't know how well it works in reality as it's just trading accuracy for speed.

PopPhoto's apparent goof doesn't look too good for their credibility. I have posted a direct question in their forum and will see what they have to say. I did use a softer tone when asking:

"I noticed that in the "AF Speed" section of the new K20D review, the published numbers for the K20D are identical to that of the K10D. Was this a printing error or did the two different bodies really perform this equally in the AF department?"

:)

RiceHigh said...

Peter, what do you expect for their answer? :-) I bet they would simply avoid to respond forever over there under the current scenario!

Well, they should publish an amendment/supplement to the test report to correct the errors in the next issue, which is the only possible remedial action for them to look better again for the case,

Peter Fang said...

Yes they had done so in the past, publishing corrected data for a certain model (a Nikon model if I recall correctly) after the initial review was printed. I hope they do the same thing for the K20D AF speed part, and while they are at it, give us some SDM AF speed numbers.

Anonymous said...

It's hard to trust PopPhoto and to discuss review which can't be named REVIEW. We are not zombies and robots to trust to strange abstract figures from PopPhoto.
As for AF test - there was no any AF test at all. :) It's not even DPREVIEW. It's much worse and unprofessionally.
Why don't they make test with K20D with SDM lenses? Pentax's kit has no SDM. Canon and Nikon have. I understand that SDM is not miracle and not helper in low light, but it's more honestly. It will be more interested to compare Pentax AF speed with Canon/Nikon/Olympus.

Anonymous said...

Can any reviewer test AF speed of K10D 1.30 + DA*16-50 and K20 + DA*16-50? Nobody made it.
To test 18-55 without SDM with K10D and K20D is silly practice.

RiceHigh said...

As far as I know, PopPhoto did all their AF tests with 50mm/1.4 lenses on all bodies (ditto for Dpreview to use 50mm lenses for their resolution chart tests). At least this was true in the past where in quite some past issues (maybe long ago, though) stated about the lenses in use. But anyway, it is not totally sure now as reports in recent years didn't mention anything. Somehow disappointed indeed.

Anonymous said...

I know users with K10D with 16-50 -really fast and quiet AF.
FA lenses were never very quick.
Even Sigma zooms, for example, 28-105/2.8-4 worked 3 (!) times faster at ist DS then my FA50/1.4.
K10D is 2-3 times faster than DS.

RiceHigh said...

I think the K10D's AF is faster than the DS but it is no way 2 to 3 times faster.

Puzzle solver ?? said...

Think the AF differences between the 35mm and digital is due to the now reduced mirror and viewfinder space. I would guess that w/ 35mm light paths you could effectively "steal" more light and still have a bright viewfinder. W/ APS size optics you may have to "sacrifice" this light. Thus not feeding as much to the AF sensor, decreasing it's performance. Just food for thought.

Chris Tham said...

Hi - I am not sure it's valid to cut and paste from four different reviews to build your comparison tables. The cameras were tested at different times, different conditions, different lens.

The differences in AF speed for example could be related to different lens design.


A more valid comparison will be a test of all four cameras done under identical conditions, with identical lens but different mounts(third party, say Sigma).

Also, since I am an exclusive RAW shooter, I am not sure how useful the noise comparisons are, since I suspect they mostly reflect different designs decisions in the image processing engines. There is no such thing as a free lunch, presumably lower noise at high ISO is achieved at the expense of detail.

As a counter view, I actually prefer a more conservative approach to noise reduction, since that implies less filtering.

Post a Comment

Related Posts

 
Creative Commons License
RiceHigh's Pentax Blog by RiceHigh is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.