Saturday, January 31, 2009

Problematic DPR "Reviews" (Esp. Recent Ones)

As I have always said, camera gear "review", by its name, is just a kind of unprofessional assessment on camera gear, which is more or less consisted of many personal opinions and comments made by individual writer rather than more on scientific measurements. For professional ones with proper measurements, which should strictly follow those well-established International Standards and/or Industrial Standards such as ISO, IEC EN, ANSI, ASTM and etc., for the testing procedures and then for the verifications of the obtained results, they are called "Lab Tests" or "Bench Tests", NOT "Reviews".

But then about ten years ago, there came the DPReview (DPR) website, which the site owner Phil Askey used to call his user reports on digital cameras and gear as "reviews". Those reports of him were just summaries of his personal findings (based on some very simple "measurements" or even not), comments and opinions about the gear: no more, no less. Since at that time, there was not much useful information on the Internet and ditto for many of those printed magazines, in which most of the "test" reports are almost useless, the DPR became more and more popular and "large". A good example for those limited information age of the Internet is no better to quote the Steve's Digicams which produced really crappy useless "reviews" (you'd better download an official user manual from an official site and could get more info about the cameras!) could still earn a living! Nowadays, the term "review" has now been widely accepted and has a generalised meaning for any gear test or comment report, regardless of how they were done, i.e., anyone can write a "review", from a novice who knows almost nothing versus a real expert who has been working in the industry for years and has acquired different professional qualifications in the relevant fields and areas of the industry.

So, let me go back to the topic. What makes me to write this blog article immediately is that I can see the quality, usefulness and creditability of those published camera reviews by the DPR are declining. But yet I know that the general public of the Internet *still* trust too much the DPR (well, it is just a tradition by now). The general audience, whom are huge for those who read their site, just rely too heavily on their reports. For all these reasons, I opt to write here something to share my observations and thoughts with a sole intention to draw the attention of any of those internet readers for the flaws and problems of those DPR reviews (more so recently, and also the upcoming reviews, very likely). In fact, I can see that DPR and Phil Askey have spent no more new effort in improving the quality and standard of their reviews with better contents. Instead, Phil's crew and staff who called themselves as "technical writers", are just knowing to follow strictly the old report template that has been used by their boss for long, which IMO is just rather outdated by today's standards.

Now, let's go on to see why the DPR reviews are now so problematic. I will elaborate more in details with *some* examples:-

1. Inconsistent Judgements/Yardsticks

Since Uncle Phil sold his DPR site to Amazon and became the Overlord of the site, he employed a team of crew of his own so that Phil could have more time to enjoy more of his life from his hard earned money, which should not be a small sum as Amazon is just a very large company! Well, just think if you had the money, you would want more time, so that you could spend your money! Right? :-)

But after that, all the DPR reviews were written by different person(s), sometimes one, sometimes two, sometimes three and sometimes four. Phil sometimes participated with his name appeared as one of the reviewers, but not often. But then what's the (real) problem? With different combinations of different persons as the reviewers, their viewpoints, judgements or simply prejudgements could be very different, even though it seems that there is a well-established "system, which is just more or less a document template for the review report format.

Nevertheless, on the other hand, I don't know whether Boss Phil would give any "input", actually I mean a "pointer" or a finger to his staff for what direction they should write for each of the DPR reviews, for his own "impressions" on different cameras for particular brands and models, or just his "intentions". But what I can see is that serious inconsistency in judgements and there exists many subjectivities, which overweighs much of the more important facts and thus harms their objectiveness much. When they made their marks and conclusions, I can see many biases, which are rather illogical, with very different "yardsticks" when different cameras are "reviewed" - it can be easily noticeable that they have more tolerances with "large" brands.

Say for example in their K-m review published today, they write in the "Conclusion" that the in-camera jpeg image quality is really rather bad for the K-m and a 6.5 mark out of 10 was given. But in their side-by-side comparison to the Oly E-420, for anyone who can read with a more careful mind, it actually shows there is NOT MUCH difference in the image quality (which they have also mentioned themselves) and that they even say the highlight and shadow transition and preservation are better for the K-m. But then the most ridiculous thing is that they still gave a 8.0 mark for the image quality for the E-420 in their E-420 review.

Whilst I do agree with their preliminary findings that the K-m in-camera jpeg is not on a par with the Canon 1000D (which is true from looking at their posted pictures), and also not as good as the RAW converted jpeg of K-m itself, the quality differences between the K-m against the E-420 and Sony A200 is not of that big differences (with 420's jpegs are close in resolution and A200's slightly better). So, why they did not find serious problems with the Sony and Olympus, even though the Canon is undoubtedly is better, for the evidence they showed.

Hence, they just have very different tailor-made yardsticks for doing different reviews, I must say. IF they compared DSLR to DSLR, all of them should be considered and compared on the same basis, NO MATTER they are Full Frame, APS-C or 4/3. They SHOULD NOT have much more "tolerances" just because the 420 is a 4/3 camera! Silly and stupid enough? (Or maybe the true reason is just about the "brand" behind?)

Whilst they say one of the biggest Cons of the K-m is that "Default JPEGs too contrasty and poorly sharpened" (the first Con they list, which led to the poor marginally "Recommended" rating for the K-m). But then the writers also know well and have mentioned that it is the "default" tone/colour curve as in different regions in the world (actually when the user selects a different language, a different colour/tone profile will be set by the camera). However, since this sharpening level and steep tone curve profile is just a "picture profile" which is just a matter of taste, what "Con" is it actually? How "Big" is that "Con"? What's the fuss about??

2. Incomplete Views by Deemphasizing Shortcomings (of the gear they "like"), Overemphasizing Shortcomings (for gear they "don't like")

Of course, none of we average Internet readers would know the reasons behind for why they actually "like" or "dislike" individual camera of a particular brand and model. But I can see many facts, for example:-

In their very recent Panasonic G1 review (the last review prior to the K-m one), they highly praised the camera and have given the camera very high ratings/overall rating. In fact, I myself have been very interested in the G1 too and I appreciated much the innovativeness and creativity of Panasonic too. Since I have been finding a compact and lightweight DSLR to replace my K100D, I just went to try it. But shortly after I tried it for several minutes, I told myself this camera was not the one I would buy (The set price tag is okay for me). Why? The G1 is a pseudo DSLR with an Electronic ViewFinder (EVF), but that EVF, despite it is quite strong in Spec on paper and is also expensive to make, it is just something not much useful if it is compared to any *optical* viewfinder: Its dynamic range is rather limited - I could not even see those well lit parts of the Pana showroom against those just somehow darker areas at the same time in that environment or vice versa. Pixelisation of the finder image was obvious (owing to the large magnification, which the DPR praise). Flickering of EVF image was obvious. Colour reproduction was fake. So, why STILL the DPR could give the G1 a "Pro" about the EVF of the G1 and praised loudly about its EVF?? If they are mind correct (I mean technically, not commercially) in their methodology, all "DSLRs" under their tests should be considered on the same basis - they should not just because it is the best EVF then they gave it a high mark and even emphasizing it it a "great" "Pro"(!?). The fact is that the EVF of the G1 is *still* far worse than any optical viewfinder, even the worse ones, e.g., those in the 4/3 entry level models - Physics. Owing to the serious biases of the DPR writers, their reviews could be very misleading.

And, as for the K-m review, another major reason they told the world for giving such a rarely seen and low rating (by now and in recent years) and why they are so disliked about the camera is yet unsubstantiated (putting aside the inferior in-camera jpeg (but yet they do not mention about the best retained colours and details by the K-m at higher ISO speeds), which is: The missing selected AF point indication (the illuminated red-points).

It should be noted that the K-m has only five AF points which are put very closely to the frame centre (which is still more than a few of those "Highly Recommended" current entry level DSLRs by other brands). In the following, I would talk more about that red indication from the viewpoint of a Photographer and then more from an Engineer.

First, from the user point of view, practically:-

A. For stationary objects: If the user want to have total control on the selection of focus point (and for knowing where the focus will fall on), he/she just has chosen the (Single) Central AF point already and do re-composition for each frame when taking the pictures. On the other hand, if the user, whom is usually a beginner or novice, let the camera decide and choose the AF point *automatically* for him/her, WHY the user cares about which point? Why it is needed? Will they care? Of course, if there is such an indication, it is mostly welcomed. But I don't think it is a serious flaw, never; and

B. For moving objects: Actually, under most circumstances, it is more important to know which focus point(s) the camera has chosen, since practically it is not feasible to re-compose and time usually does not allow. Indeed, I want the indication for my Canon 5D too. I did write at the DPR long time ago when I first got my 5D for asking Canon to implement it, at least as an option. But waaait, do Uncle and Boss Phil knows *even* the Canon 1 series truly professional DSLRs, namely, IDS and 1D MkIII, will have the red AF illuminations disabled in the Continuous / Servo AF mode (Yes, I am sure that they know!). So, what's the point, then?? And, why didn't DPR give the Canon flagships low marks and then "not recommending" the EOS 1D and IDS bodies because of this reason??? In fact, they have never done that, and never dared not and will never dare to do so (and of course!). All Nikon DSLRs have the indication even in Servo AF mode (as I have said, at least as an option - IIRC), so why not DPR trashed the Canons?

Now, from the technical point of view, the entry level Canons, say, the 1000D AF point indication is just a small and weakly illuminated red point which is hardly visible, especially when shooting outdoor, which just means that it is not much better. Since the 1000D has a particularly small viewfinder vision, such indication is yet not as useful as it would be, theoretically. In contrast, the K-m does have the largest viewfinder image magnification amongst all entry level DSLRs, which is undoubtedly larger than that of the 1000D and D60, so why didn't Uncle Phil's guys emphasize this so strongly?? (There is only one brief and vague statement mentioned in the whole review)

All in all, should the above two most "important" and "killing" shortcomings really worth for so many marks to be deducted. With such weak reasons and "grounds" to finally draw up a "Recommended (with Reservation)" rating, I am totally not convinced. It just looks like some simple excuses were to be found and it's time for DPR to show their "objectiveness", since they have just given too many and so many "Highly Recommended" ratings in recent years. And, as you can see the numerous responses on the Net already, conspiracy theory will come in again for the true reasons behind..

On the other hand, I wish to point out that the K-m does not have the LiveView function (technically fully explained here), why the DPR don't even mention this as a "Con", which is a more valid point IMO. Would it be just because the *Nikon* D60 also doesn't have that? In their D60 review, this is even not listed as one of the Cons, neither. Really very good "consistency! Well done, DPR! They were most brilliant and clear enough to mention that discrepancy in the words of the Conclusion, which are well hidden, together with some kinds of defense. In fact, for what I know about those new DSLR beginners, especially whom used to shoot with P&S or Prosumer DCs but no DSLR reflex experience before, LiveView is just a feature on the top of their requirement lists when they are to make their choice (whether the LiveView will be useful to them or not when they have their DSLRs is yet another problem).

P.S. I have all those entry level Canon and Nikon DSLRs at work and I myself own and use so many Pentax DSLRs and also a high-end Canon Full Frame DSLR. As such, I believe I actually know very well for all those practical differences, between different models by different makers.

3. Unqualified Incompetent Persons; Not Observing International/Industrial Standards; Infringement to the Intellectual Property of the ISO

- DPR measure resolution values by human eyes, with no consideration of the MTF % or simply ignore it completely! Note: MTF - Modulation Transfer Function, fully explained here (by Norman Koren), here (by Bob Atkins), here (by Michael Reichmann) and here (by Ken Rockwell). Well, DPR don't know about MTF??

- DPR do not test exposure accuracy based on Standards, not even they recorded and shown any histogram before they commented on the "exposure accuracy" of the DSLRs (Funny). But then you can download all their stuff, inspect the EXIF data, view the histograms and compare, if you have the time!

- DPR do not test AF timings before they made so many "Conclusions". Oh, yes, they concluded by *Opinions* and *Impression*, I almost forget.

- DPR do not test AF accuracy! No object/chart was tested/posted but they dared to comment! So, what's the difference of those "technical writers" from an Average Joe who writes at any user commenting websites like the Epinions. All these, including those made by the DPR, are just individual *user comments* afterall, which are based on nothing!

- DPR do not test system time lags of the DSLRs, which is just a key manufacturer's performance spec which is usually not published, as opposed to what Japanese CIPA and Imaging Resource measures.

- DPR (and actually Phil) even dared to steal and modify the ISO test chart (which Phil bought long ago) for measuring resolution and then marked a "copyrighted" by the DPR!? He is just so much conceited but was not even aware that it is illegal! Didn't Uncle Phil know that the ISO test chart was and is still copyrighted by the ISO and is the Intellectual Property owned by the ISO, not the DPR! (and of course not by Phil Askey personally - he was just one of the customers of the ISO publications!)

- DPR technical writers and editor (i.e., Phil Askey) have no professional qualification of any kind shown and told by them as related to the industry and field. Phil only told us that he was an IT man.

- And so on.. (but I would stop here, you *think* and *judge* yourself!)

Well, besides all the above facts and "Cons" of the DPR (Now I have almost completely "reviewed" them! Haha), there are more to note for the global Internet atmosphere and trend:-

1. Decreasing Reference Values

Last but not least, with the rise of Web 2.0 and interactive information and data sharing platforms, there are more and more useful materials shared on the Net daily such as Blogs (from different persons) and those end-users posted sample photos, which could be very valuable if the readers can select carefully and most importantly, those information could be more *honest* and "faithful". Other than the DPR, we now have much many choices at places like Blogger, Flickr, YouTube, or even Facebook and more. If you like, you can still pixel peeping on those un-retouched original images (till the end of the world!) which are not intentionally selected and picked. So, what are the true values of those primly commercial websites run by *merchants* with incompetent employees nowadays??

2. Conflicts of Interests

I think quite many people have already forgotten that DPR and so many alike sites are just running solely commercially, despite that they seem to be providing "free" (really?) services to the *end-users*? But what are their sources of incomes?? How come they could and can earn a living? And, when and under what situations they will be unable to survive and need to close down?? (And then they need to find a job after losing their *businesses*! ;-) Are they afraid?)

Go find the simple answers yourself!

Conclusion and My Humble Final Reminders

DO NOT take personal opinions solely and heavily relied just on *one* single source, no matter how authoritive the source looks or you think to be. Do more researches by reading more other articles and read more honest user opinions as well. Only by doing that, you can gather more facts, INSTEAD of subjective opinions and comments (including those by so-called self-claimed "technical writers" or "reviewers" who earn money for a living from all those associated commercial deals and Ads.

But on the other hand, do totally BEWARE of those brand-blinded fanboys and their opinions, which must be filtered out! When you see persons who always do not allow anyone to say anything "negative" on *any* shortcomings of "their" gear (they particularly are very sensitive to the "brand" they ""own"") and/or just that "their" gear are *always* "perfect, you should IGNORE THEM totally! And of course, whilst I believe that I am totally honest enough, please also don't take my words solely neither, do read my this disclaimer for why, just in case if you have not yet!

Most importantly, do try out your interested camera gear *yourself* hands-on before you act. Otherwise, you will have remorse!

With this trend of declining quality reviews for what DPR give us, I bet that the site will be going on a down slope in the long future once the internet community recognises that it is no more a reliable and valuable resource. Actually some more better sites which are carrying gear tests in a professional manner are coming up recently, like the DxOMark.com (and my report of its birth here), with a very good web interface too. So if you really intend to measur(e)bate and pixel-peep, do read better sites instead of those sub-standard ones! For old sites, I would rather recommend the "friend" site of DPR, the Imaging Resource, at least their camera reviews are based on the Imatest for some of their tests, which is using the ISO standards, e.g., for the exposure and colour accuracy tests, and they did select more sensible camera performance parameters to measure than the DPR.

The Earth is rotating, years ago, the type of DPR reviews and standard report format might be still good enough to satisfy the needs of the Internet audience and readers, owing to the limited information and little competition on the Net, as I have said in the beginning. Nowadays, in 2009, with the more advanced Web 2.0 and open platforms for all Internet users, I think the one-way information sharing is just outdated. In fact, I have seen no real improvement of the DPR website (their web design is dated - when you use different machines to view their pages and you will know what I am talking about) and more importantly their tests and the way they made their tests are dated also.

Lastly, I would remind here anyone to think twice and twice again if he/she is still really to make purchasing decision solely based on the DPR "reviews". My final humble advice: Take the DPR reviews nowadays as grains of salt. If you read, read their RAW materials presented *first*, which are readily downloadable from their site. Judge the materials they put but not the words from their mouths, never. They just give misleading and partial conclusions, owing to many many reasons.

20 Comments:

Ranger 9 said...

I agree that the quality and usefulness of reviews is declining, but it's a bit ironic that your rant shows many of the same flaws you complain about!

For example, brand bias: It's evident that part of the reason you're so angry at dpreview is that you're a Pentax fan and they're not.

Another example, shallowness: You condemned their evaluation of the Panasonic G1 based only on trying it in a store for a few minutes and deciding you didn't like the viewfinder. It's legitimate to say that you didn't like the VF, but it seems that a lot of other people do like it, so you can't say a review is no good just because they didn't agree with you on this point.

I'm not complaining about the personal bias, since you obviously wrote this as a personal statement, but it IS a bit snarky to condemn dpreview for doing the same thing you're doing!

I also think you're putting too much weight on "professional" testing standards such as DxOMark and Imatest. These can be very misleading for people who don't really understand the technical background of the test method. Imatest, for example, is based on flat charts photographed at a fixed nearby distance; that's very different from most real-world photography, which is of 3D objects at a range of distances. The test produces impressive-looking numbers which seem very scientific and precise, but they may be almost completely irrelevant to the way most of us actually take pictures!

The fact that there are more and more reviews to read on the Net doesn't necessarily make it easier to decide what equipment is good and what isn't -- the sheer volume of stuff makes it harder than ever to know who's really an expert and who's just some guy with some opinions and a blog! I don't think dpreview's are worse than any others; you just have to be aware that they've got biases and preferences like anyone else, and take those into account as you read what they write.

Anonymous said...

Well, then, there's two of us the G1 does not rate a "Pro' moniker. Pick one up and try it. And. . . DPR kisses Canon's you-know-what, imo.

Anonymous said...

Wow. That was the most gramatically incorrect rant on writing that I've ever seen.

Anonymous said...

Dude, don't like DPR, don't go there, simple as that. There are countless review sites out there. What's your hangup??

Anonymous said...

ranger 9: Ricehigh is a Pentax fan? Haha. They don't make "fans" like they used to ;)

Anonymous said...

My respects to you, Rice!

Anonymous said...

If Ricehigh is a Pentax fan, man, I'd hate to see someone who hated Pentax.

He's spot on about DPR bullshit reviews lately though. Especially the EVF being a "Pro". Just because it's the best of a poor technology, and one that the reviewers even said is useless in low light, but its still a "Pro". It should be a massive "Con".

They are completely inconsistent across reviews.

RiceHigh said...

I don't hate Pentax, I only hate camera issues, especially when they are Pentax specific.

I'd love to see Pentax debugged the issues or have improved something. And it holds true for the K-m, in which I can see Pentax have done that.

Reen J. said...

thank you for your post, RiceHigh

Anonymous said...

Gentlemen, pipe down.. we all have things to critisize about Pentax, the only question for I never read an answer, is, what cameras do Pentax's bigshots use, P&S ?
After reading many reviews (including DPR) I got myself a K100D, and this is still the best bang for the buck. My dream is to one day have a full frame from Pentax and start to enjoy my elder Pentax glass again.. I wouldn't mind sticking to 6.1 Mpixels either...
Ricehigh, you are doing a splendid job, thank you.

Anonymous said...

Reading at multiple sources might be confusing too because the variance might be very large. When I was considering Pentax K200D I read several reviews online and a few paper publications too. In a group test in one Finnish magazine the K200D was rated as almost as good as the Canon 450D and it won the editor's choice because it was cheaper. In another Finnish magazine the K200D was ranked as fourth. (If I remember correctly - I guess I could go and check the reviews tomorrow) These two reviews were written by the same guy!

RiceHigh said...

The K100D was undoubtedly one of the best Pentax DSLRs ever made (My old review here).

As for a Full Frame Pentax DSLR, I won't expect to see it in the foreseeable near future. But I did manage to use my 5D for adapting my Pentax lenses on it. An used 5D is just about $1000 right now and if you don't mind to do some (maybe) simple surgeries to your K-mount lenses, it is a viable solution by now.

RiceHigh said...

> Reading at multiple sources might be confusing too because the variance might be very large.

Yes, agreed. And that's why Dpreview had been so successful.

Anonymous said...

RH do you remember saying this?
I must say I am impressed with the details and rich colours retained by the K20D at high ISO, although it is obviously noisy from ISO 800 already. (The 450D and A350 are far less noisy, but there are no good IQ at all (as Phil Askey and his guy seem to be right in what they are presenting)).........................
http://www.pentaxforums.com/forums/pentax-dslr-discussion/29924-pop-photo-july-issue.html
For years you used dpreview to back up your own claims and now????
Some would call that hypocritical....
Maybe still sour grapes from being banned there and, funny, banned from Steve's as well...

..........................

Anonymous said...

I agree about DPR. The emperor has no clothes.

AndrewG NY said...

It is a shame that such limited but well-presented reviews from one site continue to carry so much weight, and you're right about much of this.

In defense of the Live View issue you mention, each camera is evaluated against its peers that are available at the time of its release. When the Nikon D60 was released over a year ago Live would not be an expected feature as nobody else was offering Live View back then either. The recently released K2000 should be evaluated against its competition a year later. The first bodies in a market segment that have significant new feature such as this should have the new feature considered a 'pro'. Other bodies should probably not be marked down too heavily as it is not yet an 'expected' feature in the market segment. Once a feature becomes more commonplace, only then does it make sense to heavily penalize a body that's missing it.

Sites like this never go back and change their reviews. You'll probably still see that a Nikon D100 is still highly recommended--and while it still has a lot to offer it has been eclipsed several times over by many newer bodies since it was reviewed back in 2002.

Anonymous said...

I lost any respect for the DPR lens reviewer when I noticed that he didn't understand that depth of field stays the same no matter how you crop your image.
You make some good points about the inconsistency in their camera reviews.

david said...

DPR have far too much power on web. Their Dynamic highlight headroom etc is so misleading and full of mis information .
PS you make me smile rice and for that i thank you in this recession depression.

Anonymous said...

You have my respect for not following others and try to tell it like you see it, even though you are not making a lot of friends, or a lot of powerful commercial friends with deep pockets.

RiceHigh said...

I don't need to make any commercial friends online as I think I don't need any money from them so that I can earn a living! :-)

Post a Comment

Related Posts

 
Creative Commons License
RiceHigh's Pentax Blog by RiceHigh is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.