Problematic DPR "Reviews" (Esp. Recent Ones)

As I have always said, camera gear "review", by its name, is just a kind of unprofessional assessment on camera gear, which is more or less consisted of many personal opinions and comments made by individual writer rather than more on scientific measurements. For professional ones with proper measurements, which should strictly follow those well-established International Standards and/or Industrial Standards such as ISO, IEC EN, ANSI, ASTM and etc., for the testing procedures and then for the verifications of the obtained results, they are called "Lab Tests" or "Bench Tests", NOT "Reviews".

But then about ten years ago, there came the DPReview (DPR) website, which the site owner Phil Askey used to call his user reports on digital cameras and gear as "reviews". Those reports of him were just summaries of his personal findings (based on some very simple "measurements" or even not), comments and opinions about the gear: no more, no less. Since at that time, there was not much useful information on the Internet and ditto for many of those printed magazines, in which most of the "test" reports are almost useless, the DPR became more and more popular and "large". A good example for those limited information age of the Internet is no better to quote the Steve's Digicams which produced really crappy useless "reviews" (you'd better download an official user manual from an official site and could get more info about the cameras!) could still earn a living! Nowadays, the term "review" has now been widely accepted and has a generalised meaning for any gear test or comment report, regardless of how they were done, i.e., anyone can write a "review", from a novice who knows almost nothing versus a real expert who has been working in the industry for years and has acquired different professional qualifications in the relevant fields and areas of the industry.

So, let me go back to the topic. What makes me to write this blog article immediately is that I can see the quality, usefulness and creditability of those published camera reviews by the DPR are declining. But yet I know that the general public of the Internet *still* trust too much the DPR (well, it is just a tradition by now). The general audience, whom are huge for those who read their site, just rely too heavily on their reports. For all these reasons, I opt to write here something to share my observations and thoughts with a sole intention to draw the attention of any of those internet readers for the flaws and problems of those DPR reviews (more so recently, and also the upcoming reviews, very likely). In fact, I can see that DPR and Phil Askey have spent no more new effort in improving the quality and standard of their reviews with better contents. Instead, Phil's crew and staff who called themselves as "technical writers", are just knowing to follow strictly the old report template that has been used by their boss for long, which IMO is just rather outdated by today's standards.

Now, let's go on to see why the DPR reviews are now so problematic. I will elaborate more in details with *some* examples:-

1. Inconsistent Judgements/Yardsticks

Since Uncle Phil sold his DPR site to Amazon and became the Overlord of the site, he employed a team of crew of his own so that Phil could have more time to enjoy more of his life from his hard earned money, which should not be a small sum as Amazon is just a very large company! Well, just think if you had the money, you would want more time, so that you could spend your money! Right? :-)

But after that, all the DPR reviews were written by different person(s), sometimes one, sometimes two, sometimes three and sometimes four. Phil sometimes participated with his name appeared as one of the reviewers, but not often. But then what's the (real) problem? With different combinations of different persons as the reviewers, their viewpoints, judgements or simply prejudgements could be very different, even though it seems that there is a well-established "system, which is just more or less a document template for the review report format.

Nevertheless, on the other hand, I don't know whether Boss Phil would give any "input", actually I mean a "pointer" or a finger to his staff for what direction they should write for each of the DPR reviews, for his own "impressions" on different cameras for particular brands and models, or just his "intentions". But what I can see is that serious inconsistency in judgements and there exists many subjectivities, which overweighs much of the more important facts and thus harms their objectiveness much. When they made their marks and conclusions, I can see many biases, which are rather illogical, with very different "yardsticks" when different cameras are "reviewed" - it can be easily noticeable that they have more tolerances with "large" brands.

Say for example in their K-m review published today, they write in the "Conclusion" that the in-camera jpeg image quality is really rather bad for the K-m and a 6.5 mark out of 10 was given. But in their side-by-side comparison to the Oly E-420, for anyone who can read with a more careful mind, it actually shows there is NOT MUCH difference in the image quality (which they have also mentioned themselves) and that they even say the highlight and shadow transition and preservation are better for the K-m. But then the most ridiculous thing is that they still gave a 8.0 mark for the image quality for the E-420 in their E-420 review.

Whilst I do agree with their preliminary findings that the K-m in-camera jpeg is not on a par with the Canon 1000D (which is true from looking at their posted pictures), and also not as good as the RAW converted jpeg of K-m itself, the quality differences between the K-m against the E-420 and Sony A200 is not of that big differences (with 420's jpegs are close in resolution and A200's slightly better). So, why they did not find serious problems with the Sony and Olympus, even though the Canon is undoubtedly is better, for the evidence they showed.

Hence, they just have very different tailor-made yardsticks for doing different reviews, I must say. IF they compared DSLR to DSLR, all of them should be considered and compared on the same basis, NO MATTER they are Full Frame, APS-C or 4/3. They SHOULD NOT have much more "tolerances" just because the 420 is a 4/3 camera! Silly and stupid enough? (Or maybe the true reason is just about the "brand" behind?)

Whilst they say one of the biggest Cons of the K-m is that "Default JPEGs too contrasty and poorly sharpened" (the first Con they list, which led to the poor marginally "Recommended" rating for the K-m). But then the writers also know well and have mentioned that it is the "default" tone/colour curve as in different regions in the world (actually when the user selects a different language, a different colour/tone profile will be set by the camera). However, since this sharpening level and steep tone curve profile is just a "picture profile" which is just a matter of taste, what "Con" is it actually? How "Big" is that "Con"? What's the fuss about??

2. Incomplete Views by Deemphasizing Shortcomings (of the gear they "like"), Overemphasizing Shortcomings (for gear they "don't like")

Of course, none of we average Internet readers would know the reasons behind for why they actually "like" or "dislike" individual camera of a particular brand and model. But I can see many facts, for example:-

In their very recent Panasonic G1 review (the last review prior to the K-m one), they highly praised the camera and have given the camera very high ratings/overall rating. In fact, I myself have been very interested in the G1 too and I appreciated much the innovativeness and creativity of Panasonic too. Since I have been finding a compact and lightweight DSLR to replace my K100D, I just went to try it. But shortly after I tried it for several minutes, I told myself this camera was not the one I would buy (The set price tag is okay for me). Why? The G1 is a pseudo DSLR with an Electronic ViewFinder (EVF), but that EVF, despite it is quite strong in Spec on paper and is also expensive to make, it is just something not much useful if it is compared to any *optical* viewfinder: Its dynamic range is rather limited - I could not even see those well lit parts of the Pana showroom against those just somehow darker areas at the same time in that environment or vice versa. Pixelisation of the finder image was obvious (owing to the large magnification, which the DPR praise). Flickering of EVF image was obvious. Colour reproduction was fake. So, why STILL the DPR could give the G1 a "Pro" about the EVF of the G1 and praised loudly about its EVF?? If they are mind correct (I mean technically, not commercially) in their methodology, all "DSLRs" under their tests should be considered on the same basis - they should not just because it is the best EVF then they gave it a high mark and even emphasizing it it a "great" "Pro"(!?). The fact is that the EVF of the G1 is *still* far worse than any optical viewfinder, even the worse ones, e.g., those in the 4/3 entry level models - Physics. Owing to the serious biases of the DPR writers, their reviews could be very misleading.

And, as for the K-m review, another major reason they told the world for giving such a rarely seen and low rating (by now and in recent years) and why they are so disliked about the camera is yet unsubstantiated (putting aside the inferior in-camera jpeg (but yet they do not mention about the best retained colours and details by the K-m at higher ISO speeds), which is: The missing selected AF point indication (the illuminated red-points).

It should be noted that the K-m has only five AF points which are put very closely to the frame centre (which is still more than a few of those "Highly Recommended" current entry level DSLRs by other brands). In the following, I would talk more about that red indication from the viewpoint of a Photographer and then more from an Engineer.

First, from the user point of view, practically:-

A. For stationary objects: If the user want to have total control on the selection of focus point (and for knowing where the focus will fall on), he/she just has chosen the (Single) Central AF point already and do re-composition for each frame when taking the pictures. On the other hand, if the user, whom is usually a beginner or novice, let the camera decide and choose the AF point *automatically* for him/her, WHY the user cares about which point? Why it is needed? Will they care? Of course, if there is such an indication, it is mostly welcomed. But I don't think it is a serious flaw, never; and

B. For moving objects: Actually, under most circumstances, it is more important to know which focus point(s) the camera has chosen, since practically it is not feasible to re-compose and time usually does not allow. Indeed, I want the indication for my Canon 5D too. I did write at the DPR long time ago when I first got my 5D for asking Canon to implement it, at least as an option. But waaait, do Uncle and Boss Phil knows *even* the Canon 1 series truly professional DSLRs, namely, IDS and 1D MkIII, will have the red AF illuminations disabled in the Continuous / Servo AF mode (Yes, I am sure that they know!). So, what's the point, then?? And, why didn't DPR give the Canon flagships low marks and then "not recommending" the EOS 1D and IDS bodies because of this reason??? In fact, they have never done that, and never dared not and will never dare to do so (and of course!). All Nikon DSLRs have the indication even in Servo AF mode (as I have said, at least as an option - IIRC), so why not DPR trashed the Canons?

Now, from the technical point of view, the entry level Canons, say, the 1000D AF point indication is just a small and weakly illuminated red point which is hardly visible, especially when shooting outdoor, which just means that it is not much better. Since the 1000D has a particularly small viewfinder vision, such indication is yet not as useful as it would be, theoretically. In contrast, the K-m does have the largest viewfinder image magnification amongst all entry level DSLRs, which is undoubtedly larger than that of the 1000D and D60, so why didn't Uncle Phil's guys emphasize this so strongly?? (There is only one brief and vague statement mentioned in the whole review)

All in all, should the above two most "important" and "killing" shortcomings really worth for so many marks to be deducted. With such weak reasons and "grounds" to finally draw up a "Recommended (with Reservation)" rating, I am totally not convinced. It just looks like some simple excuses were to be found and it's time for DPR to show their "objectiveness", since they have just given too many and so many "Highly Recommended" ratings in recent years. And, as you can see the numerous responses on the Net already, conspiracy theory will come in again for the true reasons behind..

On the other hand, I wish to point out that the K-m does not have the LiveView function (technically fully explained here), why the DPR don't even mention this as a "Con", which is a more valid point IMO. Would it be just because the *Nikon* D60 also doesn't have that? In their D60 review, this is even not listed as one of the Cons, neither. Really very good "consistency! Well done, DPR! They were most brilliant and clear enough to mention that discrepancy in the words of the Conclusion, which are well hidden, together with some kinds of defense. In fact, for what I know about those new DSLR beginners, especially whom used to shoot with P&S or Prosumer DCs but no DSLR reflex experience before, LiveView is just a feature on the top of their requirement lists when they are to make their choice (whether the LiveView will be useful to them or not when they have their DSLRs is yet another problem).

P.S. I have all those entry level Canon and Nikon DSLRs at work and I myself own and use so many Pentax DSLRs and also a high-end Canon Full Frame DSLR. As such, I believe I actually know very well for all those practical differences, between different models by different makers.

3. Unqualified Incompetent Persons; Not Observing International/Industrial Standards; Infringement to the Intellectual Property of the ISO

- DPR measure resolution values by human eyes, with no consideration of the MTF % or simply ignore it completely! Note: MTF - Modulation Transfer Function, fully explained here (by Norman Koren), here (by Bob Atkins), here (by Michael Reichmann) and here (by Ken Rockwell). Well, DPR don't know about MTF??

- DPR do not test exposure accuracy based on Standards, not even they recorded and shown any histogram before they commented on the "exposure accuracy" of the DSLRs (Funny). But then you can download all their stuff, inspect the EXIF data, view the histograms and compare, if you have the time!

- DPR do not test AF timings before they made so many "Conclusions". Oh, yes, they concluded by *Opinions* and *Impression*, I almost forget.

- DPR do not test AF accuracy! No object/chart was tested/posted but they dared to comment! So, what's the difference of those "technical writers" from an Average Joe who writes at any user commenting websites like the Epinions. All these, including those made by the DPR, are just individual *user comments* afterall, which are based on nothing!

- DPR do not test system time lags of the DSLRs, which is just a key manufacturer's performance spec which is usually not published, as opposed to what Japanese CIPA and Imaging Resource measures.

- DPR (and actually Phil) even dared to steal and modify the ISO test chart (which Phil bought long ago) for measuring resolution and then marked a "copyrighted" by the DPR!? He is just so much conceited but was not even aware that it is illegal! Didn't Uncle Phil know that the ISO test chart was and is still copyrighted by the ISO and is the Intellectual Property owned by the ISO, not the DPR! (and of course not by Phil Askey personally - he was just one of the customers of the ISO publications!)

- DPR technical writers and editor (i.e., Phil Askey) have no professional qualification of any kind shown and told by them as related to the industry and field. Phil only told us that he was an IT man.

- And so on.. (but I would stop here, you *think* and *judge* yourself!)

Well, besides all the above facts and "Cons" of the DPR (Now I have almost completely "reviewed" them! Haha), there are more to note for the global Internet atmosphere and trend:-

1. Decreasing Reference Values

Last but not least, with the rise of Web 2.0 and interactive information and data sharing platforms, there are more and more useful materials shared on the Net daily such as Blogs (from different persons) and those end-users posted sample photos, which could be very valuable if the readers can select carefully and most importantly, those information could be more *honest* and "faithful". Other than the DPR, we now have much many choices at places like Blogger, Flickr, YouTube, or even Facebook and more. If you like, you can still pixel peeping on those un-retouched original images (till the end of the world!) which are not intentionally selected and picked. So, what are the true values of those primly commercial websites run by *merchants* with incompetent employees nowadays??

2. Conflicts of Interests

I think quite many people have already forgotten that DPR and so many alike sites are just running solely commercially, despite that they seem to be providing "free" (really?) services to the *end-users*? But what are their sources of incomes?? How come they could and can earn a living? And, when and under what situations they will be unable to survive and need to close down?? (And then they need to find a job after losing their *businesses*! ;-) Are they afraid?)

Go find the simple answers yourself!

Conclusion and My Humble Final Reminders

DO NOT take personal opinions solely and heavily relied just on *one* single source, no matter how authoritive the source looks or you think to be. Do more researches by reading more other articles and read more honest user opinions as well. Only by doing that, you can gather more facts, INSTEAD of subjective opinions and comments (including those by so-called self-claimed "technical writers" or "reviewers" who earn money for a living from all those associated commercial deals and Ads.

But on the other hand, do totally BEWARE of those brand-blinded fanboys and their opinions, which must be filtered out! When you see persons who always do not allow anyone to say anything "negative" on *any* shortcomings of "their" gear (they particularly are very sensitive to the "brand" they ""own"") and/or just that "their" gear are *always* "perfect, you should IGNORE THEM totally! And of course, whilst I believe that I am totally honest enough, please also don't take my words solely neither, do read my this disclaimer for why, just in case if you have not yet!

Most importantly, do try out your interested camera gear *yourself* hands-on before you act. Otherwise, you will have remorse!

With this trend of declining quality reviews for what DPR give us, I bet that the site will be going on a down slope in the long future once the internet community recognises that it is no more a reliable and valuable resource. Actually some more better sites which are carrying gear tests in a professional manner are coming up recently, like the DxOMark.com (and my report of its birth here), with a very good web interface too. So if you really intend to measur(e)bate and pixel-peep, do read better sites instead of those sub-standard ones! For old sites, I would rather recommend the "friend" site of DPR, the Imaging Resource, at least their camera reviews are based on the Imatest for some of their tests, which is using the ISO standards, e.g., for the exposure and colour accuracy tests, and they did select more sensible camera performance parameters to measure than the DPR.

The Earth is rotating, years ago, the type of DPR reviews and standard report format might be still good enough to satisfy the needs of the Internet audience and readers, owing to the limited information and little competition on the Net, as I have said in the beginning. Nowadays, in 2009, with the more advanced Web 2.0 and open platforms for all Internet users, I think the one-way information sharing is just outdated. In fact, I have seen no real improvement of the DPR website (their web design is dated - when you use different machines to view their pages and you will know what I am talking about) and more importantly their tests and the way they made their tests are dated also.

Lastly, I would remind here anyone to think twice and twice again if he/she is still really to make purchasing decision solely based on the DPR "reviews". My final humble advice: Take the DPR reviews nowadays as grains of salt. If you read, read their RAW materials presented *first*, which are readily downloadable from their site. Judge the materials they put but not the words from their mouths, never. They just give misleading and partial conclusions, owing to many many reasons.