Statistician: ‘Wine ratings show major flaws’

Prof. em. Robert Hodgson of the American Humboldt State University is a statistician but also a wine lover and wine maker. In two articles in the Journal of Wine Economics he shows major flaws in points systems for assessing wine. Hodgson is a juror for the California State Fair Wine Competition. Four years long he performed a controlled scientific study on how the judges of the contest judge wines and give points. Seventy jurors assess approximately one hundred wines over two days. Hodgson poured each juror each wine three times. The test samples were each time from a single bottle. The jurors were obviously not informed. The results are striking, although for someone with some knowledge of statistical samplings not that surprising. The jury uses the popular one hundred points system, once introduced by Robert Parker. Only one in ten jurors managed to consistently evaluate the same wine with a deviation of only two points. However, the jurors could not repeat this “consistent” performance year after year. So even the performance of the best jurors is mostly due to chance. The points of most jurors showed a much larger deviation. The same wine for example received 91 points in the first instance, then 87, then 95. For some jurors the deviations were even larger. The results of this study first appeared in the January issue of the Journal of Wine Economics. ( Full text of the article in pdf .)
In the September issue appeared another Hodgson study with a statistical analysis of the results for four thousand wines in thirteen different American wine competitions. His statistical analysis shows that a wine that wins a medal in one competition, has a large chance to win no medal in a different competition. Or, in other words, there’s a random chance of a wine winning a medal in a wine competition. Each wine has about 9% chance to win a medal in any wine competition. ( Full text of the article in pdf .)