An HHHC entrant asked the following question:
I was looking over the results of your past homebrew competitions posted on the website and noticed on several categories that the 1st, 2nd, and 3rd places are not necessarily scored by highest points awarded. I have judged at a couple competitions and entered in several, and never seen this before. Could you explain this scoring system to me? (In terms that even an idiot could understand?)
John Sullivan Replied:
Thanks for expressing an interest because there have been misunderstandings about the purpose of the HHHC scoring in the past. Scoring in the St. Louis Brews HHHC is virtually the same as other large competitions, including the AHA Regionals, Nationals and MCAB. First let me lay out some basic parameters.
Scoring creep is an issue for which there is no known solution. Judges tend to score either too high or too low at the beginning and ends of each round. As each successive beer is brought forward, scoring becomes somewhat normalized somewhere near the middle of the flight and then it will normally start to skew a bit near the end of the flight. To ensure that judges are evaluating the beers in a somewhat consistent manner between each other, we have a rule that the final scores of all judges on a particular beer be within 7 points of each other. While this is a good rule, it does not eliminate the scope creep phenomenon because the judges tend to "creep" together. However, for good and great beers that have a chance to win (no matter what the assigned score), the judges maintain a small tally sheet with the entry numbers of those beers. At the end of the flight, the judges confer and retaste if necessary those good beers that have a chance to win. In some cases, the judges will request the second bottle of the three bottles entered so that fresh samples may be evaluated side by side. They select the best beers and sometimes the order of finish is not reflected in the scoring. Again, remember that we DO NOT go back to rescore the flight to reflect the winner.
Split flights also present a problem with reconciling point totals. A split flight is conducted when there are too many entries in a particular category for one set of judges to evaluate. For example, if there are 26 category 6 beers (Pale Ale), it is unreasonable to expect one set of judges to evaluate all of those beers. So, those 26 beers are split into two flights of 13 each and two sets of judges are assigned. Each set of judges will determine first, second and third in their particular flight. When that judging is finished, the top judges from both flights will judge the top three beers from each flight (6 beers total). When this is done, score sheets ARE NOT filled out again. It is conducted as a Mini Best of Show. The top six beers are sampled side by side, merits discussed openly and the top three are identified out of the six finalists without filling out score sheets again.
In both the scoring creep and split flight scenarios, there is another phenomenon occurring and that is palate fatigue. The more beers that you taste, especially those that are extremely hoppy and/or alcoholic, the more that your palate becomes numbed a bit. This will affect raw scoring since the first beers in the flight are tasted on fresher palates than the last beers in the flight. However, even with palate fatigue, judges can still identify the good beers in the flight. That is why that last step, the reconciliation step where retasting and conferring occurs, under both scenarios is so important. When the great beers that have a chance to win are tasted side by side under the same palate conditions, the best beer can be and is selected.
I hope this sheds some light on why top score does not always win. We do believe however that the best beer wins and this is the most important factor. Again remember that this is not unique to our competition, but happens in virtually all large competitions. In fact, unless organizers do rescore beers, I would be skeptical of a competition where the top scores always win. In that scenario, I would suggest that the good beers and in many cases the winning beer is getting lost due to scoring creep, split flights and palate fatigue. In other words, I would have concerns that the best beers in the flight are not winning. If you have any further concerns about our judging and scoring methods, please let me know.