Scoring Whisky: Simple or Quantum Mechanics?

Scoring whisky is an interesting topic to me and it seems to be a bit of a controversial one amongst the community. In reality, most topics/hobbies that introduce a scoring system to rate one product against another have similar and maybe more extreme issues. In the video game industry, for instance, there are strong fandoms that refuse to accept that the latest installment in their favorite franchise could be anything other than a perfect 10/10 or 100/100. A reviewer that does not agree can be bombarded with social media vitriol and have their YouTube comments filled with angry trolls, fanboys, and maybe even the game developers themselves. Context is rarely taken into account, as if you were to read the article or watch their video in detail you would clearly see that there are objective flaws with the game or that the game systems, genre or story motif do not strike a chord with the reviewer and thus a 7/10 or 82/100 is a valid score from their perspective. This is of use to their community if you assume that they have built their community around a common love of similar games.

Equally, crowdsourced reviews via sites like www.metacritic.com or game distribution platforms like Steam can be used to punish developers and publishers for business related decisions (releasing a game before it is done, greedy monetization schemes) or poor corporate behavior (ill-treatment of employees, etc.). Large groups of people, some not even interested in ever playing the specific game, will “review bomb” these sites by giving the game a 0/10 and drag the overall score down, inevitably slowing sales and causing the media to pay attention and, at times, stock prices to fluctuate. I am not here to support or criticize these tactics, only to point out that subjective scoring is a volatile and often politicized tool that people can take advantage of to either get a message across that may very well need to be heard, or to advance an agenda or an opinion that may or may not be relevant to the actual subject of the review.

I have not seen too much of this in my time lurking in the whisky-sphere although there will always be folks who disagree with certain scores and they may have valid points, assuming they give a constructive reply as to why they don’t feel the review is fair. The primary crowdsourced review site, www.whiskybase.com, is generally filled with subjective reviews that are useful to fellow readers to varying degrees. There are certainly biases towards styles, regions, distilleries, presentation, pricing and many more. However, these biases are usually framed in context and are understandable. We all have our favorites afterall, and I am no different.

I am biased. It’s impossible not to be. 

There are going to be whiskies that I want to like, those I am curious to see if I like, and those that I expect I will not. I cannot think of any that I do not want to like, but I am not going to recuse myself from that level of bias just yet. I am still fairly new to this after all. I suppose if I sat down to review a bottle of Johnny Walker Red and it turned out to be the one true nectar of the gods, manna incarnate, I would hesitate. Maybe that means I do not want to like Johnny Walker Red because giving it a perfect score would delegitimize me in the eyes of anyone who reads my blog. I would be stuck between being honest and just writing a review that nobody would think twice about and move on while enjoying the spoils of my new found bounty, unless of course I outlined what made me love it so much and the reader took the time to try and understand. Maybe I’d even change a mind or two…

Anyways, there seem to be two schools of thought for scoring whisky. The 1 to 100 scoring system and the 1-10 scoring system. There are going to be different ways to interpret these rating systems, but there does seem to be some correlation as to how they are used across outlets. Here is how I generally interpret these systems, and I do mean a completely personal interpretation:

1 to 100

• “Good” whisky seems to be scored at an 80 or above. 
• Below 80 is a definite “Avoid”
• 90+ is a treat and something you should absolutely seek out.
• Nothing but undrinkable swill will score below 70.
• 100 is unobtainable
• More granularity is allowed. Reviewers openly struggle with bumping up a score from 88 to 89, 91 to 92, etc..
• Recency bias and nostalgia come into play within the granularity in direct comparisons to recent reviews or a long gone beloved bottle.
• This is meant to compare bottle to bottle, at least from a reader perspective. If bumping a score up one point is a major increase in quality of experience, which many reviewers seem to emphasize, then it must be worth buying that over something one point less.
• There seems to be an exponential increase in quality for points above 90. A bottle that is an 80 is still in the general ballpark of quality as something that gets an 85. However, a bottle that scores a 95 is a massive increase in quality over a bottle that scores a 90. In other words, every point closer to 100 is worth more than the last. Additionally, it feels like going from an 89 to a 90 is more difficult than an 88 to an 89, again creating an inequality between points.

1 to 10

• This results in more of a ‘categorization’ of whisky which puts bottlings into groups of relative quality.
• Limits bias towards style/region/etc.  Broader categories of scores catch and eliminate or filter some degree of bias if you are not agonizing over giving nuanced scores.
• Scoring a 5 is a truly average whisky of acceptable quality.
• A perfect 10 is rare but possible because there is no risk of one day having something “better.”
• A score of 1 is probably less likely than a 10 in the modern era, if not impossible.
• There is still agonizing over bumping up points, but it makes more sense to waver as increases are more meaningful to recommendations of buying that whisky from the reviewer.
• Reviews may be somewhat less impactful in purchasing decisions. It’s possible a reader already knows some options are good, but wants to know what is better and if they all fall in the same ‘category’ they may have to move on to other reviews for more granular scoring. Contect in the written review should be enough to decide, but not everyone reads the reviews.

I have chosen to use the 10 point system. Trying to find the level of granularity to provide accurate X/100 reviews feels too difficult for me at this point in my journey. My reference points are simply too small, even if I have had 100s of different whiskies of various styles at this point. I also feel that the categorization of whisky into X/10 scores more accurately represents what whisky is. It changes through the bottle, from night to night, from glass style to glass style, etc.. It is a more accurate representation of how I feel about a particular bottle. I feel it also allows me to more confidently place samples that I may review in relative position to whiskies of similar quality without drinking an entire bottle. 

Pricing

The next big topic is whether or not the cost of the whisky should factor into the score. This seems to be a dividing line in the review world. In all honesty, I was absolutely prepared to include pricing into my scoring, however once I decided to use the 10 point scale, I started thinking about it and I could only think of reasons to not include it, provided that I incorporate the general price I paid and a comment in the review.

Here are the reasons I am choosing to not include pricing into my scores, but rather to include value in the context of my reviews. Generally, this is specific to using the 10 point scale as the granularity of the 100 point scale makes some of these points irrelevant.

• It seems like you inevitably set limits for pricing on whisky. No 8/10 can ever cost more than $200 for example. It just feels wrong to draw a line in the sand. If you don’t do this then you are still using the 100 point scale in a sense. “It was a poor value but it was a high 7 so I won’t dock it a point.”
• Not being worth it is relative. $20 too much, $50 too much, $100 too much? How much do you punish a whisky, especially when a price increase goes into effect? Punishing something relative to its previous price seems natural, but so is inflation. How do you factor in the general trend of increasing prices across an entire industry? Prices go up, and yes, they are going up considerably faster than normal in 2022. I can see rewarding a distillery for not raising prices, but especially in the US the distillery does not have control over the network of distributors who can charge what they want and reward certain stores based on volume, etc.. You are throwing a dart at a moving board.
• This feels like an inevitable sliding scale/fibonacci sequence type scenario. A $50 to $70 increase feels different than a $200 to $220 increase. Do you punish/reward the same? Probably not, but in the wider view of the whisky landscape it seems like this can get muddled quickly. Additionally, previous reviews can become irrelevant even faster than they would otherwise.
• Consider that Whisky A is $50 and Whisky B is $200, both of similar styles. Whisky B is clearly much better than A, but A is still high quality. Whisky B is generally not $150 (4x) better in the eyes of the reviewer. Whisky A is a 6 for Quality, but is a great value so it’s given a 7. Whisky B is an 8, but is not a value (it crossed the price threshold), so it’s given a 7 or maybe lower. The impression is that these whiskies are equal in some way, but they are not. Whisky B is being punished for A’s reward as they are tacitly being compared and this results in something resembling a zero sum game.
• It is known that there are diminishing returns for pricing in whisky as with most things in life. It comes down to flavor chase over time, finding something unique, prestige, rarity, etc.. A $1000 bottle is not 10 times “better” than a $100 bottle. But that’s not why people buy $1000 whisky. Vintage whisky is a good example. Is a 1972 Brora or an old Macallan worth what they go for in auctions based on flavor? Probably not. Are you giving them a 4/10 even though they may be considered some of the best whisky to ever exist (yes, reviews on these whiskies are exceedingly rare, so this may be a bad example)? I tried the 37 year Brora release and it almost made me cry. It is on the shelf in the US for $2300 I believe. Frankly, it’s worth it. I just can’t justify buying a bottle. I can’t punish it for not being a “value” even though it’s probably the best thing I have ever tasted. 
• Value is as subjective as taste, if not more so. Most of us have biologically similar taste buds excluding the super taster or specific sensitivities which are easily identifiable (sulfur) and communicable to a reader, but our bank accounts are very different. Experiences are more disparate when it comes to taste based on culture, means, etc.. Assigning a score based on taste is difficult enough and highly subjective. Adding pricing into the score equation makes the score feel like something that is approaching random based on a large number of variables that are not transparent to the reader. Reviewers are not posting salaries and stock portfolios alongside their reviews so their definition of value is somewhat of a question mark to a random reader. “I can’t afford it” is not the same as “It’s not good value” (see my comment about the Brora). Readers ultimately have to trust that the reviewer is able to see value from their viewpoint and there is no direct evidence that is ever going to be true.
• Lowering the score of a whisky from an 8 to a 7 based on value would be the same as lowering it 10 points in the 1 to 100 scale, which does not happen in my experience. Using value to bump up or lower a score in the 100 point scale is easier and makes more sense as you have more control over the dial. Dropping a whisky 10 points in that scale would, again, equate it to whiskies that are clearly inferior in quality which does not seem right and therefore shouldn’t happen in a 10 point scale.
• Why is simply posting the MSRP, RRP, or current auction prices in the review and allowing the reader to decide if they want to spend that much money on a X/10 whisky with a given set of tasting notes not sufficient? Who am I to decide if something is worth it for YOU? I can say it isn’t worth the price to ME in my review which is my perspective taking everything above into account. You can take that for what it is based on your situation.

I will provide additional scoring metrics in another, more succinct post, but for now I feel that the best option for me is to use a 10 point scale that does not factor in price. I will do some research to see what the going rates are around the world and give my two cents in the review in regards to overall value. I am open to dissent however, so please feel free to leave any feedback in the comments or on social media. I would be happy to engage further in this discussion as I feel it is important to get this right in the long term.