9/10 Doesn't Mean What it Used to



As the clock struck 2pm last Friday, something that we had long suspected was officially confirmed: Batman Arkham City will be in the running for game of the year honours. Two days later and it was sitting pretty with an average score of 98% on Metacritic, though this is taken from a limited sample of only eight different reviewers. It may be early days, but the consensus is that Arkham City is something special.

Visiting some of the bigger video game websites, and feeling rather good about my pre-order, I stumbled across a very detailed video review. The reviewer clearly enjoyed the game and went to great lengths to set out the reasons why they were so smitten, whilst avoiding the PR talk that will ruin any review. Their gushing praise culminated in a score of 9/10 which, despite being fractionally below the average, is a bloody good score. However, it seems that this favourable review didn’t go far enough for some commentators – a sentiment that was echoed on other sites.

There was much bellyaching among users about the lack of a perfect score for a game that seems to have fulfilled its vast potential. Statements ran along the lines of "I feel its worth a 10/10" and "What's wrong with this reviewer? IGN gave it 9.5!". Unless these cretins have somehow gotten their mitts on an advance copy, then they are passing judgement, as well as questioning a far more informed source, on a game that they couldn't possibly have played. My intolerance for stupidity aside, the sentiment was clear: 9/10 isn't perceived as being a high enough score for a game that is already being discussed as best of the year material.

For this sad state of affairs, review outlets have no one to blame but themselves. Over the years there has been a distinctive shift towards a higher average score doled out by print and online reviewers, culminating in a culture of "80% or bust". On any other scale, five would be deemed average with anything below being rather less meritorious. Sixes and sevens would be commendable, with eight and above being reserved for the cream of the crop. This is clearly not the case today, where in many places anything below an eight is seen as being of questionable worth, and there are far too many AAA titles settling nicely into the 9-9.5-10 bracket. If your job is to offer balanced opinions and to approach things from a critical point of view - they are called critics for a reason - then I'd say you are failing pretty miserably if you are awarding perfect scores. A 10/10 is most unhelpful for an interested consumer as it is unlikely to offer any constructive information. If I wanted unquestioned praise, then I'd have saved myself some time and just read a PR release.

Its not my intention to root out every single cause in this shift, as I think that's already been exhausted by far better and more informed writers than I. But just to touch on it briefly, Metacritic has elevated scores far above their station, and it has gotten to the point where sequels are commissioned or cancelled based on scores from this hugely influential site. As useful as it may be, the Metacritic system is rife with problems, including which review sources to use as well as having to convert all scores onto the same scale - for example 1UP reviews in an alphabetical, not numerical manner - and the simple fact that all reviewers will score slightly differently.

It would make sense that the rise of Metacritic would in turn lead to an increase in pressure from PR companies for reviewers to award higher scores. It is no secret that tiered, score based embargoes are placed on many new games, to control the information that the consumer receives in the vital opening week at retail. This entails reviewers being told when their reviews can go live, dependant on the score they award. For example, if you have given the game in question a 90% or above, then you may be able to post your review on Monday, whereas anything lower cannot go live for another couple of days. This ensures a favourable Metacritic score during the first few vital days, where consumers will keep or cancel their pre-orders and buy games at full retail price. For more on this worrying practice, I would recommend checking out the October edition of EDGE magazine, which featured an interesting article that touched upon publisher enforced embargoes.


Lost Planet 2 suffered a 70% Metacritic, which meant it was DOA at retail


When I post reviews here at toomanywires, I don't usually award a score. This is simply my preference as the author, as I don't feel the need to reduce 1500 words into one arbitrary number, and I like to think that the majority of people who have made the effort to visit will take the time to read most, if not all of the review. However, when I review elsewhere, I am usually required to award a score out of ten, and I must admit that I sometimes struggle with it. I have perhaps become overly reliant on awarding scores of seven and eight, which I feel accurately represents a game that is very good, but not quite great. I also feel the need to compare the game I'm playing to similar ones I have reviewed in the past, as reviewing in a vaccuum is not particularly helpful for the reader. If I gave Dragon Age 2 7/10, and then six months later awarded Shadows of the Damned an eight, despite not feeling that is a superior game, then am I doing something wrong? But then again, what business do I have comparing an action RPG to a third-person shooter.

Although I don't use them here, I do think scores are important and have a vital role in molding the public's perception of a game. I must admit, before reading a review I will always glance at the score first, and then I delve into the meat of the article in search of information to support the score given and discover how the writer settled upon it. However, when the scale is so bent out of shape, am I doing myself and the writer a disservice by giving it the time of day?

Its not all doom and gloom, as there are clearly no shortage of writers and publications that are wary of this problem. EDGE has earned its reputation as an outlet that is quite critical in its reviews, but its not uncommon to find a game scoring 6/10 with a write-up that is far more positive than not. I have also seen a number of writers on twitter bemoaning the current state of affairs. For a good dose of common sense, I'd strongly recommend following Gamespot's
Kevin VanOrd, whose riffing on this topic the last couple of days partially inspired this post.

This problem is not something that can be easily fixed, and one might argue that we shouldn't even try. After all, reviews are a hugely personal thing, peculiar to each individual reviewer, so applying any sort of rigid universal scale would be utterly pointless. However, the current system has been loosely settled upon by the majority of the gaming press, even if each will have their own slightly different interpretation of it, and as long as the general rules are understood by reviewer and reader then I'm sure we can live with it. Even still, a 9/10 doesn't mean what it used to and, as gamers, I feel its important that we keep this in mind.

Comments

  1. I dislike review scores. Mostly because I never know what they mean. Every site has a different scale and it is so inconsistent that it is annoying. On some sites it does feel like a 7 means its trash. It shouldn't and on many sites it doesn't, but sometimes it does for a certain reviewer. Also if you don't give a score, I like to think it puts more of a focus on the actual text of the review.

    But you are right, we can live with it. I look more at the actual text anyway than the score.

    As for commenters... ugh. I don't even look at comments on reviews. They are so brutal. People are quite crazy about what they expect of games they never played.

    ReplyDelete
  2. Inconsistency is a big problem, and I think its an area where Metacritic is actually quite helpful. You can search average scores by publication, so its very easy to see who is overly generous with their scoring, so you can take their reviews with a pinch of salt.

    Comments on reviews on some of the bigger sites can make your blood boil. Must be hard sometimes for reviewers to bite their tongue.

    Cheers

    ReplyDelete
  3. I admit that i usually look at scores before reading the article, but I agree entirely with what you've said here.

    The problem with scores is that they tend to lead you to think that the experience you have when playing a game can be understood on a definite scale - when in fact they can be quite arbitrary. Like you said, how do you compare an rpg to a shooter, or a beat-em-up to a interactive novel? Then you end up with a game like Deadly Premonition, which is a blast to play but technically...well Im just not sure.

    I do always end up giving time to review scores, but i wish i didn't. also im shocked by review score embargos - i had no idea this went on.

    ReplyDelete
  4. When a restaurant critic goes to review a restaurant, they don't tell anyone, they just walk in. Then they pay for their meal. Then they write a review of the restaurant. That's a review. Restaurants get bad reviews all the time.

    A video game "review" is written after the publisher in question sends out a free copy of their game in hopes of generating more publicity. The "reviewer" accepts the free game and writes about it. That's not a review. That's a link in a marketing chain. Most video games receive ratings far higher than average.

    And that sets the precedent. No one goes to school for video game writing. Everyone just learns it from watching other video game writers. So even though we pay for our games, we still think in the vocabulary of the semi-independent PR departments we call video game websites and magazines.

    When it comes to the comment wars on those websites, that's just gamers being gamers. Of course our people get carried away and project their own egos and preferences into a system that involves scoring. It IS retarded, but we do spend every day saturating ourselves in a paradigm of overly simplified contests where someone must always win.

    ReplyDelete
  5. @ Joe - Embargoes are a worrying trend, though I doubt they are peculiar to VG. Another way of PR putting pressure on reviewers, some of whom appear to be far too willing to cave.

    I'm glad I didn't have to review Deadly Premonition!

    @ Thirdrail - You may have already seen this, but check out this link to Kotaku - http://kotaku.com/5851157/ea-caught-pressuring-publications-over-battlefield-3-reviews - regarding PR pressure and selection on potential reviewers for Battlefield 3. It would appear this kind of shit happens all the time.

    As for the comment wars, some people are just cunts, gamer or not, and the internet is the perfect place to revel in that status, albeit anonymously. Commenting on bigger website reviews seems like a pretty futile exercise, and one that I feel most people will avoid like the plague.

    ReplyDelete
  6. Too true. A 7/10 is the new "average", and anything below that is garbage... That's why actually "reading" the review is a better way of gauging a games quality. Too bad I don't think most people bother to do that.

    Now about comment wars... The internet sucks! lol. I know it's not as simple as that, but rude and nasty comments have become commonplace. Anonymity makes politeness an option rather then a rule.

    ReplyDelete
  7. The scoring system "is" whacked now days. 5 out of 10 for example should be at least be middle of the road but in the game industry it's an "epic fail" whereas a "7" is barely enough to get by.

    I agree with Ceva. You got to read reviews to know what's going on due to the fact the number system is completely broken.

    ReplyDelete
  8. @ Ceva - A 7/10 by any other account would be a good score. Just not in the world of video games!

    @ Broken - I wonder how many people do read the whole review. To be honest, I'm more likely to skim read - something I got pretty good at while studying History at univ - unless its a game, or a writer that I'm really interested in. when visiting big sites like IGN or Gamespot, I'm much more likely to watch the video review than read the written article.

    ReplyDelete

Post a Comment

Popular posts from this blog

TGS 2019 - PC Engine Mini Hands-on

It's 2024 and I'm Buying a Nintendo GameCube

The Massacre at Guthrie Farm