The summary: game reviewers have to play games at a frenetic, no-side-alleys pace, to meet practical deadlines (in the ballpark of release dates) while covering the flood of games in the pre-Christmas rush. Games are not always designed to be played this way, and this isn't always the way players play them. (jvm once queried: isn't anyone who'd review a "casual game" necessarily the wrong person for the job?)
I think the problem is unique to the medium. Movies, food, and wine don't require the same time commitment to review, and books don't change character particularly much when you read to a deadline (although I do enjoy them less that way.) Perhaps the closest thing is reviewing certain conventional games, but the world of "real-world" game reviews is tiny and undeveloped.
For me, this highlights the "real" purpose of reviews (to push new product on consumers) and contrasts with my actual use of the medium. Movie reviews have to be timely because people want to invest in the social experience of going out to see a good movie, and they need guidance as to which ones are "good" while they're still in theaters. But even mediocre videogames are generally available for years, and (except for games that depend on an online community to play with) the social experience will be the same: either I play alone, or with friends I invite over.
I'm disregarding some significant factors justifying the time pressure on reviews: a whole lot of games _do_ depend on an online community, and pre-Christmas reviews are important for nongamers buying gifts for us. Nevertheless, the criticism remains: the average review is trying to assess how fun a game is, by playing it in the way that sucks the most fun out of it. And I should heed this recommendation why, again?
Also, some of this is why I have tried from time to time to record my progress on my game journal blog. (I'll hopefully be doing that with Uncharted: Drake's Fortune next week.) You can see me get through each part of the game, with immediate comments about how I felt. Then the review here on CG is meant to be a reflection of those posts.
I think only two worked out completely so far: Tomb Raider Anniversary and Shadow of the Colossus.
Previously, I did do a sequence of posts from beginning to end right here on CG, starting with first impressions (fimps) and finishing with either a review (finished the game) or final thoughts (didn't finish, and explain why).
While we're talking about reviews, I try to catch David Edelstein on Fresh Air each Friday to hear a movie review. When someone starts reviewing games like he reviews movies (however that might work), we'll be getting somewhere. Between him and The New Yorker cinema reviewers, it's difficult for me to read the stuff in the newspaper anymore.
Late night ramble...off...
I think the game reviews are increasingly unreliable for the reasons given plus others.
One, from a practical point of view regarding the play experience it's almost impossible to review MMOs and large games like Oblivion so their is clearly a large extrapolation going on.
Two, more and more I see the application of expectations rather than what's in the product. For my money, for example, you can't dock Ratchet & Clank for having NO multiplayer... you review what's there, a single player only game. Not every game should have or needs multiplayer and I'm particularly losing patience with this comment appearing more and more.
Lastly, there is clearly what I call the 'blockbuster effect' which sees certain games get inflated scores based on status and community expectations - Halo 3 and Bioshock I'm looking at you, two great games but not that perfect by any means. This is happening more and more.
the average review is trying to assess how fun a game is, by playing it in the way that sucks the most fun out of it
That *is* the job of the reviewer. I doubt Matt's Edelstein is, no, make that I know he *isn't* watching movies the same way that the conventional viewers are. It's not as likely to be relaxing, and certainly isn't a break from the proverbial grind.
Bart Farkas told me once, while I writing a guide to playing Pokemon Stadium, that there comes a point where you simply can't play twitch games with the dedication needed to create an exhaustive guide. He hadn't quite gotten there yet, but playing the games had gone from fun to grind at times, iirc.
books don't change character particularly much when you read to a deadline
This is a particularly important point/distinction, imo. They most certainly do change character if your goals aren't simply to enjoy the book -- to consume it. How could a 19th century reviewer understand the importance and intricacies of The Con Man (Meville; admittedly I don't know how it was received)?
Reviewers do have a manageable job. Tell us if we'll enjoy consuming the game enough to be worth our cash. That's something you can do, if you're a pro, by playing like mad before Xmas. How many people play games to completion? How many WoW folk have finished end game content? Even the designers admit end gamers are 1%ers!
That reviewing games at Xmas is a manageable task only makes the incestuous relationship between gaming publishers and reviews that much more vile.
Enjoyed the post. Get that Bob guy to post more.
I think Gabe is right, and I think he's right because we all know this time of year is a waterfall of games. I would not be shocked if half the reviews you see coming out are done on games the reviewer did not even complete.
The funny thing is having been on 1UP (Ziff) forusms for years the question of how many hours and how much the reviewer achieved has come up again and again. The editors at Ziff say they will never tell how many hours of actual gameplay go into their reviews and have said such a metric would be meaningless. To me that sounds like they know the answer will look bad and they know the metric is meaningful.
I think the time for transparency has come. Game reviews should occur only on games played to the end, or if not finished they should say, "This game was ass., so we only played 60%, therefore we won't give a score just an impression." But I think that means we will see more reviews cooing out in magazines a month after the game hits the shelf, but that is the price we should pay for actual reviews not glorified previews.
Kinda explains how a monoaural Wii version of Guitar Hero III snuck past every professional review on the planet too.
I've always thought reviews should be this:
Final review is "how much this game is worth" compared to the MSRP.
Then a list of modifiers ("$-5 if you really want multiplayer. $+10 if you're a fan of the series and want to find out where the story is going. ")
Something like Halo 3 would be
This game is worth: $70. MSRP is $50. +10 if you love multiplayer. +10 If you want to continue the storyline. -3 if you don't care about mp. -40 if you have FPS games. -10 if you're tired of Halo. Etc. etc. etc.