Hi, I've been reading bit-tech for a while now and I absolutely adore this site. I'm not a regular poster to the forums, though I do lurk. I just wanted to post and maybe incite a little discussion about ratings systems. Personally, I'm a little bored with the ratings systems in general, for a number of reasons that apply to not just this site, but to the standard de facto for all hardware/software review sites. My gripe is with arbitrary ratings values (the "10 point scale") that I perceive meaningless. I think the arbitrary/percentile system is kind of weak. It's a little flawed in the sense that it groups everything into one particular generalization which is usually multifaceted. For example, two games can get a graphics rating of "8" though one might be technically inferior, because the graphics accomplish a certain effect. In other words, two aspects of graphics (technical performance and sentimental value) are rated opposite, yet equally. Worse yet, and perhaps as a direct result, most ratings fall between 7 and 8, because the extremas aren't right. Nobody will outright say "this sucks" or "this is the greatest thing ever made." And even if they do (case in point: the original half life review in PC Gamer mag), it's not going to be long before something else even better comes out only to achieve a lower rating. That line of reasoning turns the 10 point scale into something of a 5 point scale (6-10) in all but the most extreme circumstances, and most of those extremes are usually not worth the time to publish articles about. I think saying that "yeah, I'd use this" or "I highly recommend this" has much more meaning than a single "8" value. I always take more heart with the "bit-tech recommended," or "bit-tech excellence," etc notes than with the arbitrary "8/10". My suggestion for change would be to perhaps have several people contribute to review pieces and then just offer opinions on whether they liked it or would recommend it to other folks -- Siskel and Ebert style. I'd much rather see a short list of office recommendations (or perhaps a semi-detailed debate amongst several office staffers) rather than a confusing and arbitrary set of values that are effectively meaningless. If for some reason a scale is absolutely necessary to the existence of these review sites, then I'd recommend the 5 point likert system (1 - least, 5 - most, "standard" average of 3.4) instead of the 10 point system.
Scores of 7 or 8 are a bit common. Thankfully, unlike other review communities, bit-tech doesn't overuse (as much) the 7 and 8 ratings for lack of a well-thought conclusion. To me, if the product doesn't do enough to improve on prior models, then it scores a 6 at most, while products that are just run-of-the-mill should be 5. Being awarded anything around a 9 or 10 should suggest one astronomical product, honestly. The common mistake is connecting the grading system from schools (where 50 is considered zero, illustratively) with your rating system. Zero should be zero. 10 is invincibly flawless and indomitable. 5 is middle-class.
Bit needs a better than, worse than, as good as, 'such game', that would clear up which is better when over all they have the same score, which is an average of other scores such as graphics etc..
I would have thought by the time you've read the whole review you would have made up your own mind on the game / mobo / etc. Personally I hardly ever look at review scores, as I'm more interested in what the person has written about rather than a "marks out of 5/10".
The thing is that today there are a lot of "good" products (i.e. 7/10) that don't show anything overly special - I guess that is the way the industry is maturing though, in that there are less bad products and more good products. Generally, we look at products on the perspective of being average to start with (somewhere between a 6 and a 7) and depending which side of "average" it is, depends on whether it gets a 6 or a 7 (or higher/lower if it warrants it). Obviously, there are products that are much better than "good" (or above average) and there are products that are worse than average. We are actually working on an article that helps to explain our ratings system a bit more clearly and where our scores come from. The reason they're there is that it not only helps some readers (that like reading graphs), it also helps manufacturers understand where they fell short of the mark (or did particularly well) on a particular product. There have certainly been a number of times where I've spent upwards of an hour on the phone with manufacturers have been unhappy because their product hasn't won an award. We reserve the awards for products that are really worth them, because that makes the system that bit more valuable; not only to readers, but to manufacturers too. In the past, a manufacturer that had a, say, 7/10 product would be classed the same as a manufacturer that had a 1/10 or 0/10 product because neither would get an award. It was a case of award or nothing, but now there is more of a scale and I think that the manufacturers appreciate that and it also keeps them honest. I hope this sheds some light on why we're using a ratings system and I really appreciate some of the suggestions that you have come up with so far. In fact, if there are some very good ideas that come from the community, I am more than happy to consider implementing them if they will help (like Oddball walking's suggestion). Tim
The thing is average is no longer in the middle as the are more degrees of bad than good. I cant wait for the article to clear things up though.
How do you rate the graphics in Darwinia? better or worse than Lost Planet? They're rated in context. H.B.
The other thing you've got to consider when complaining that there are a lot of 7's being handed out is that Bit don't want to review rubbish products (3's and 4's) because you (the readers) aren't going to buy them. It makes sense to order in for review products that the consumer actually wants.
Yep, it's a waste of thier time to review crappy games that no-one wants to buy, they don't have a warehouse of reviewers like dedicated game sites that review every game under the sun. I very rarely take into account scores web sites give anyway, if I'm lasy I look at the conclusion since thats usually a good place to find a summary of what the reviewer thought. Although, I do like the suggestion that several staff members comment on the game, I've seen this done occasionally, I wouldn't mind seeing this done more to get lots of sides to the arguments.
