The way it's meant to be reviewed/evaluated

At the moment, there are a number of thoughts going through my head as to the way that I would ideally like to see graphics cards being reviewed, or evaluated. But I'm at the stage where I've come up with a "plan", but I would like to get feedback on how far you guys would like me to go.

I have the inclin to move totally away from time demos and pre-recorded scenes (and benchmarks, such as 3DMark05), for the simple reason that they do not represent a full on real-world gaming experience. Also, time demo's (such as Doom 3's timedemo feature), don't have frame rate caps like actual game play has, so frame rates may be misrepresented and inaccurate when you come to actually play the title at the resolutions that we've chosen.

I'm looking to move in to using a wide range of titles, from a wide range of game types with the hope of catering for as many people as possible. It would be good to get an idea of what you would like to see included. I'll need to be able to use FRAPS to record frame rates, and there must be portions of the title that are able to be repeated.

You're all used to seeing the apples-to-apples comparisons with graphics cards, but it's not easy to do nowadays, with the two major graphics contenders doing things completely differently. It's impossible to get a true apples to apples comparison between the two manufacturers any more in my opinion. Also, the 3DFX mentality is slowly moving out of the picture, no longer is the highest frame rate important. It is the highest image quality at a more than playable frame rate is what counts nowadays, in my opinion.

So what I have planned is to move away from the apples to apples, and move towards "best playable image quality" rather than "highest frame rate at a set resolution and detail setting". I feel that it's the way forward, after all, you guys are playing the games, and you're going to want to know what resolution/detail/image quality you'll be able to run your SLI'd pair of GeForce 6800Ultra's at.

I would say that setting a minimum average frame rate of 40-50 frames per second would be about right for a "playable" frame rate, with hope that the minimum frame rate doesn't drop too far below 20 frames per second.

I can expand things if people aren't quite sure on what I'm trying to get at here, after all, I'd rather you guys tell me that I'm not doing things the right way, than going off, doing it, and then realising that it's totally the wrong thing to do.

Now, the question is... how far are you willing for me to go?

 

I've gone for "Best Image quality, with a playable frame rate". as that's how I play games myself. Although IMO your review are great as they are

 

I personally shoot for a 40 FPS average framerate. The places in my mind where framerates are most important is the classic first person shooter. Having to repeat things to get accurate numbers is tedious and time consuming. If fraps doesnt have one, you should use a statistical measurement that keeps track of the lowest Framerate, highest framerate, and average. I agree with your philosophy of "Decent playable framerate and High Image quality" over "Maximum Frames per second". I for one will be attempting to follow your philosophy when I get the opportunity to review graphics cards for www.moditory.com

Go as far as you feal necessary in order to get an accurate representation of real-world performance. If bit-tech had the server space and bandwidth (and we all had internet connections fast enough) Id say record high resolution clips and put them up for analysis. youre doing a great job bigz; I commend you for your efforts and think your plan is good If there is one issue that is worth considering is that image quality is very subjective. With the graphics industry as it is, its getting difficult to make firm conclusions on which card is better. Other than that, I think you're good to go.

 

I've voted 'image quality' and to be honest, looking at the current poll results, think it's going to be pretty much a one horse race.

I do think that there should still be a place for pure frame rate tests though, even if they aren't always on a level playing field, that's what a percentage of people will always be interested in. With my aging 9500 I'm not really one of them. However I'd imagine that a dual pronged approach such as this would increase the workload of the reviewing.

Ultimately some people will always want the card that 3dMarks the fastest. I applaud the movement towards this new way of apprasing gfx cards. The leading enthuiast sites have the power to shift the way the community look at gfx performance. Once this occurs the big players in the industry will be forced to change what they deliver in any given product.

Having said that 'Image quality' seems to be a difficult thing to quantify and very subjective. How do you plan on presenting / illustrating the findings in the reviews?

 

You're correct, it isn't an easy thing to quantify... but if a screenshot was taken (exact same screenshots, at the best "playable" image quality), comparing them along side each other should show you that a GeForce 6800Ultra is going to look considerably better than a GeForce 6600 (vanilla), as it's capable of playing at a much higher resolution and detail setting.

Aside from that, we can make brief judgements about Anti-Aliasing, Anisotropic Filtering and Texture quality on each of the cards. At the end of the day, image quality is very subjective, but on looking at the images you will be able to make your own decisions as to which card offers the better image quality for your needs.

Graphically, I'm not sure how to go about things... I would most likely end up using minimum and average frame rates in the form of a bar graph, as they're the two most important frame rates when playing a game.

 

as people have said if it looks pretty and doesnt jump to the eye then who can complain?

if it does 40more fps but looks identical no one will know so it wont matter. unless you voted option one.


Honestly the only use i see for benchmarks like 3dmark 05 is for people to use it as part of a suite of tools in overclocking to see how different changes will effect your computer. Not how games will work or look.

 

I like your thinking Bigz, moving away from the standard boring was of benching and testing.

I answered best image quality at playable framerate, but secretly I do think that a HIGH (as in NO CHOPS or hickups or NOTICABLE fps losses) framerate is very important.

 

Imo, 80fps average is a playable framerate for an fps, with a minimum of 60, for the responsiveness required in a game such as counterstrike or call of duty, ut2k4 etc....

Games such as comand and conquer generals and similar only seem to need around 40fps, as it isnt as important to be so responsive to mouse movements.

I'd like to know what detail settings I would be expecting to run, with a few full size screenshots, on a particular card in a reference system.

 

I do agree about this, but I feel that is for you to find out... we're attempting to stress graphics cards and find a frame rate that is playable... of course, lower the detail slightly and you will improve performance. We'll hope to be giving you a "benchmark" as to what you can expect to achieve at close-to the limits. Take the results as a maximum - 40-to-50 frames per second is pretty playable imho... it's just a case of finding a happy medium for graphically representing the performance of each card without it getting TOO confusing.

Answers on a post card

 

Thinking about it... maybe covering image quality first, and then moving on to frame rate? After all, if the focus is to be on image quality, that should be your first comparison

 

Sounds good to me

<other topic>
Wow, Nvidia certainly doesnt seem to be going for reviews "The way they were meant to be reviewed". The UK got completely screwed up the Ass with the 6200 and not a board was to be seen on that side of the Atlantic. I am dissappointed and surprised that nvidia would do something so stupid. Wow.
</other topc>

Good job with your efforts to do good reviews, bigz.

 

boards got caught in customs, from what I've heard... still if that is the case, there was no need to rush the launch out imho.

 

Of course that means you will have to compile a "most used" list of Screen Res's correct??.

In other words we shouldn't be seeing the results of a test @ 640X480..

I guess what i'm trying to get at is what will be the standard and what will be the varables? Set the res to the monitors native then reduce the toys (shadowing AA AF etc) untill it's playable? Or turn on all the toys then reduce the res till playable?.

I ask becouse i can't see you making both a varable as that would intail a huge amount of test runs for each game.

I gues i'm trying for a better defination of Best Image quality

 

Not every board will be able to run with AA/AF, or high resolution. I'll have a rough idea of performance from running a demo, and then I can establish the rough area where I'm likely to find the best looking, playable frame rate. Once I settle on a setting/resolution, there'll be a screenshot of the game settings. Driver settings will always remain at "Quality" with *valid* optimisations in place, i.e. optimisations that don't degrade image quality to an extent that turns out to be very poor.

I spend quite a lot of time reviewing boards at the moment, but I feel that testing with definitive resolutions/detail settings will prove to be a better way around things.

The variables will include game settings, as well as resolution and driver settings. We'd like to provide a guide that is as definitive as possible - sure, it may make more work, but I still feel that it's possible to crank out one a week at minimum.

 

(w00t, minority)

I personally prefer a high framerates, but if i can get good framerates and decent image quality I'll adjust my settings accordingly.

 

although i voted for better quality, i can live with poorer quality if i get a better frame rate to make up for it, but cards should be tested to there limits in terms of image quality.

 

I say best quality. i aways play the games with highest settings possible. i force 16xAF and 8xAA in my control panel and pick the highest settings in games.

 

Thanks for the input guys, all valid points that are being taken on board here