3DMark and Benchmarking Editorial

Repeat after me :

I will read before I post, I will read before I post, I will read before I post.

Mind the door *******.

 

Sabretooth most of your comments are based on assumptions rather than facts which makes me wonder what you actually think constitutes a benchmark.

Certainly as far as graphics cards go, there are NO REAL benchmarks nor are there ever likely to be. A benchmark by definition requires a reference against which to compare. Unfortunately given the almost limitless configurations of hardware and settings not to mention the constant stream of new hardware/technologies possible in todays computers, all of which will affect any measurments, it is impossible to maintain a constant reference system. The bottom line is a card tested on one system cannot be accurately compared to the same card tested on another system thus making any measurments comparison virtually meaningless.

The same applies to games and application software used for benchmarking. How they are configured will have an affect on the test results. Since new hardware is constantly being developed, the testing software also needs to be constantly updated to test the new features thus negating any comparison to any tests that have gone before.

The use of a synthectic benchmark with no configuration options, locked down to only run on systems of tolerable carefully defined specification seems to me to be the way to go. Obviously this benchmarking software would have to be accurately representitive of the software in everyday use and simulate real world usage.

Even this set up is very limited since the test system would need to be changed regularly to remain representitive of the current state of the technology. (Imagine if we were still testing the latest GFX cards on a 486!).

I do agree that 3DMarkxxxx is not a real benchmark but rather a measure of bragging rights. It is up to the computing community, including the hardware manufacturers to come up with a real and practical method of accurately testing and comparing products. Until then as someone once said, (I forget who), "The World has the computers it deserves".

I see

 

hmmmm tbh i think you are spot on the mark haz with your article- hardocp seem to have there own ideas and poss more clout in the hardware world - and a reputation for being outspoken ...

i am a regular both here and my def homepage is hardocp ... purely for the hardware news - but i disagree with kyle too on various points ...

you need to remeber that the reason the manufacturers tune their drv's or cards to make them perform better is just a trick to sell more -- the bottom line is they make video cards to make money ... the way to do this is to beat the competition and prove they are faster, bigger , better - its all about the money at the end of the day - its what business is about.

hardocp might be one of the premier hardware sites ... but this is THE modding site - I have'nt found any better or more useful / feature rich ( sorry tigermain - but we do come close eh?) and more importantly I consider this country the forefront of moddin ... makes me proud to be british

/me wipes tear from eye

 

I've been coming to Bit-Tech for about 6 months now after finding it through Google. I have always enjoyed the articles, reviews, and forum views that I have found here. This article was no different. It expressed an opinion which piqued my curiosity and encouraged me to research and learn more about the subject of benchmarking. Nuff said about that.

3DMark2003 isn't very useful to me. It doesn't help me identify the bottlenecks in my system and in my opinion doesn't represent real-world performance. From my point of view it is primarily for bragging rights as evidenced by the number of copies downloaded in the first 24 hours after release. Yes, I did also download it out of curiosity, mainly to watch the graphics tests because they're kind of cool. It certainly wasn't to post my score on a website somewhere. But according to my score (92, I'm still using a Geforce 2 MX 400), my video card is pretty much useless, although every game that I've played still has decent frame rate and quality.

When the UT2K3 demo came out, I downloaded it and ran the benchmarks and received playable numbers (not at the highest quality settings but still fine for what I need). This was much more useful to me since I can go out and buy the game and not worry about my video stuttering and generally p***ing me off. On the other hand, when the leaked Doom 3 found it's way onto my system, the framerate topped out at 7 FPS. I am currently saving my money, hoping to beat Doom 3's release, to buy a new video card. This was much more useful in a real world way.

Maybe what someone should do is license the Quake 3, Serious Sam SE, UT2K3, and Doom 3 engines and use those to run benchmarks. They represent the evolution of older games to newer and would show where my performance expectations should be.

 

I just remembered why I'm such a loyal member of this website and forum group.... if you need me, I'm going to my quite place and pray.

 

I think that both articles hit on one point specifically...
The whole point of "the wonderful world of benchmarking" should be about the ppl. 'Benchmarks' as they are defined by community is something that is run on 2 systems that are about the same excluding 1 thing in order to determine which is better. However, given that this, in reality can not be done (due to differences in architecture of the northbridge, mobo, and CPU and the differences in the drivers).
Personally, seeing as I only play a few select games and don't have the money to upgrade as often as I would hope, I would like to see specific benchmarks in the games themselves, but that is a side-note.

The only way I think this is going to work is to get ALL parties involved (programers, hard-ware techies, AMD, Intel, VIA, Nvidia, Ati, Compaq, HP, etc., etc.) in a non-profit group that will attempt to search this subject extensively, thoroughly and up to date in a timely fashion. I think we all can see the massive scales we are now diving into here. In order to do this, I think it would take an uprising within the masses (i.e. the consumers) to force everyone to use a common standard. I think we can compare this to the English-Metric measurement debate, can we not?

Perhaps the best for everyone involved is both, a benchmark within each game as well as a synthetic benchmark as well, and from there, let the consumers become informed of thier prospects and make confident purchases.

 

Well, nice to see that some people are actually interested in discussing the topic (I'm guessing that's everyone but Sabretooth)

3DMark2003 is pretty and all, and if you compare yourself to similar systems online (rather than exact systems) you can see a small differential in scores that indicate things like overclocking etc But overall, it has far to much bias worked into it to be a standardised tool.

I think the latest article on tomshardware.com sums it up pretty well after Nvidia re-ran the test with a new driver release and managed to outstript the 9700 by about 500 marks after being well behind.

To paraphrase the last section, a person is reported to have said that with a few changes written into drivers that cue for benchmarks, it isn't hard to make any graphic driver play nice for a benchmark. Given that, is any synthetic benchmark trustworthy?

Real world game marks are still the most trustworthy way of measuring performance, but with increasingly different systems. And considering that card manufacturers have been caught in the past fiddled drivers to yield better timedemos, can anything be trusted any more.

Results can say anything you want. Nvidia says one thing, ATI will work out an arcane way to top it and vice-versa. The only thing that really matters is are you really happy with your graphics currently?

I personally think the benchmark looks really good on my Radeon 9700pro, but it does chug a little (P4 2.53, 512 DDR333), but most of my games run really damn well. To bitch about poor performance in a synthetic benchmark would be incredibly bad form considering how good my games run.

If you want to know whether a card is a good upgrade, go to sites such as bit-tech (yeah, I know it's shameless to plug you on your own forum, but what the hell) or tomshardware and get the scoop. I wouldn't buy an FX if my life depended on it in it's current state, and I came to that conclusion based on what people in the know had to say about it, not a single benchmark.

Cheers,
Asmodai

ps sorry bout length of rant, long afternoon at work has left me with the need to express myself at the cost of others

 

A good example of how synthetic benchmarks can be wrong is that when i put another stick of 256mb ddr ram in my 3d mark score dropped but the performance of my comp improved drastically.

Also people probably shouldn't just rely on benchmarks. As haz pointed out there aren't many alternatives.

 

Wow, interesting topic ya got going here. I first read the [H] article and have to say by the time I finished it I pretty much completely agreed with Kyle. Then I read Haz's article and agreed with it. It brought up some things I hadn't considered at first. Now I'm somewhere between the two. Mostly because of what Macroman points out. The differences between systems makes the scores meaning less than perfect. I watercool and I don't like posting my temps because the amount of variables that come into play make them almost completely useless as a comparison to anybody elses. One thing I find useful with both temps and bechmarks are before/after comparisons. If I compare results after making some type of changes in my system, then I can find some useful meaning in the numbers. That is my #1 use for benchies. If any of you are familiar with BillA'a water block testing method at overclockers.com where emphasis is placed on repeatable results and eliminating variables, not to mention calibration and margin errors for the test equipment, then you have an idea of he type a benchmarking that Macroman speaks of.

After saying all of that, I won't deny that benchmarks aren't here to stay and manufacturers won't cater to them. Those are just facts of life. I would like to see improvements though. And to me there seems to be two distinct areas for benchmarking.
1) For reviews.
2) For end users.
For review type benchies, I really don't know what can be done about that area. For the end user here's kinda what I would like to see. Something free or cheap. And rather than all the game makers having benchmarks with the game, I'd like to see them make available plugins for the user benchie. That way I could d/l the plugins I wanted, for games I have or games I might b interested in. This would give me some idea of how the games will run on my system. Although a lot of games use the same engine, they use them much differently and these plugins would reflect that fact. Also I could do some tweaking and have the before/after usefulness still available. This would save me the expense of buying a game just for a benchmark. Also I would think the game developers could do this easier and cheaper than the demos they put out, and probably still get the same excitement/sales generated out of it. This is a very important factor in getting their co-operation. Also something like this could be used for reviews, but if so many game plugins are available ( like I'd hope) it could possibly result in more variables thrown into the equation. Possibly a standard set of plugins could be agreed apon to avoid too much confusion. And lastly, because the plugins would be using the version of the engine found in the specific game, the hardware makers would be tweaking the game and benchie results at the same time. Well thats my spin on it from an end user's point of view, also included are a bit of economic factors, which sadly, will be the deciding ones. There is a possibility for it to work out for the good of all. Hope the ball bounces that way. Sorry for being so long winded.

peace.
unloaded

 

I think Sabertooth is a witch and is overly sensitive about the subject...

But seriously, I don't think Haz was insulting [H], merely voicing ideas brought about by their article.

However, my evil alter ego whispers, "Look at [H]! I'm sure they have connections to nVidia. I mean, that one time they reviewed Visiontek's replacement company (BGD, BGF or BFG or something) they gave them an amazingly high score. And that company was in cahoots (yes, cahoots I say!) with nVidia!"

*ahem*

I like 3DMark 2003. It's pretty. But I only use it as a benchmark to brag.