Hardware The Problem with Real World Benchmarking

With Nvidia, VIA and AMD exiting from BAPCo, we take a look at real world benchmarking.

http://www.bit-tech.net/bits/software/2011/07/05/the-problem-with-real-world-benchmarking/1

 

It's overcomplicating things. Why not simply ask:

But can it run Crysis?

EDIT: Oh, and in response to Tularin's post (#4): First!

 

I could justify upgrading our entire estate based on that

 

"But can it run Crysis?"
Congratulations, you're responsible for making the 4,877,921,204th post containing this joke. Unfortunately, it's now less funny than "lolcats". You could have at least wasted your comment by partaking in Internet tradition, spewing "First!" meaninglessly into the world.

If only we could have a time when people contribute meaningful discussion to articles, perhaps wondering if Intel influences Bapco's habit of only focusing on a tiny sliver of the real world?

 

As pointed out by the article, it's unsurprising that Nvidia and AMD walked out because *puts on whinny voice* 'it doesn't represent GPGPU usage'.

What is typical PC usage? Most of the time I'm using an internet browser with about 10 pages open, outlook, three or so word documents, a few PDFs, an excel spreadsheet with perhaps only 300 cells being used and a couple of paint files. Do I need the latest and greatest hardware to do this? Of course I bloody don't!

 

Benchmarking Crysis is a necessary tool in the hardware reviewers arsenal and relevant to this article.

 

I agree with Pete, If you want basic stuff you could do with everything with a PC from 6 years ago. If you want to play the latest games just put in a middle segment recent video-cart and you can play it fine.

I on the other hand have just a amd quadcore with 8gb of ram and a GTX460, all more then capabple to furfill my needs. Like movie editing, fast unpacking of downloaded movies and series, playing games and other regular stuff you normally do. I call that a wise investment.

Why on earth do i want to have a 1200watt PSU feeding quad sli with a i7 990X processor on 32BG ram and a 400 dollar motherboard with lights and knobs on it which i will never use once it is inside the case and a 160GB SSD for which I can buy 6x 2TB drives.

So I can read my mail faster or start my PC faster? yeah right..

For the record, my PC is en sleep-mode 24/7 and starts in 3 seconds with the move of a mouse

When I was young and naive I always wanted the latest and the best. I even bought newer revision of a motherboard, Yeah crazy! I know better now...

 

Says the guy with tri-sli 580s.

 

This is the same benchmark that Intel has had the same say in for many years. It's one that AMD championed for much of that time - making it the standard for government purchasing orders in America. Why did they do that? - because the A64 was a great cpu, it did well in BABCO and hence AMD could win some deals over Intel despite it being the smaller player.

So why are now are they really pulling out?

The strong suspicion is because bulldozer is weak, because if those same government dept's that keep using BABCO AMD will sell nothing. Hence it's not really to do with break down, or gpu compute or any of that it's to do with selling bulldozer. AMD need to minimise use of BABCO for competitive benchmarks of Xeon/SB vs bulldozer. It's all smoke, mirrors and marketing.

 

and nvidia and via walked out, just to show amd some support. buttom line is tech is changing with more cores/threads and some software does run better on gpgpu FACTS and benchmarks should reflect that. why did amd champion it before? because they had better cpus? yes, but everything ran on the cpu back then.

 

Notepad is very hard to run I'll have you know!

In all seriousness though, if I didn't need to do CAD, I could probably still get away with using my first PC I ever got - 14 years ago. Ah, the good old days of Windows 95.

 

Here's an idea - how about task-specific benchmarks, with their own individual scores? Interested in video editing? Run a test of VideoMark. Are you running complex financial calculations in Excel or Matlab? Run CalcMark. Do a lot of editing in Photoshop/Lightroom? PhotoMark will help you decide. You get the idea.

 

Why didn't Bit-tech mention the fact that if you change VIA's cpu ID (which you can) to Intel it gets a higher score (not in 2012 but it was occurring, so to with PC Mark 2005) and it becomes blatantly clear why companies want to leave.

Intel cheat. Flat out. Now i've only ever bought Intel CPU's so fanboi accusations won't work, but its just a fact. Have a look at register and compiler optimizations for Intel in older versions of SuperPi as well.

 

Bulldozer doesn't have a gpu.

 

Good article, thanks BT.

 

SysMark has always been a CPU benchmark, you want GPGPU, use something else.

call me cynical, but i thought the reason AMD walked out was due to their new processor was unable to compete.


then there's the matter of real world benchmarks. if simple task is all you will do, then why even buy latest CPU or GPU based on benchmarks? how about just buy a tablet? fast-enough computing doesn't need benchmarks, all it needs is a simple user feedback of "can it handle my facebook page".


in summery, i think people focus on the benchmarks way too much. call me Apple fanboy, i value user experience more than underlying hardware, a fast-enough system is the same as fastest system when the software is the same. Also, benchmarks can only tell you A is faster than B at task C. it can never tell / has never told anyone A is always better than B. so what's all the commotion about?

 

Never said it did. And wouldn't that mean if the score did include gpgpu bulldozer would score less? Amd have a range of cpus, including gpgpu would give a more rounded performance benchmark, excluding it gives a less honest result.

 

Yesterday (no that's a lie, it was only 20,000 rows representing 3 months data)

Web browsing, social networking and Youtube are either barely allowed (web browsing) or blocked (social networking and youtube)

Very true.

Yup, updating on a business scale is a giant pain.

Well it IS called a business benchmark, not a Joe-average-looking-for-lolcats-videos-benchmark
...
Anyway, a Office and email productivity benchmark would be right for businesses (okay, incorporate SAP maybe)
For "average-user-Joe" we'd need a different benchmark, I agree.

 

35k rows in Excel is not unusual. How else do you validate millions of rows of data when building databases, or building reports. It's a valid measure for those of us that use massive amounts of data in our day job.

Benchmarks should be selected based on their appropriateness to you. A portable computer that'll play Crysis may well not yield the battery life that work users crave. It's simply a metric to guide users. Manufacturers have always picked and chosen the grounds on which to pitch their marketing fights, and this is no exception.

If the perfect piece of hardware that does EVERYTHING fastest had been invented, then this wouldn't even be being discussed. Until then, we have to learn to look for trustworthy sourced that bench using applications that are relevant to us. That's why I have a suite of short-cuts to different sites to read their opinions/results.

 

okay who is this mystical "average user" you guys are constantly talking about?

Also, for a work PC I got an AMD quad core with 4GB RAM and a GeForce 210 card powering dual 1080p monitors. I mostly do customer support, and registrartion so I need to use Excel a lot, Outlook, several web browsers, PuTTY and other small tools here and there. I also run VMWare to run some older operating systems in case some customer runs older software.

All except VMware would probably fly on a older single core computer with only 1GB RAM.
But this ensures I never have to wait, ever on anything running locally.

I'm nor sure how Sysmark gauges this type of workflow. You would have to have some benchmark framework that would allow you to set up your own tasks to run.