Take a look at the chart below ... it's telling you something... isn't it?
It's more than performance numbers and marketing, it's data... REAL data!
But what does it mean - and ultimately - how can you relate to it?
If you're really into high-powered computing, you're probably quite familiar with common benchmark data. With every new CPU release, there are tons of new statistics, models, and ways to test the increased performance of the newer technology device - in this case, the 45nm based CPUs just recently launched this month. But what exactly does all this data amount to? Reading benchmarks is more than just seeing a bar chart - there's a science to digging into the data...
First, lets take a step back for some of you who may not fully understand what benchmarking is for. Benchmarks help to provide a common ground for comparing the performance of various systems across different CPU/system architectures. A common set of instructions (or programs) are setup to run within a regulated guideline to ensure the testing is performed equally across the competing platforms or architectures. Very much like in sports, if you have two different runners - they run the same path - i.e. the 100 yard dash. This creates the comparative benchmark.
So let's get back to the latest hot stuff - the Intel Xeon 5400 Series and Core 2 Extreme QX9650 Quad Core based processors. In the past 18 months, computing models have taken a giant leap forward by adding more CPU's per socket thereby increasing the thread density of your platform. In dual socket systems, you used to have two threads you now have four or even eight! And in quad socket systems the count can go up to 16! You're increasing your capacity to perform computational data by a factor of 3 or 4 depending on the platform. This has made a tremendous change in how benchmarks have had to be setup to run and we have to evaluate the testing methods to ensure we're maximizing the computability of each platform.
There are a few key steps to take before you consider benchmarking your system:
identify your problem area (processing power, network bandwidth, memory utilization, etc)
identify your competing products
evaluate the 'leaders' in your problem area
survey for available benchmarking tools
evaluate 'best practices' for testing (e.g. lower idle power based processors won't really help much if you're only doing high-end computing)
and then - implement your findings in your chosen architecture(s)
In the high-end server space you usually see more vendor specific data rather than end-user testing. Primarily because of the finite set of data that server administrators are looking for. Many of these 'industry standards' are monitored for efficiency and ensure the end-user that the testing was properly performed and the results are repeatable:
Industry Standard Benchmarks
BAPCoan industry consortium developing benchmarks for Windows personal computers
Intel uses many of these standards for benchmarking - as you can see here in the Xeon 5000 Series based Processors Benchmark Page
Even if you're a server admin, you most likely interact with clients for day to day performance as well. If you search the web for CPU benchmarks the most commonly viewed benchmarks are performed on the client side of computing, mainly because of a few factors:
clients are usually cheaper and more abundant to test with
visuals in client computing are usually more fun to watch than seeing SQL data fly across the screen (hey - just being honest here!)
and servers in general are built for more specific reasons, whether it's application, storage, modeling or other specialties
Many of you have probably heard of benchmark sites such as: Anandtech, Toms Hardware, FiringSquad, HardOCP and many others (respond with your favorites please!) Each of these sites use common tools/applications to benchmark the latest and greatest hardware against each other. Depending on what you're looking to do with your hardware really determines what/how you want to benchmark your system (or look for data reviews for your configuration). After all, a machine that can run the latest games at over 60 frames per second may not be the best SQL server for your datacenter - right?
If you're looking for quick 'brute force' computational tools to try your hand at CPU benchmarking, try something simple like BOINC, Super PI, or you can get more elaborate by using some methods as described by C-Net by using Cinebench, or SiSoftware Sandra. Once you've figured out some of the basics - and can repeat these simpler tests - you can jump into those Industry Standards and get into some serious work!
So in closing, there are so many variables to account for when looking to validate the performance of a given system. Processor speeds, I/O subsystem configuration, memory latencies, network bandwidth, power utilization, etc... the permutations are nearly endless. So you have to be diligent in initially addressing your key problem(s), and attack the solution in benchmarking using the best known methods. Also, when reading benchmark information BE SURE to read the configurations of the systems in question - are they truly comparable? are the components running at spec level or overclocked? Are the speed differences negligible, or substantial in real-world evaluation? And finally, focus on what's important to you and your computing requirements - after all, you need to be sure you've picked the correct system for your needs.
