Here’s the 8th follow-up post in my 10 Habits of Great Server Performance Tuners series. This one focuses on the eighth habit: Use the Right Tool for the Job.

There are many different reasons why people undertake performance analysis projects. You could be looking to fine-tune your compiler-generated assembly code for a particular CPU, trying to find I/O bottlenecks on a distributed server application, or trying to optimize power performance on virtual server, just to name a few. As I discussed in habit 2, there are also different levels where you can focus your investigation – mainly the system, application, and macro or micro-architecture levels.

It can be overwhelming thinking of all the different ways to collect and analyze data and trying to figure out which methods apply to your particular situation. Luckily there are tools out there to fill most needs. Here are some of the things you should consider when trying the find the tool(s) that are right for you.

After determining your specific needs, take a look at the tools out there (you might start with the lists available on wikipedia or at HP’s Multicore Toolkit.) Of course I recommend you also check out the Intel® Software Development Products. There are several specifically for performance analysis:

Finding the right tool for your situation can greatly reduce frustration and the time needed to complete your project.  Good luck, and keep watching The Server Room for information on the last 2 habits in the coming months.

The following are some considerations prior to tuning your MP Xeon 7400 series server. I can speak to this subject as I was asked to tune this system using the TPC-C and TPC-E benchmarks for internal measurements at Intel. While you may not be setting up thousands of hard disk spindles for your performance work, this blog post attempts to capture some of the key tuning considerations of this Xeon-based server.

Understand your system

The key to tuning any system, whether it is a formula one race car (I promise to stay away from silly car performance analogies) or a server is to understand it. Identify what components have an effect on performance and what components don't. This will narrow down your tuning efforts.

Architecture

Like all of Intel's platforms, an MP Xeon 7400 series server is made of several ingredients. Of course since I work for Intel I need to start out the ingredient list with the central processor. Our website has a good description of this processor here The MP Xeon 7400 processor is made up of three Core 2 Duo T5000/T7000 series processors. This provides six (yes six) cores for processing goodness. Each of the Core 2 Duo T5000/T7000 series processors provide 2 32KB level 1 caches (1 for data and 1 for code) and a 3MB level 2 unified cache. In addition to these two levels of cache the MP Xeon 7400 processor provides a 16MB level 3 unified cache. The other major ingredient to this platform is the Intel® 7300 Chipset. This chipset provides four independent front side bus links to the four CPU sockets. In addition, this chipset provides a snoop filter and four channels of FBD memory.

If some is good, then more is better:

The key thing to take away here is that an MP Xeon 7400 system fully populated with top bin processors will provide a whopping 24 cores of processing power in a four socket system. This is great for the enterprise benchmarks I use for performance testing as those applications are multithreaded and designed for multi-core processors. The same may not be true for your application, so please keep that in mind.

Another thing to remember is that an MP Xeon 7400 processor's design follows a growing pattern in the Xeon processor family. Specifically, I am referring to the addition of the level 3 cache (L3). This is also known as the last level cache (LLC). This follows the design of the Potomac (Xeon MP 64-bit) and Tulsa (7100-series) processors. The value of the large LLC is that it reduces the number of cache misses that would require the machine to go to FBD memory for the latest copy of a cache line. This additional level of on-chip cache comes at a price, though: higher latency. While the latency penalty is relatively low when compared to the latency to memory it is important to mention it here. Again, the LLC greatly benefits enterprise benchmarks I use for performance testing as they have a large memory footprint. The same may not be true for your application.

BIOS / Firmware / Drivers

It is very important to remember to update your system's BIOS, firmware, and OS drivers before you do any deep performance tuning. I can not over state the importance of this step. Your system's manufacturer should be able to provide the latest BIOS and firmware associated with your server. OS drivers are available through many sources these days. Typically these can be downloaded from OS vendors, hardware vendors, from the Linux open source community, or the platform's manufacturer.

Prefetchers

Intel processors have traditionally provided four prefetchers. These are accessible via model specific register IA32_MISC_ENABLE and sometimes via your OEMs BIOS. These features are meant to help the processor load data in a predictive manner to keep the cache hierarchy filled with the most pertinent cache lines. This is great if the application uses data in a somewhat predictable way. If your application uses cache lines in a random fashion, then the prefetchers may negatively impact performance. My best advice for you is to test your application with the prefetchers enabled and disabled. Table B-3 (MSR 0x1A0) in this link covers the prefetchers I am referring to.

Memory Population

As mentioned before, an MP Xeon 7400 series server will provide four channels of FBD memory. There are a couple of considerations here. First, latency to memory increases for every DIMM added to the system. This is important to note because you can keep the memory latency to a minimum by adding fewer high capacity DIMMs. Second, be sure to evenly distribute the DIMMs across all the channels. In other words, don't fill up all the slots on one channel and then lightly populate the rest.

An External Factor that may affect performance

Like many Intel designs, an MP Xeon 7400 series server will choose dishonor over death. I am referring to how it deals with high temperatures. The FBD memory inside an MP Xeon 7400 series server makes use of a thermal monitor on each DIMM. If the memory becomes too hot the chipset will begin to throttle memory bandwidth in an effort to reduce the temperature of the system. This will have a drastic negative impact to performance. So, keep your server room nice and cool.

To wrap things up here, we have looked at the architecture, the importance of BIOS/ firmware/ OS drivers, the prefetchers, memory population, and the effects of high temperatures. Your application's performance will vary, but I hope I have given you some things to narrow down your testing. So, by now you might be asking. "Where do I start?" Well not to be too self serving, but I would check out more of our blog posts here. A great place to start for performance methodologies would be Shannon Cepeda's blog. This series is a great resource for anyone interested in computer performance methodologies.

Here's the 6th follow-up post in my 10 Habits of Great Server Performance Tuners series. This one focuses on the sixth habit: Try 1 Thing at a Time.

Like habit 2, Start at the Top, this habit looks easy to understand and to keep. But, due to the constant desire for productivity, I and most others I know in the performance community have broken it many times. Some times I even get away with it. But trying to keep this habit is important, because when I don't get away with it, breaking this rule results in even more work than I was trying to save.

The concept behind this habit is simple - when you are optimizing your platform or your code, make only one change at a time. This allows you to measure the effect of each change, and only accumulate the positive changes (however small) into your workload. I have seen instances, for example, where 2 small changes applied at the same time to a workload cancelled each other out: one caused a small in performance and the other a small increase. If these changes weren't tested individually, we would have missed out on that performance gain.

Another thing that can happen in a complex workload is that two changes that seem independent can interact with each other. Like many developers know from fixing bugs, changing one thing may affect something else. Keeping all your changes separate can help you identify these interactions more easily.

You may be wondering when it is acceptable to break this habit. I think of performance methodology, and this rule in particular, as similar to the scientific method we learned in school. It's always good to follow it - doing so will help you quantify your successes and failures, stay organized, and defend your conclusions - but, you can still make a big breakthrough without it. In some cases, like when you are making small local changes to source code in completely different modules, or when you are changing two things you are certain won't interact, the habit can be broken. But the advice I give, especially to those involved in long-term optimization projects, is to follow it.

What has your experience been? Please share your "changing multiple things at one time" stories.

Keep watching The Server Room for information on the other 4 habits in the coming weeks.

Here's the 5th follow-up post in my 10 Habits of Great Server Performance Tuners series. This one focuses on the fifth habit: Know Your Workload.

Spend some time getting to know your workload.

The idea of a "workload" is integral to the concept of performance. The workload is the set of software and tests that you run on the server in order to measure its performance. Also part of the workload is the is concept of the "metric", which means, the number you will use to quantify performance. You should understand as much as you can about your workload in order to characterize and interpret your system's execution.

Let's look at the real-life example of a car's fuel economy. The EPA measures fuel economy using 2 workloads: city and highway. Each workload tests different aspects of the car's performance, and the metric used to quantify that performance is miles per gallon (MPG). Like the EPA's fuel economy test, a good workload for server performance tuning should have the following three characteristics:

• Measurable - There is a quantifiable metric.

• Reproducible - Measurements are repeatable and consistent.

• Representative - The workload should be typical of normal operating conditions and should stress the parts of the system (including code) where performance is most critical.

Depending on the usage model for the server(s) you are tuning, some example appropriate workloads might be: loading websites , processing XML, encoding/decoding MP3s, responding to database queries, rendering frames, etc. Metrics could be time to run, number of users serviced, transactions processed per second, etc. If your metric is time, take special care that you are measuring it accurately.

After choosing or creating a suitable workload, spend some time getting to know it. Measure the variance between runs. Use O/S and processor-level tools (to be discussed in the blog for habit #8) to sample the workload's characteristics at various points during its execution.

One thing to remember about sampling is that you want to make your sample interval at least as long as the amount of time it takes to complete a unit of work in your workload. For example, suppose your workload is a stream of web page requests and you are measuring response time. If the longest response time you see is about 2 seconds, then you want to make sure you take samples over 2 seconds in length. It's best to use a multiple of your longest operation time, so 4 or 6 seconds in this case. This way you can be sure your samples include one complete operation in the workload. Then try to determine if the workload is stable - meaning, do the characteristics vary at different times during execution? (If so, you will need to sample more often to understand the workload or possibly split it into phases). Use the data to get an idea of your workload's CPU, memory, network, and I/O usage.

At the application level, become familiar with the software stack you will use. How is the workload generated (user, clients, test files, etc)? Understand the major operations that occur - what components of the O/S are needed? What device drivers are used? And finally, study the application(s). Know whether the application(s) being tested are single- or multi-threaded and as much as you can about the internals.

Choosing (or developing) an appropriate workload is necessary for correct performance measurement and tuning. Being as familiar as you can with the workload will help you to interpret your performance data and identify areas for optimization.

Keep watching The Server Room for information on the other 5 habits in the coming weeks.

Here's the 4th follow-up post in my 10 Habits of Great Server Performance Tuners series. This one focuses on the fourth habit: Know Your BIOS.

My last blog talked about beginning your system tuning by consulting a block diagram. The other thing you should always look at is your system's BIOS. Many server BIOSes these days allow you to configure options that affect performance. Like everything in the performance world, which set of BIOS options will be best will depend on your workload!

First things first, how do you find this "BIOS"? Most servers have a menu called "Setup" (or something similar) that you can access while the system is booting, before it starts loading the operating system. This "Setup" menu allows you to access your system's BIOS. Changes that you make here will affect how the operating system can utilize your hardware, and in some cases how the hardware works. If you change something here, you usually have to reboot and then the change will "stick" through all future reboots (until you change it again). As platforms grow increasingly sophisticated, they are offering a widening array of user-configurable options in Setup. So a good practice is to examine all the menu options available whenever you get a new platform. Here are some of the most common options on Intel platforms that could affect performance:

• Power Management - Intel's power management technology is designed to deliver lower power at idle and better performance/watt (without significantly lowering overall performance) in most circumstances. There are 2 types - P-States, which attempt to manage power while the processor is active, and C-States which work while the processor is idle. In some BIOSes, both of these features are combined into one option which you should enable. In other cases they are separated. If they are separate, here's what to look for:

• Intel EIST (or "Enhanced Intel Speedstep" or "Intel Speedstep" or "GV3" on older platforms) - This is the P-State power management that works while the processor is active. Leave it enabled unless directed to change it by an Intel representative.

• Intel C-States - If you have this option or something similar, it is referring to the power management used when the processor is idle. Enable all C-States unless directed by an Intel representative.

• Hardware Prefetch or Adjacent Sector Prefetch - These options try to lower overall latencies in your platform by bringing data into the caches from memory before it is needed (so the application does not have to wait for the data to be read). In many situations the prefetchers increase performance, but there are some cases where they may not. If you don't have time to test these options, then go with the default. Intel tests the prefetch options on a variety of server workloads with each new processor and makes a recommendation to our platform partners on how they should be set. If, however, you are tuning and you have the time to experiment, try measuring performance using each of the prefetch setting combinations.

There are several other options that might affect performance on specific platforms. Some examples might be a snoop filter enable/disable switch, a setting to emphasize either bandwidth or latency for memory transactions, or a setting to enable or disable multi-threading. In these cases, if you don't have time to test, use your Intel or OEM representative's suggestion or go with the default setting.

Being familiar with how your system's BIOS is configured is another basic component of system tuning.

Keep watching The Server Room for information on the other 6 habits in the coming weeks.

Here's the 2nd follow-up post in my 10 Habits of Great Server Performance Tuners series. This one focuses on the second habit: Start at the top.

Let me start by relating a true (although simplified) story. My team at Intel has built up years of expertise running a particular benchmark. So when the time came to start running a new, similar benchmark, we thought: "No problem." We began running tests while the benchmark was still in development. Immediately we had an issue: the type of problem that would normally indicate our hardware environment wasn't set up properly. We checked everything that we had seen cause the issue in the past, and we couldn't find anything. So, we blamed the new benchmark. After all, we were experts and we had been setting up these environments for years! We knew what we were doing. You can probably guess where this story is going: after weeks of doing things to work around the "benchmark issue", we figured out that we had mis-configured the environment, resulting in a bottleneck on one part of our testbed. We didn't thoroughly test that part of the environment because it had never caused us problems with the old benchmark. And of course, on the new benchmark it was critical. We had broken one of the most important rules of performance tuning: Start at the Top.

So now you know how easy it can be to not Start at the Top. Even seasoned performance engineers can get overconfident and forget this rule. But the consequences can be dire:

• 1. You have to eat major crow when you realize your mistake. I'm just now getting over the humiliation.

• 2. You might have put tunings in place to address issues that weren't really there. This is at best wasted work and at worst something that you have to painstakingly undo when you fix the real issue.

So...how do you avoid this situation? Simple: use the Top-Down Performance Tuning process. This means you start by tuning your hardware. Then you move to the application/workload, then to the micro-architecture (if possible). What you are looking for at each level are bottlenecks: situations where one component of the environment or workload is limiting the performance of the whole system. Your goal is to find any system-level bottlenecks before you move down to the next level. For example, you may find that your network bandwidth is bottlenecked and you need to add another NIC to your server. Or that you need to add another drive to your RAID array, or that your CPU load is being distributed un-evenly. Any bottlenecks involving your server system hardware (processors, memory, network, HBAs, etc), attached clients, or attached storage is a system-level bottleneck. Find these by using system-level tools (which I will touch on in the future blog for Habit #8), remove them, then proceed to the application/workload level and repeat the process.

Being vigilant about using the top-down process will ensure you don't waste time tuning a non-representative system. And it just may save you some embarrassment!

Always measure your bottlenecks!

Keep watching The Server Room for information on the other 8 habits in the coming weeks.