The Server Room Blog

Your server workloads may be cheating you on power... get it back!

todd.christ@intel.com — Fri, 06 Nov 2009 20:12:17 GMT

Prior to the Intel Xeon X5500 Server Platforms*, measuring server power was done via expensive equipment and could only be performed in a discrete fashion. Unless you had tons of monitoring equipment to mash-up your power data - it was a tedious process. Now, using Intel DCM and Node Manager - you can pull multiple servers worth of power info to make some important power decisions in your datacenter.

First of all, you need to baseline your workload. If you're confident that you can replicate workload patterns then you've got a starting point. Otherwise, it's usually a good idea to start monitoring and looking for some cyclical patterns and/or common data points (time, power, thermals, etc) to keep track of.

In this scenario (like in my last blog) we're using a SQL workload which can be modified to run the CPU at high levels for a relatively set amount of time. The base workload runs for 7 min 30 seconds, as shown in the Intel DCM screencap below.

In this test case: Idle power for the 4 servers is 782W, and under load - the power increases to 1174W - which is a delta of 392W. This power increase occurs when work is given to the server and the P/T states react to the workload and increase power/voltage to the system to increase performance. Exactly what we've been used to seeing even since EIST was introduced several years ago.

Now, what I'll show you is something that may be very interesting in scale... I will power cap the servers by 20W each, and set the Intel DCM Power Policy to only allow 1095W for the 4 servers in the rack.

What is awesome here is that we can still finish the workload in the same 7 minutes 30 seconds. So essentially, we have saved 80W of power for each set of 4 servers and still get the same amount of work completed! In a large datacenter this can be HUGE in energy savings.

Let's do some quick math: 20W power savings per serer x 10,000 servers = 20kW power savings and you still get the work done. I hope I just helped some of you server admins get some new ideas on your next "I need a raise" talk with your manager

*your mileage may vary, so test your own workloads and report out!

Intel Node Manager Ecosystem Video IDF 2009

todd.christ@intel.com — Mon, 28 Sep 2009 21:56:09 GMT

The ecosystem is growing...

Sean Maloney's keynote presentation at IDF 2009 highlighted Intel Node Manager. This is the video from his keynote which shows customers from Baidu, BMW, Oracle, and Telefonica, who have been working with Intel on Intel Intelligent Power Node Manager.

Check out the final slide showcasing the OEM/ODM/Console providers and customers using Intel Intelligent Power Node Manager.

Use Cases for Intel Intelligent Power Node Manager

todd.christ@intel.com — Fri, 28 Aug 2009 20:31:09 GMT

Intel Intelligent Power Node Manager is a new technology that is available with the Xeon 5500 Series Platforms released earlier this year. Many of you have asked me questions via Twitter (@Toadster) about "How can I use Node Manager?" - so I wanted to present some simple use cases to simplify the explanation of Node Manager and how you can best use the technology in your own enterprise.

First of all, let's explain the growth problem at hand. As servers shrink in size, the density of each server 'footprint' is growing from a power perspective... a few years ago, a single 42U rack could hold about 21 servers (estimating 2U servers) - and usually hosting one or two apps/servers per physical server, depending on if you had single or dual-socket servers. In modern datacenters, that same 42U rack can hold 42 servers (1U each) with 2P per server - so you have an immediate density increase of 2X the # of servers, and 2-4X the number of sockets - which can equate to 16X the number processor threads per rack... one good thing is that Intel has been developing newer technologies to keep the TDP of each CPU roughly the same over the same time period between processor updates... where you used to have 2 or 4 cores, you now have 8 to 16 cores at the same thermal envelope!

Knowing how much power your platform uses is a key factor in populating racks and rows in your datacenter. Prior to Node Manager technology, most Datacenter Managers would base rack population on 'nameplate' power - or the (W) rating on your power supply. That's the 'max' power utilized by the platform, and what the PSU is rated for (worst case). See the image below...

As you can see - using Intel Intelligent Power Node Manager technology, you can view your system's power utilization in real-time using Intel Datacenter Manager and the administrator can implement the power caps to ensure your server rack stays within your required power limits. By utilizing the 'actual' power limits instead of nameplate power, you can increase your rack density thereby increasing your ROI, and decrease your TCO! Lets face it - everyone loves saving money!

Many of us are familiar with this next scenario... it's summertime, and the power company is announcing that the power grid is under strain. Personal homes start having their A/C cut-off to save the power grid from brown-outs... now your enterprise can help reduce those risks as well!

Over the next few weeks, I hope to post more blogs/videos:

1. Single Node Power Monitoring & Management
2. Group/Rack Power Monitoring & Management
3. Thermal Monitoring & Management

Please provide some feedback, and post your questions and ideas for upcoming blogs!

Intel Intelligent Power Node Manager... where's the on-off switch?

todd.christ@intel.com — Fri, 17 Jul 2009 01:04:49 GMT

One of the recurring themes that I've been noticing from end-users who are testing or evaluating Intel Intelligent Power Node Manager (or Node Manager) - the question is "How do we turn it on or off?" To put it simply - when you have a Node Manager capable platform - you can simply put it to work and let your power policies decide when to enable/disable the features...

So let me step things back a bit and talk about the technology itself first. Node Manager is very much like any *T technology that Intel has deployed over the past several years, it's an ingredient - or in this scenario a mix of ingredients that is available at the platform level. Here are the 'ingredients' that when combined, give you the ability to monitor/manage power, and in some cases monitor thermal events.

The platform is based on the Xeon 5500 Series Chipset (codename Tylersburg-EP) server board
Xeon 5500 Series Processors (codename Nehalem EP)
Node Manager Enabled Firmware with the Manageability Engine
Server chassis components that meet IPMI 2.0 specifications for monitoring (e.g. thermal monitoring)
PMBUS Power Supply - this communicates with the Baseboard Management Controller (BMC) for platform power usage

For those of you wanting to get your hands on this technology TODAY - check out the Intel Server linueup:

Intel® Server Board S5500WB (codename Willowbrook) which is optimized for IPDC deployment, and supports IPMI 2.0, Intel Intelligent Power Node Manager, and can also support the Data Center Manageability Interface (DCMI) 1.0 specification.
Intel® Server Board S5520UR (codename Urbanna) is the mainstream Enterprise platform which support IPMI 2.0 and Intel Intelligent Power Node Manager

Both platforms work in conjunction with Intel® Data Center Manager (Intel® DCM) which is the SDK which provides power and thermal monitoring and management. This SDK allows group and policy based management for single server, rack, logical group, lab, or whole datacenter models.

Ok - so that reads like a bunch of marketing stuff... but here's the 'guts' of the technology...

When you purchase a Node Manager enabled server, there are a few simple steps to take to set things up to monitor/manage your server.

Most likely you'll need to setup your BMC, Intel provides a CD based implementation to help with this in our servers - it's called the Intel Deployment Assistant. This lightweight OS bootable CD can setup the most common BIOS settings, check versions of firmware and update them via Internet connection to ensure you have the latest BIOS, BMC, ME and Sensor firmware. Each OEM will have their own methods but should be similar in function when it comes to setting up the server for monitoring.

The BMC needs an ip address, netmask, and default gateway setup - and according to IPMI specifications - you can also set the administrative (user) access rights if you would like to tighten down security a bit. Once you have these access points setup - you can utilize standard IPMI commands to communicate with your server or use Intel DCM to really 'visualize' the capabiliites of Node Manager.

Here's a great demo video showcasing some of the Node Manager & Intel DCM use cases:

How many of you have worked with IPMI management before?

The technology that has been around for a while, but now Intel has put automation and policy based management features into the platform - thereby reducing costs, increasing responsiveness to power policies, and also making Xeon Servers more energy efficient than before. Many of our customers are asking for Node Manager enabled servers - is your OEM on track to deliver?

A new disaster recovery model using Xeon 5500 Series Servers...

todd.christ@intel.com — Fri, 12 Jun 2009 23:32:26 GMT

Recently in our test lab, we experienced a cooling failure... and I wasn't even sitting in the lab to realize it. In fact, I wasn't in the same state!

With the recent launch of the Xeon 5500 Series servers - I have been testing some use-cases against four of our servers in our lab when I noticed that the temperature was rising pretty drastically in there. How did I see this? Using Intel® Intelligent Power Node Manager embeddd in our Xeon Servers and using our Intel Data Center Manager (DCM) SDK software interface - the data is presented in a visual format.

In the graph above, the dark colored line is the "front panel inlet" temperature, and in a matter of minutes, the temperature in the lab rose from 71F to 87F - 16 degrees! What I didn't have setup is the scenario is a power policy that activates on a thermal trip. Here is how you would setup this policy in Data Center Manager under the Policies section for this rack:

In the event that a thermal event occurred that would cause the room to heat up to 78F (as shown above) - Intel DCM would send the IPMI commands to the platform which in turn would tell the Node Manager firmware to throttle-back the Xeon CPUs to their lowest P-state possible. This would reduce energy consumed across the systems in the policy group as well as reduce the thermal output of each server. This would in turn generate less heat across the servers thereby reducing the load placed on an already overheated lab or datacenter.

This gives the server managers more time to gracefully shutdown systems, and/or move the workloads to cooler sections of the datacenter. If you have ever experienced a cooling failure in the datacenter, it's a usually a frenzy to shutdown machines to minimize heat and/or power utilization overall. This thermal policy can give you more time before you reach a critical temperature where you start losing components, servers and ultimately - loss of data and productivity.

Using standard the standard IPMI interface, the Data Center Manager SDK and Node Manager on the Xeon 5500 series platform enable power monitoring, power management, and front panel inlet monitoring. This gives a server/datacenter manager the capcity to measure power usage per server, where you'd have to previously have more expensive power measurement tools. External power meters cost anywhere from a cheap $15 to spendy $1000 - but now the technology is embedded into the firmware on the machine.

You can learn more about the Xeon 5500 Series Processors on the Intel Xeon website.

Real applications of power management for the Intel Xeon 5500 Platform

todd.christ@intel.com — Thu, 23 Apr 2009 23:40:39 GMT

Back in the ‘dot-com’ days – many companies would build datacenters across the globe with one thing in mind – performance – and costs weren’t an issue. It was all about getting the job done, with little concern about the costs. Well, times have changed and companies have become more energy conscious, not only to become better stewards in using natural resources, but consumers are looking for companies who can design and develop products that can meet their own ‘green energy’ power needs. It’s not as important anymore to make or build something to be the ‘best of class’ it also has to be ‘efficient’ while being the best.

Corporate initiatives to reduce power but still “KTBR” (Keep the Business Running) are imperative to sustaining business today. Not only do you need the best performing servers – but they need to be efficient at what is done. Most of us would agree to cut overhead costs with energy efficiency versus headcount cuts. It’s better for the environment, better for the business, and benefits everyone.

Enterprise companies have been ‘going green’ for a while now. Initiatives like Climate Savers, LessWatts.org and others have been pushing the technical envelope on how to reduce power usage for businesses large and small. Intel has a large hand in contributing to the conservative ecology in power usage. People need computers, computers need power, and power is used – but can the power be reduced and still give the same experience? With the Xeon 5500 Series platforms – the answer is a resounding - YES!

You’ve most likely read about the performance stats around Intel Xeon 5500 (Nehalem) Processors, but I want to show you how their efficiency can give back to your enterprise. Not only do the Xeon 5500 Series give you power and efficiency – there is more technology ‘under the hood’ to be looked at. Intel Intelligent Power Node Manager has been released in the Xeon 5520 and 5500 Chipsets (previously called Tylersburg-EP).

There are several scenarios we can go through concerning server workloads – but let’s take a real world example of Company “X” (I can’t tell you who right now) but their workloads have been very stable and growing over the past few years. One of the issues is that their servers are heavily worked during the beginning of the week, pushing the server farm at 85-90% utilization. The work is reduced gradually over the week and by Friday; the server utilization is around 10-20%.

As you can see in the chart above, the server farm would start out the week with 85-90% load and the servers would run at full power the entire week. This would burn energy at 90% cost, even though the workload had died off toward the end of the week to about 15% - not very efficient. It’s like leaving the stove on all day long, and only using it for a few minutes when you cooked a meal.

Once we brought in the Xeon 5500 based systems, we also enabled ACPI power management which is much more pronounced with the Xeon 5500 because of the increased number of P-states. We are able to add power capping using Node Manager to limit the power usage by the racks to meet the daily requirements of the customer workloads. This helped us to have the power available when needed, and reduce the power when the workloads aren’t as power hungry.

Another key feature that we’re going after is to increase the server density using Node Manager and Intel Datacenter Manager by measuring the power usage and capping the maximum power utilized by the entire rack. The benefit is that with Intel Intelligent Power Node Manager, the data comes in real-time and we can modify the power curves on a regular basis through the server console.

Like many customers, power savings can have a large impact in your overhead costs. If this sounds like a solution that your company could benefit from, then definitely ask your favorite OEM when their Node Manager enabled Xeon 5500 series platform will be available. Intel Server Products are available today and ready for your datacenter – the ROI is estimated to be a short 8 months, so a little green goes a long way.

Driving down server complexity with Intel manageability

todd.christ@intel.com — Fri, 20 Feb 2009 22:49:15 GMT

Let’s face it; it’s getting harder to measure server density in rack units, and measuring by compute threads in a rack isn’t getting any easier with the core/thread counts increasing year over year. I still remember from 12 years ago when Intel was acquiring companies who were really good at piecing together single core multi-processor systems and those systems were literally hanging from engine hoists (for demo purposes) because they were so large… I believe they had eight Intel Pentium Pro processors and 128MB of RAM. In comparison - today’s netbooks have more 4 times that amount of memory, in a base configuration.

Modern server micro-architectures have such a large increase in transistors alone, that it’s hard to equate the exponential growth in the complexity of the systems. While power must still be consumed, the same amount of power can be distributed across several cores and platforms now - which is more power efficient, but it also adds more complexity as the number of nodes increase. But just because you have more nodes, doesn’t mean that you can’t manage their efficiency.

David Ott (from the Intel Software Services Group) presents many of the provisioning/power/manageability problems at hand in the video below (5m16s), and explains how Intel is providing the 'touch points' to manage server platforms:

With the upcoming Intel Xeon 5500 Series Processors, not only do you have a high-performing platform; and in Intel fashion they’re also more power-efficient. With the capabilities to self-throttle power usage via managed P-states per node or be managed via policies by group, time, etc. Managing for servers isn’t new, but the way that Intel is doing it is a huge leap ahead in manageability at the node level.

So I ask:

What manageability tools are you using for your enterprise servers today?
Is Intel Node Manager on your (or your OEM's) roadmap to gather information on a ‘per server’ basis?
Would more discrete information enable you to run your datacenter more efficiently?
What manageability items do you struggle within your own datacenter, and what would you like to see in future platforms?

If Power Manageability is new to you, I highly suggest you check out Intel Dynamic Power Datacenter Manger, and if you're running a Linux based server - please check out http://www.lesswatts.org to ensure you have the latest ACPI compliant kernel.

And as a fun exit, here’s a video that we shot in one of our labs – further strengthening the need for virtualization

(and more importantly – the need for virtualized networks!)

Dialing in your Datacenter - using Intel Dynamic Power Datacenter Manager

todd.christ@intel.com — Thu, 28 Aug 2008 16:57:51 GMT

Everyone is talking “green-energy” and “power-efficiency” these days. Reducing carbon footprint, renewable energy, CFLs, solar power, biking instead of driving, etc… the list goes on forever. Many people are excited to do something to change power consumption, but as a server administrator - are the proper tools in place?

Many of you have probably experienced the power/efficiency example at home. When the summer gets hot - many of us run to the thermostat and set it accordingly. When it's REALLY hot outside, we tend to twist the dial cooler - knowing all along, that our electric bill will most likely be higher at the end of the billing cycle. So, what do we do?

Some of us just live with the higher bills, some of us turn off the A/C and struggle in the heat - but I'd hope that most of us set the thermostat to a 'livable' temperature - it may not be the coolest, but it's enough to do the job and keep the electricity bills at a more moderate level - in a sense, it's a happy medium. In today's modern age, thermostats are programmable - taking a lot of the guesswork out of our hands and automating many of the old day-to-day temperature functions that our parents had to follow... Intel server platforms are evolving in this realm as well!

As a server admin, do you have the tools and technologies to reduce power consumption? There are several avenues addressing this issue, and I suggest reading the post from Lori Wigle on http://communities.intel.com/openport/community/openportit/server/blog/2007/11/14/data-center-efficiency. The datacenter is different from the desktop… server admins aren’t likely to enable sleep states to save energy – but rather, increase utilization on fewer servers to maximize your performance output in relation to your server footprint.

When was the last time you looked at your server’s power footprint? Do you even know how much power you’re using? Some of you may have some power meters and can monitor a server (or a few servers) at a time… but how many of you can monitor a rack or servers or a datacenter?

What if this capability was built into your current generation Xeon server platform? The good news is that modern processors DO have power management capabilities. Based on the ACPI specs:

P0 Performance State

While a device or processor is in this state, it uses its maximum performance capability and may consume maximum power. Thereby the processor uses it's maximum power allocation.

P1 Performance State

In this performance power state, the performance capability of a device or processor is limited below its maximum and consumes less than maximum power.

Pn Performance State

In this performance state, the performance capability of a device or processor is at its minimum level and consumes minimal power while remaining in an active state. State n is a maximum number and is processor or device dependent. Processors and devices may define support for an arbitrary number of performance states not to exceed 16.

Each Pn State is a "notch" in the processor's performance powerband (as seen below)

As these performance notches are set, the processor will lower it's power envelope and reduce the power needed in order to save energy. Just as a note, EIST must be enabled in the BIOS for this performance enhancement to work on your platform.

If you attended Intel’s IDF (Intel Developer Forum) you may have run into a few demos in regards to Datacenter Power Management, my booth showcased 4 current generation Intel Servers based on Bensley/Starlake Xeon DP boards and Xeon 54xx Series (codename Harpertown) Processors.

Here’s a quick video showcasing the demo – and just a note - we’ll be redoing this in a higher-quality format soon – so stay tuned!

Hopefully if you’ve watched the video – you’ve got some questions! The good news is that we have a new website from the Intel Software Network that is focused on Intel® Dynamic Power Datacenter Manager. The site lists the features, system requirements, downloads, and FAQ to get you started!

I’m looking forward to your feedback and questions!

Performance Benchmarking 101

todd.christ@intel.com — Tue, 27 Nov 2007 00:15:00 GMT

Take a look at the chart below ... it's telling you something... isn't it?

It's more than performance numbers and marketing, it's data... REAL data!

But what does it mean - and ultimately - how can you relate to it?

If you're really into high-powered computing, you're probably quite familiar with common benchmark data. With every new CPU release, there are tons of new statistics, models, and ways to test the increased performance of the newer technology device - in this case, the 45nm based CPUs just recently launched this month. But what exactly does all this data amount to? Reading benchmarks is more than just seeing a bar chart - there's a science to digging into the data...

First, lets take a step back for some of you who may not fully understand what benchmarking is for. Benchmarks help to provide a common ground for comparing the performance of various systems across different CPU/system architectures. A common set of instructions (or programs) are setup to run within a regulated guideline to ensure the testing is performed equally across the competing platforms or architectures. Very much like in sports, if you have two different runners - they run the same path - i.e. the 100 yard dash. This creates the comparative benchmark.

So let's get back to the latest hot stuff - the Intel Xeon 5400 Series and Core 2 Extreme QX9650 Quad Core based processors. In the past 18 months, computing models have taken a giant leap forward by adding more CPU's per socket thereby increasing the thread density of your platform. In dual socket systems, you used to have two threads you now have four or even eight! And in quad socket systems the count can go up to 16! You're increasing your capacity to perform computational data by a factor of 3 or 4 depending on the platform. This has made a tremendous change in how benchmarks have had to be setup to run and we have to evaluate the testing methods to ensure we're maximizing the computability of each platform.

There are a few key steps to take before you consider benchmarking your system:

identify your problem area (processing power, network bandwidth, memory utilization, etc)
identify your competing products
evaluate the 'leaders' in your problem area
survey for available benchmarking tools
evaluate 'best practices' for testing (e.g. lower idle power based processors won't really help much if you're only doing high-end computing)
and then - implement your findings in your chosen architecture(s)

In the high-end server space you usually see more vendor specific data rather than end-user testing. Primarily because of the finite set of data that server administrators are looking for. Many of these 'industry standards' are monitored for efficiency and ensure the end-user that the testing was properly performed and the results are repeatable:

Industry Standard Benchmarks

Standard Performance Evaluation Corporation (SPEC)
Transaction Processing Performance Council (TPC)
BAPCoan industry consortium developing benchmarks for Windows personal computers
Synchromesh Computing benchmark tests
The Embedded Microprocessor Benchmark Consortium (EEMBC)

Intel uses many of these standards for benchmarking - as you can see here in the Xeon 5000 Series based Processors Benchmark Page

Even if you're a server admin, you most likely interact with clients for day to day performance as well. If you search the web for CPU benchmarks the most commonly viewed benchmarks are performed on the client side of computing, mainly because of a few factors:

clients are usually cheaper and more abundant to test with
visuals in client computing are usually more fun to watch than seeing SQL data fly across the screen (hey - just being honest here!)
and servers in general are built for more specific reasons, whether it's application, storage, modeling or other specialties

Many of you have probably heard of benchmark sites such as: Anandtech, Toms Hardware, FiringSquad, HardOCP and many others (respond with your favorites please!) Each of these sites use common tools/applications to benchmark the latest and greatest hardware against each other. Depending on what you're looking to do with your hardware really determines what/how you want to benchmark your system (or look for data reviews for your configuration). After all, a machine that can run the latest games at over 60 frames per second may not be the best SQL server for your datacenter - right?

If you're looking for quick 'brute force' computational tools to try your hand at CPU benchmarking, try something simple like BOINC, Super PI, or you can get more elaborate by using some methods as described by C-Net by using Cinebench, or SiSoftware Sandra. Once you've figured out some of the basics - and can repeat these simpler tests - you can jump into those Industry Standards and get into some serious work!

So in closing, there are so many variables to account for when looking to validate the performance of a given system. Processor speeds, I/O subsystem configuration, memory latencies, network bandwidth, power utilization, etc... the permutations are nearly endless. So you have to be diligent in initially addressing your key problem(s), and attack the solution in benchmarking using the best known methods. Also, when reading benchmark information BE SURE to read the configurations of the systems in question - are they truly comparable? are the components running at spec level or overclocked? Are the speed differences negligible, or substantial in real-world evaluation? And finally, focus on what's important to you and your computing requirements - after all, you need to be sure you've picked the correct system for your needs.

Bloggers video intro to The Server Room

todd.christ@intel.com — Thu, 08 Nov 2007 22:56:00 GMT

Welcome to The Server Room, we've put together some quick videos today for you in order to put a name to the face of some of the bloggers and get a more personal touch for our interaction. Arijit Bandyopadhyay, Nikhil Sharma and yours truly are found below... Enjoy! and BLOG AWAY!