Prior to the Intel Xeon X5500 Server Platforms*, measuring server power was done via expensive equipment and could only be performed in a discrete fashion.  Unless you had tons of monitoring equipment to mash-up your power data - it was a tedious process.  Now, using Intel DCM and Node Manager - you can pull multiple servers worth of power info to make some important power decisions in your datacenter.

First of all, you need to baseline your workload.  If you're confident that you can replicate workload patterns then you've got a starting point.  Otherwise, it's usually a good idea to start monitoring and looking for some cyclical patterns and/or common data points (time, power, thermals, etc) to keep track of.

In this scenario (like in my last blog) we're using a SQL workload which can be modified to run the CPU at high levels for a relatively set amount of time.  The base workload runs for 7 min 30 seconds, as shown in the Intel DCM screencap below.

In this test case: Idle power for the 4 servers is 782W, and under load - the power increases to 1174W - which is a delta of 392W.  This power increase occurs when work is given to the server and the P/T states react to the workload and increase power/voltage to the system to increase performance.  Exactly what we've been used to seeing even since EIST was introduced several years ago.

Now, what I'll show you is something that may be very interesting in scale... I will power cap the servers by 20W each, and set the Intel DCM Power Policy to only allow 1095W for the 4 servers in the rack.

What is awesome here is that we can still finish the workload in the same 7 minutes 30 seconds.  So essentially, we have saved 80W of power for each set of 4 servers and still get the same amount of work completed!  In a large datacenter this can be HUGE in energy savings.

Let's do some quick math:  20W power savings per serer x 10,000 servers = 20kW power savings and you still get the work done.  I hope I just helped some of you server admins get some new ideas on your next "I need a raise" talk with your manager

*your mileage may vary, so test your own workloads and report out!

I'm always looking for good ways to describe to end-users what Intel Intelligent Power Node Manager can relate to everyday activities.  Over the weekend, I was helping a buddy of mine move to a new home, and of course we rented a truck.  While we were driving, we noticed a cool gauge on the dash and a pretty simple sticker describing what it does:

Keep it in the Green - what a simple concept!  Most everyone can relate to the gas pedal in your vehicle directly with gas mileage. If you have a lead-foot, you burn more gas.  But people who want to conserve, and keep it green - use cruise control.

Well, Intel servers can also be managed to optimize the energy consumed by the platform.  Power Optimzed servers using X5500 Series Processors (Nehalem) and the X5500 chipset in conjunction with Node Manager is like cruise-control - you set your "speed" and the servers keep that maximum speed.  It's all managed via P/T states using Intel Datacenter Manager.

Of course, at times the RED ZONE is needed - work needs to get DONE - so you throttle up, kick in the Turbo Boost and release that power cap!  But there are also times when all that energy isn't needed - so you lift your foot off the gas pedal, and set your speed for the work that needs to be done. Intel Xeon based servers can transition to higher/lower power states using technologies like EIST, DBS, and Node Manager.

Keep your eyes on the lookout for more data on Intel and server power management at the Intel Developer Forum 2009

Cloud Power Management with Intel® Microarchitecture (Nehalem) Processor-based Platforms

Check Twitter for more details @IDF and @IntelNews and search #IDF09

* disclaimer: giving credit where credit is due U-Haul owns that sticker and tagline!

I would like to elaborate on the topic energy vs. power management in my previous entry.

Upgrading the electrical power infrastructure to accommodate additional servers is not an option in most data centers today.  Landing additional servers at a facility that's working at the limit of thermal capacity leads to the formation of hot spots, this assuming that electrical capacity limits are not reached first with no room left in certain branch circuits.

There are two types of potentially useful figures of merit, one for power management and one for energy management.  A metric for power management allows us to track operational "goodness", making sure that power draw never exceeds limits imposed by the infrastructure.  The second metric tracks power saved over time, which is energy saved.  Energy not consumed goes directly to the bottom line of the data center operator.

To understand the dynamic between power and energy management let's look at the graph below and imagine a server without any power management mechanisms whatsoever.  The power consumed by that server would be P(unmanaged) regardless of any operating condition.  Most servers today have a number of mechanisms operating concurrently, and hence the actual power consumed at any given time t is P(actual)(t).  The difference P(unmanaged) - P(actual) is the power saved.  The power saved carried over time t(1) through t(2) yields the energy saved.

Please note that a mechanism that yields significant power savings may not necessarily yield high energy savings.  For instance, the application of Intel(r) Dynamic Power Node Manager (DPNM) can potentially bring power consumption by over 100 watts, from 300 watts at full load to 200 watts in a dual-socket 2U Nehalem server that we tested in our lab.  However, if DPNM is used as a guard rail mechanism, to limit power consumption if a certain threshold is violated, DPNM may never kick in, and hence energy savings will be zero for practical purposes.  The reason why we do this is because DPNM works best only under certain operating conditions, namely high loading factors, and because it works through frequency and voltage scaling, it brings a performance tradeoff.

Another useful figure of merit for power management is the dynamic range for power proportional computing.  Power consumption in servers today is a function of workload as depicted below:

The relationship is not always linear, but the figure illustrates the concept.  On the x-axis  we have the workload that can range from 0 to 1, that is, 0 to 100 percent.  P(baseline) is the power consumption at idle, and P(spread) is the power proportional computing dynamic range between P(baseline) and power consumption at 100 percent workload.  A low P(baseline) is better because it means a low power consumption at idle.  For a Nehalem-based server, P(baseline) is roughly 50 percent of power consumption at full utilization, which is remarkable, considering that it represents a 20 percent over the number we observed for the prior generation, Bensley-based servers.  The 50 percent figure is a number we have observed in our lab for a whole server, not just the CPU alone.

If a 50 percent P(baseline) looks outstanding, we can do even better for certain application environments such as load-balanced front end Web server pools and the implementation of cloud services through clustered, virtualized servers.  We can achieve this effect through the application of platooning.  For instance, consider a pool of 16 servers.  If the pools is idle, all the servers except one can be put to sleep.  The single idle server is consuming only half the power of a fully loaded server, consuming one half of one sixteenth of the cluster power.  The dormant servers still draw about 2 percent of full power.  Hence, after doing the math, the total power consumption for the cluster at idle will be about 8 percent of the full cluster power consumption.  Hence for a clustered deployment, the power dynamic range has been increased from 2:1 for a single server to about 12:1 for the cluster as a whole.

In the figure below note that each platoon is defined by the application of a specific technology or state within each  technology.  This way it is possible to optimize the system behavior around the particular operational limitations of the technology.  The graph below is a generalization of the platooning graph in the prior article.  For instance, a power capped server will impose certain performance limitations to workloads, and hence we assign non time critical workloads to that platoon.  By definition, an idling server cannot have any workloads; the moment a workload lands on it it's no longer idle, and its power consumption will rise.

The CPU is not running in any of the S-states than S0.  The selection of a specific state depends on how fast that particular server is needed online.  It takes longer to bring up a server online in the lower energy states.  Servers in G3 may actually be unracked and put in storage for seasonal equipment allocation.

A virtualized environment makes it easier to rebalance workloads across active (unconstrained and power capped) servers.  If servers are being used as a CPU cycle engines, it may be sufficient to idle or put to sleep the subset of servers not needed.

The extra dynamic power range comes at the expense of instituting additional processes and operational complexity.  However, please note that there are immediate benefits in power and energy management accrued through a simple equipment refresh.  IBM reports an 11X performance gain for Nehalem-based HS22 blade servers versus the HS20 model only three years old.  Network World reports a similar figure, a ten-fold increase in performance, not just ten percent.

I will be elaborating on some of these ideas at the PDCS003 Cloud Power Management with the Intel(r) Nehalem Platform class at the upcoming Intel Developer Forum in San Francisco on the week of September 20th.  Please consider yourself invited to join me if you are planning to attend this conference.

There are two technologies available to regulate power consumption in the recently introduced Nehalem servers using the Intel® Xeon® processor 5500 series.  The first is power proportional computing where power consumption varies in proportion to the processor utilization.  The second is Intel® Dynamic Power Node Manager (DPNM) technology which allows the setting of a target power consumption when a CPU is under load.  The power capping range increases with processor workload.

An immediate benefit of the Intel® Dynamic Node Manager (DPNM) technology is the capability to balance and trade off power consumption against performance in deployed Intel Nehalem generation servers.  Nehalem servers have a more aggressive implementation of power proportional computing where idle power consumption can be as small as 50 percent of the power under full load, down from about 70 percent in the prior (Bensley) generation.  Furthermore, the observed power capping range under full load when DPNM is applied can be as large as 100 watts out for a two-socket Nehalem server with the Urbanna baseboard observed in the lab to draw about 300 watts under full load.  The actual numbers you will obtain depend on the server configuration: memory, number of installed hard drives and the number and type of processors.

Does this mean that it will be possible to cut the electricity bills by one third to one half using DPNM?  This is a bit optimistic.  A typical use case for DPNM is as a "guard rail".  It is possible to set a target not to exceed for the power consumption of a server as shown in the figure below.  The red line in the figure represents the guard rail.  The white line represents the actual power demand as function of time; the dotted line represents the power consumption that would have existed without power management.

Enforcing this power cap brings operational flexibility: it is possible to deploy more servers to fit a limited power budget to prevent breakers from tripping or to use less electricity during peak demand periods.

There is a semantic distinction between energy management and power management.  Power management in the context of servers deployed at a data center refers to a capability to regulate the power consumption at a given instant.  Energy management refers to the accumulated power saved over a period of time.

The energy saved through the application of DPNM is represented by the area between the dotted line and the white graph line below; the power consumed by the server is represent by the area under the solid white graph line.  Since power capping is in effect during relatively short periods, and when in effect the area between the dotted line and the guard rail is relatively small, it follows that the energy saved through the application of DPNM is small.

One mechanism for achieving significant energy savings calls for dividing a group of servers running an application into pools or "platoons".  If servers are placed in a sleeping state (ACPI S5 sleep) during periods of low utilization it is possible to bring their power consumption to less than 5 percent of their peak power consumption, basically just the power needed to keep the network interface controller (NIC) listening for a wakeup signal.

As the workload diminishes, additional servers are moved into a sleeping state.  The process is reversible whereby servers are taken from the sleeping pool to an active state as workloads increase.  The number of pools can be adjusted depending on the application being run.  For instance, it is possible to define a third, intermediate pool of power capped servers to run lower priority workloads.  Capped servers will run slightly slower, depending on the type of workload. 

Implementing this scheme can be logistically complex.  Running the application in a virtualized environment can make it considerably easier because workloads in low use machines can be migrated and consolidated in the remaining machines.

We are conducting experiments to ***** the potential for energy savings.  Initial results indicate that these savings can be significant.  If you, dear reader have been working in this space, I'd be more than interested in learning about your experience.

If this topic is of interest to you, please join us at the Intel Development Forum in San Francisco at the Moscone Center on September 22-24.  I will be facilitating course PDCS003, "Cloud Power Management with the Intel(r) Xeon(r) 5500 Series Platform."  You will be the opportunity to talk with some of our fellow travelers in the process of developing power management solutions using Intel technology ingredients and get a feel of their early experience.  Also please make a note to visit booths #515, #710 and #712 to see demonstrations of early end-to-end solutions these folks have put together.

Back in the ‘dot-com’ days – many companies would build datacenters across the globe with one thing in mind – performance – and costs weren’t an issue.  It was all about getting the job done, with little concern about the costs.  Well, times have changed and companies have become more energy conscious, not only to become better stewards in using natural resources, but consumers are looking for companies who can design and develop products that can meet their own ‘green energy’ power needs.  It’s not as important anymore to make or build something to be the ‘best of class’ it also has to be ‘efficient’ while being the best. 

Corporate initiatives to reduce power but still “KTBR” (Keep the Business Running) are imperative to sustaining business today.  Not only do you need the best performing servers – but they need to be efficient at what is done.  Most of us would agree to cut overhead costs with energy efficiency versus headcount cuts.  It’s better for the environment, better for the business, and benefits everyone.

Enterprise companies have been ‘going green’ for a while now.  Initiatives like Climate Savers, LessWatts.org and others have been pushing the technical envelope on how to reduce power usage for businesses large and small.  Intel has a large hand in contributing to the conservative ecology in power usage.  People need computers, computers need power, and power is used – but can the power be reduced and still give the same experience?  With the Xeon 5500 Series platforms – the answer is a resounding - YES!

You’ve most likely read about the performance stats around Intel Xeon 5500 (Nehalem) Processors, but I want to show you how their efficiency can give back to your enterprise. Not only do the Xeon 5500 Series give you power and efficiency – there is more technology ‘under the hood’ to be looked at.  Intel Intelligent Power Node Manager has been released in the Xeon 5520 and 5500 Chipsets (previously called Tylersburg-EP). 

There are several scenarios we can go through concerning server workloads – but let’s take a real world example of Company “X” (I can’t tell you who right now) but their workloads have been very stable and growing over the past few years.  One of the issues is that their servers are heavily worked during the beginning of the week, pushing the server farm at 85-90% utilization.  The work is reduced gradually over the week and by Friday; the server utilization is around 10-20%. 

As you can see in the chart above, the server farm would start out the week with 85-90% load and the servers would run at full power the entire week. This would burn energy at 90% cost, even though the workload had died off toward the end of the week to about 15% - not very efficient.  It’s like leaving the stove on all day long, and only using it for a few minutes when you cooked a meal.

Once we brought in the Xeon 5500 based systems, we also enabled ACPI power management which is much more pronounced with the Xeon 5500 because of the increased number of P-states.  We are able to add power capping using Node Manager to limit the power usage by the racks to meet the daily requirements of the customer workloads.  This helped us to have the power available when needed, and reduce the power when the workloads aren’t as power hungry. 

Another key feature that we’re going after is to increase the server density using Node Manager and Intel Datacenter Manager by measuring the power usage and capping the maximum power utilized by the entire rack.  The benefit is that with Intel Intelligent Power Node Manager, the data comes in real-time and we can modify the power curves on a regular basis through the server console. 

Like many customers, power savings can have a large impact in your overhead costs.  If this sounds like a solution that your company could benefit from, then definitely ask your favorite OEM when their Node Manager enabled Xeon 5500 series platform will be available.  Intel Server Products are available today and ready for your datacenter – the ROI is estimated to be a short 8 months, so a little green goes a long way.