The Server Room Blog

Dynamic Power Allocation for Server Groups

enrique.g.castro-leon@intel.com — Mon, 08 Jun 2009 00:35:50 GMT

The Intel(r) Dynamic Power Node Manager technology allows setting a power consumption target for a server under load as described in a previous article. This is useful for optimizing the number of servers in a rack when the rack is subject to a power budget.

Higher level software can use this capability to implement sophisticated power management schemes, especially schemes that involve server groups. The range of control authority for servers in the Nehalem generation is significant. The power consumption of a fully loaded server consuming 300 watts can be rolled back by roughly 100 watts. In virtualized utility computing environments additional control authority is possible by migrating the virtual machines out of a host and consolidating them into fewer host. The power consumption of the power capped host now at 200 watts, can be brought down by another 50 watts, to 150 watts.

The reader might ask about the possibility of constantly running servers in capped mode to save energy. Unfortunately capping entails a performance tradeoff. The dynamic is not unlike driving an automobile. The best mileage is obtained by running the vehicle at a 35 MPH constant speed. This is not practical in a freeway where the the prevailing speed is 60 MPH. The vehicle could be rear ended, or perhaps a more mundane motivation, the vehicle driver drives the vehicle at 60 MPH because she wants to get there sooner. Like a server, the lowest fuel consumption in a running vehicle, at least in gallons per hour, is attained when the vehicle is idling. No real work is done with an idling engine, but at least the vehicle can start moving in no time. Continuing with the analogy, turning a server off is equivalent to storing a car in the garage with the engine stopped.

This document provides an example of the performance tradeoff with power capping. Please look in page 5, Figure 2.

The following example illustrates how group power capping works. The plot is a screen capture of the Intel(r) Data Center Manager software managing the power consumption in a cluster of four servers. The four servers are divided in a cluster of two server sub-groups of two servers each, labeled low-priority and high-priority

The light blue band represents the focus of the plot. The focus can be changed with a simple mouse click. The current focus in the figure is the whole rack. Hence the power plot is the aggregated power for all four servers in a rack. If the high priority sub-group were selected, then the power shown would be the power consumed by the two servers in that sub-group. Finally, if a single server is selected, then the power indicated would be the power for that server only.

There are four lines represented in the graph. The top line is the plate power. It represents an upper bound for the server’s power consumption. For this particular group of servers the plate power is 2600 watts. The servers are identical, and hence rated at 2600 / 4 = 650 watts.

The next line down is the derated power. Most servers will not have every memory slot or every hard drive tray populated. The derated power is the data center’s operator guess about the upper bound for power consumption based on the actual configuration the server. The derated power is still a conservative guess, considerably higher than the actual power consumption of the server. As a rule of thumb, it is ~70% of the nameplate. The derated power has been set at 1820 watts for the rack or 455 watts per server.

Finally, the gold line represents the actual power consumed by the server. The dots represent successive samples taken from readings from the instrumented power supplies.

The servers are running at full power using the SPECpower benchmark. The rack is collectively consuming a little less than 1300 watts. At approximately 16:12 a policy is introduced to constrain power consumption to 1200 watts. DCM instructs individual nodes to reduce power consumption by lowering the set points for Node Manager in each node until the collective power consumption reaches the desired target.

When we instructed Data Center Manager to hold a power cap for the group rack (2), it makes an effort to maintain power at that level, in spite of unavoidable disturbances in the system.

The source of the disturbances can be internal or external. An internal disturbance can be the server fans switching to a different speed causing a power spike or dip. Workloads in servers go up and down, with a corresponding uptick or dip in the power consumption for that server. An external disturbance could be a change in the feed voltage or an operator action. In fact at T = 16:14 we introduced a severe disturbance: we brought the workload of the bottom server, epieg3urb07 down to idle.

Note that it takes a few seconds for Data Center Manager to react and to reach the original power level. Likewise, when the bottom server is brought to idle, it also pulled back the power consumption for the group. However, the group power went back to the target power consumption after a couple of minutes. If we look at the plot of the individual servers, we can see Data Center Manager at work maintaining the target power.

The figure above captures the behaviors of the individual servers. Note how DCM allocates power to individual nodes yet it maintains a global power cap. When the server at the bottom is suddenly idled, there is a temporary dip in power server consumption for the group, but it soon recovers to the target capped level. Also note that the power not used by the bottom server is reallocated to the remaining three nodes until they get close to the previously unconstrained level.

Top 10 Reasons Why Nehalem is Ideal for the Cloud

kevin.j.huiskes@intel.com — Tue, 31 Mar 2009 15:24:54 GMT

Yesterday – Intel officially launched the Intel® Xeon® 5500 processor (formerly codenamed “Nehalem”) for servers and workstations. One of the most exciting uses of this new platform will be as a key building block in cloud computing infrastructure. Whether you’ve bought into the hype of cloud computing or are a jaded IT realist – you can’t afford to pass up this list of 10 reasons the Intel Xeon 5500 processor is perfect for the cloud.

Efficiency. To get the greatest efficiency – the leaders of large-scale Internet providers place their datacenters next to hydroelectric power or other low-cost energy sources. Each watt saved flows straight to the bottom line. Similarly – cloud computing companies intensely scrutinize their server purchases – weighing some variation of this question: how much performance (and by extension, revenue) can I squeeze out of the equipment – versus the cost of procurement and operations. This is the essence of “efficiency”. And now – with Intel’s new Xeon 5500 processor – there’s great news for anyone building efficient cloud infrastructure. The Xeon 5500 can deliver up to 2.25X the computing performance at a similar system power envelope compared to Intel’s previous generation Xeon 5400 series¹. (By the way – the Xeon 5400 is no efficiency slouch – as it’s been leading the industry-standard SpecPower results for two socket systems since the benchmark was created.²) Need more evidence of Xeon 5500 efficiency? Look no further than the amazing results announced by IBM – a score of 1860, which is a 64% leap over the previous high score for a two socket system.³ Results like this clearly demonstrate that the Xeon 5500 has the extremely efficient performance that cloud operators are seeking.
Virtualization performance. If a cloud service provider has leveraged a virtualization layer in its architecture - the performance of virtual machines and the ratio of VMs to servers are key concerns. Enter the Xeon 5500 which boasts a stellar jump in virtualization performance, up to 2 times the previous generation Xeon 5400 series⁴ allowing virtualized clouds to squeeze even more capability out of their infrastructure.
Adaptable. Cloud computing environments tend to be highly dynamic as usage ebbs and flows during the day, some applications scale rapidly while some shut down, and so on. To meet such shifting demand – it’s critical to have adaptable cloud building blocks. And here Intel’s Xeon 5500 shines: this processor has unique new intelligence to increase performance when needed (Intel Turbo Boost) and to reduce power consumption when demand falls (Intel Intelligent Power Management Technology).
Designed for higher operating temperatures. Across the datacenter industry – there’s growing interest in the notion of running datacenters at warmer temperatures to conserve energy. For cloud computing mega-datacenters, this concept has been in practice for several years. But it’s not just the datacenter staff that needs to handle warmer climates - the equipment must tolerate the conditions as well. Intel’s Xeon 5500 has been designed to run at higher temperatures providing one more piece of the puzzle to enable more efficient cloud infrastructure environments⁵.
50% lower idle power. Cloud computing providers – like airlines and phone companies – need to run at the highest utilization possible to maintain a healthy P&L. Yet there are times when usage – and thus server utilization – drops and at these times, cloud service providers desire processors with low power consumption. The Xeon 5500 processor now boasts an idle power that’s up to 50% lower than the prior generation systems, reducing energy costs⁶.
Advanced power management. Intel has incorporated special platform level power technologies into the Xeon 5500 platform – which open new avenues to managing server energy consumption beyond what’s already built into the processor. Intel Intelligent Power Node Manager is a power control policy engine that dynamically adjusts platform power to achieve the optimum performance-power ratio for each server. By setting user-defined platform energy policies – Node Manager can enable datacenter operators to increase server rack density while staying within a given power threshold. While results vary based on the type of application and server – Intel demonstrated up to 20% improvement in rack density by using Node Manager in a recent proof-of-concept with Baidu, a leading search engine⁷.
High Performance Memory Architecture. Cloud computing and other highly scalable Internet services are often relying on workloads where it makes more sense to keep large volumes of memory in DRAM, close to the CPU, rather than on slower, more distant hard drives. “Memcached” – a distributed caching system used by many leading Internet companies – is but one example. The Intel Xeon 5500 offers several exciting memory architecture benefits over the previous generation: (1) Up to 3.5X the memory bandwidth⁸ by leveraging an integrated memory controller and Intel Quick Path Interconnect (QPI), (2) supports a larger memory footprint (144GB versus 128GB), and (3) DIMMs and QPI links automatically move to lower power states when not active. In these new caching and distributed workloads, where large memory architectures are crucial, the Intel Xeon 5500 offers real advantages.
Perfect when paired with SSDs. Few technologies get datacenter gurus more excited than solid state drives – which can offer impressive performance gains over their rotating hard drive cousins at far lower energy consumption. But with SSDs that can read 1000 times more data into the CPU versus a HDD – you want a ravenous processing beast to handle the traffic. And – you’re catching on to the blog theme – the Xeon 5500 can provide up to 72% better performance using SSDs than even the previous generation Xeon systems⁹. Intel Xeon 5500 is truly a perfect engine to complement SSDs.
Ideal for optimized server boards. For cloud infrastructure – where every watt is a pernicious tax – you need more than just an extremely efficient processor such as the Xeon 5500. You also need an optimized server platform that has been stripped of every unneeded feature, configured with world-class energy efficient components, and designed for reduced airflow that minimizes the use of fans. One such product is an Intel server motherboard – codenamed “Willowbrook” which has an impressively low idle power below 70W, considering it’s a dual Xeon 5500 performance rocket¹⁰.
A competitive lever for cloud operators. Lastly, for a service provider scaling out its infrastructure – systems based on Intel Xeon 5500 processors could offer a competitive advantage versus service providers whose servers are 2 to 3 years old. Because of the performance leaps in Intel server processors in the past few generations – Intel Xeon 5500 based servers can handle the same performance load as up to three times the number of 3-year old dual core servers¹¹. The benefit is clear: providing the same performance level but with far fewer servers means a leg-up on those service providers with more antiquated, less efficient infrastructure.

If you have made it through this lengthy top 10 list – you should have a better sense for the advantages of Intel’s latest processor for cloud computing environments. Of course, the best way to really see the benefits is to get an Intel Xeon 5500 based system from your preferred vendor and test with your own code.

^{1 - 11}For Footnotes, Performance Background, and Legal information, please refer to the attached document.

Driving down server complexity with Intel manageability

todd.christ@intel.com — Fri, 20 Feb 2009 22:49:15 GMT

Let’s face it; it’s getting harder to measure server density in rack units, and measuring by compute threads in a rack isn’t getting any easier with the core/thread counts increasing year over year. I still remember from 12 years ago when Intel was acquiring companies who were really good at piecing together single core multi-processor systems and those systems were literally hanging from engine hoists (for demo purposes) because they were so large… I believe they had eight Intel Pentium Pro processors and 128MB of RAM. In comparison - today’s netbooks have more 4 times that amount of memory, in a base configuration.

Modern server micro-architectures have such a large increase in transistors alone, that it’s hard to equate the exponential growth in the complexity of the systems. While power must still be consumed, the same amount of power can be distributed across several cores and platforms now - which is more power efficient, but it also adds more complexity as the number of nodes increase. But just because you have more nodes, doesn’t mean that you can’t manage their efficiency.

David Ott (from the Intel Software Services Group) presents many of the provisioning/power/manageability problems at hand in the video below (5m16s), and explains how Intel is providing the 'touch points' to manage server platforms:

With the upcoming Intel Xeon 5500 Series Processors, not only do you have a high-performing platform; and in Intel fashion they’re also more power-efficient. With the capabilities to self-throttle power usage via managed P-states per node or be managed via policies by group, time, etc. Managing for servers isn’t new, but the way that Intel is doing it is a huge leap ahead in manageability at the node level.

So I ask:

What manageability tools are you using for your enterprise servers today?
Is Intel Node Manager on your (or your OEM's) roadmap to gather information on a ‘per server’ basis?
Would more discrete information enable you to run your datacenter more efficiently?
What manageability items do you struggle within your own datacenter, and what would you like to see in future platforms?

If Power Manageability is new to you, I highly suggest you check out Intel Dynamic Power Datacenter Manger, and if you're running a Linux based server - please check out http://www.lesswatts.org to ensure you have the latest ACPI compliant kernel.

And as a fun exit, here’s a video that we shot in one of our labs – further strengthening the need for virtualization

(and more importantly – the need for virtualized networks!)

Server Instrumentation and Power Capping - solving power & cooling issues in the data center

david.e.jenkins@intel.com — Wed, 18 Feb 2009 14:05:08 GMT

So are you among the approximately 40% of data center managers that are projected to run out of power or cooling capacity in the next 12-24¹ months and need new options to deal with ever increasing demand for compute capacity? In my discussions with IT professionals, it’s clear that a “business as usual” approach to the design and operation of the data center is no longer sufficient.

In the coming weeks, you will see a number of bloggers write about using Intel Xeon Processor 5500 (Nehalem) servers to refresh the data center – a concept first discussed on this site back in late 2007 - to more efficiently use limited power, cooling and floor space resources in the data center. Today, I want to touch on another means of addressing these issues at hand - using instrumentation as a source of data and controls to better monitor and manage the data center.

Individual pieces of the data & control picture have steadily come into the mainstream via instrumentation of individual server components. Think processors that allow power & frequency to be modulated. Power Supplies that report system level power consumption. Memory that reports its temperature. Fans that can scale RPMs and power to the actual air flow requirements. Really cool capabilities, but these somewhat fragmented sources of data and control don’t provide the capability to manage at the rack or data center level. The challenge at hand is to take all of these individual points of component instrumentation and develop system and data center level capabilities – what I call extended instrumentation – to provide unique and innovative tools that data center managers need.

One of the more exciting extended instrumentation capabilities that has evolved is power capping. Power limits or caps defined and communicated by console management software are enforced by system level functionality, enabling the ability to limit system power in a dynamic fashion. Applications of the use of power capping range from increasing performance density to temporarily shedding compute load to ride through power or thermal events in the datacenter to enabling power based dynamic resource balancing. Power Capping gives IT managers a tool to squeeze additional compute performance out of their existing data center – making more efficient use of their limited and valuable power, cooling and floor space resources to lower costs, improve availability and extend the life of the current data center.

Are you evaluating this capability? Are you using it already? I’m interested in discussing your thoughts on instrumentation and power capping.

1. http://www.infoworld.com/article/08/03/26/Datacenters-heading-for-cash-crunch_1.html

Dialing in your Datacenter - using Intel Dynamic Power Datacenter Manager

todd.christ@intel.com — Thu, 28 Aug 2008 16:57:51 GMT

Everyone is talking “green-energy” and “power-efficiency” these days. Reducing carbon footprint, renewable energy, CFLs, solar power, biking instead of driving, etc… the list goes on forever. Many people are excited to do something to change power consumption, but as a server administrator - are the proper tools in place?

Many of you have probably experienced the power/efficiency example at home. When the summer gets hot - many of us run to the thermostat and set it accordingly. When it's REALLY hot outside, we tend to twist the dial cooler - knowing all along, that our electric bill will most likely be higher at the end of the billing cycle. So, what do we do?

Some of us just live with the higher bills, some of us turn off the A/C and struggle in the heat - but I'd hope that most of us set the thermostat to a 'livable' temperature - it may not be the coolest, but it's enough to do the job and keep the electricity bills at a more moderate level - in a sense, it's a happy medium. In today's modern age, thermostats are programmable - taking a lot of the guesswork out of our hands and automating many of the old day-to-day temperature functions that our parents had to follow... Intel server platforms are evolving in this realm as well!

As a server admin, do you have the tools and technologies to reduce power consumption? There are several avenues addressing this issue, and I suggest reading the post from Lori Wigle on http://communities.intel.com/openport/community/openportit/server/blog/2007/11/14/data-center-efficiency. The datacenter is different from the desktop… server admins aren’t likely to enable sleep states to save energy – but rather, increase utilization on fewer servers to maximize your performance output in relation to your server footprint.

When was the last time you looked at your server’s power footprint? Do you even know how much power you’re using? Some of you may have some power meters and can monitor a server (or a few servers) at a time… but how many of you can monitor a rack or servers or a datacenter?

What if this capability was built into your current generation Xeon server platform? The good news is that modern processors DO have power management capabilities. Based on the ACPI specs:

P0 Performance State

While a device or processor is in this state, it uses its maximum performance capability and may consume maximum power. Thereby the processor uses it's maximum power allocation.

P1 Performance State

In this performance power state, the performance capability of a device or processor is limited below its maximum and consumes less than maximum power.

Pn Performance State

In this performance state, the performance capability of a device or processor is at its minimum level and consumes minimal power while remaining in an active state. State n is a maximum number and is processor or device dependent. Processors and devices may define support for an arbitrary number of performance states not to exceed 16.

Each Pn State is a "notch" in the processor's performance powerband (as seen below)

As these performance notches are set, the processor will lower it's power envelope and reduce the power needed in order to save energy. Just as a note, EIST must be enabled in the BIOS for this performance enhancement to work on your platform.

If you attended Intel’s IDF (Intel Developer Forum) you may have run into a few demos in regards to Datacenter Power Management, my booth showcased 4 current generation Intel Servers based on Bensley/Starlake Xeon DP boards and Xeon 54xx Series (codename Harpertown) Processors.

Here’s a quick video showcasing the demo – and just a note - we’ll be redoing this in a higher-quality format soon – so stay tuned!

Hopefully if you’ve watched the video – you’ve got some questions! The good news is that we have a new website from the Intel Software Network that is focused on Intel® Dynamic Power Datacenter Manager. The site lists the features, system requirements, downloads, and FAQ to get you started!

I’m looking forward to your feedback and questions!