Intel Intelligent Power Node Manager is a new technology that is available with the Xeon 5500 Series Platforms released earlier this year.  Many of you have asked me questions via Twitter (@Toadster) about "How can I use Node Manager?" - so I wanted to present some simple use cases to simplify the explanation of Node Manager and how you can best use the technology in your own enterprise.

First of all, let's explain the growth problem at hand.  As servers shrink in size, the density of each server 'footprint' is growing from a power perspective... a few years ago, a single 42U rack could hold about 21 servers (estimating 2U servers) - and usually hosting one or two apps/servers per physical server, depending on if you had single or dual-socket servers.  In modern datacenters, that same 42U rack can hold 42 servers (1U each) with 2P per server - so you have an immediate density increase of 2X the # of servers, and 2-4X the number of sockets - which can equate to 16X the number  processor threads per rack...  one good thing is that Intel has been developing newer technologies to keep the TDP of each CPU roughly the same over the same time period between processor updates... where you used to have 2 or 4 cores, you now have 8 to 16 cores at the same thermal envelope!

Knowing how much power your platform uses is a key factor in populating racks and rows in your datacenter.  Prior to Node Manager technology, most Datacenter Managers would base rack population on 'nameplate' power - or the (W) rating on your power supply.  That's the 'max' power utilized by the platform, and what the PSU is rated for (worst case).  See the image below...

As you can see - using Intel Intelligent Power Node Manager technology, you can view your system's power utilization in real-time using Intel Datacenter Manager and the administrator can implement the power caps to ensure your server rack stays within your required power limits.  By utilizing the 'actual' power limits instead of nameplate power, you can increase your rack density thereby increasing your ROI, and decrease your TCO!  Lets face it - everyone loves saving money!

Many of us are familiar with this next scenario... it's summertime, and the power company is announcing that the power grid is under strain.  Personal homes start having their A/C cut-off to save the power grid from brown-outs...  now your enterprise can help reduce those risks as well!

Over the next few weeks, I hope to post more blogs/videos:

1. Single Node Power Monitoring & Management
2. Group/Rack Power Monitoring & Management
3. Thermal Monitoring & Management

Please provide some feedback, and post your questions and ideas for upcoming blogs!

I wrote a while back about how the Xeon 7400(Dunnington) processor series compared to RISC. Since then I have shared information through other blog posts and sharing content about how Xeon 7400 and Xeon 5500 will compare to both SPARC and POWER.

Xeon 7400 and Xeon 5500 are the current products shipping into the marketplace today. I.M.H.O they offer a pretty compelling alternative from both a performance and TCO perspective Vs SPARC and POWER. But I will not try and repeat all the reasons here

What I wanted to share with you was some thoughts about what the next product to succeed Xeon 7400 will bring to the RISC party. Nehalem-EX is the code-name for our next generation of product designed to serve workloads currently serviced by Xeon 7400 today (i.e. Database, ERP,  BI etc). EX btw is what we all would traditionally call MP or multi processor servers

Don't stop reading now, here is why I'm EXCITED about what Nehalem-EX will bring to the RISC party.

My excitement is actually based on real customer discussions about what Nehalem-EX will do for them and why it delivers some new stuff (my code for features and benefits) which they see as a pre-requisite to make the move from RISC to Xeon. For some customers the TCO and performance of  products have been enough to convince them to move. For some other customers there are still some checkboxes remaining which I believe Nehalem-EX will address

Here is a snapshot of some of the cool new stuff which is actually convincing customers (from some real deals that I have worked)

1. Improved bandwidth. Up to 9 times memory bandwidth of previous generations
2. Introduction of Quickpath Interconnects to the EX systems
3. Add new RAS features previously seen on Itanium products to Xeon products
4. Significant improvement in performance vs previous generations e.g. Database 2.5xe
5. More scalable platforms through 8 OEMs offering >8S. These platforms are key to manage large databases and for large scale consolidation
6. Mainframe class availability in scalable platforms

For more information check out the press briefing from May. See more the details in the presentation

Nehalem-EX goes into production later this year and I am pretty excited about how it will change the game. What do you think?

Did you know that using an electricity rate of 11.4 cents per kWh provides a simple method of calculating annual electricity cost of any device?

1 watt of power consumption, with an electricity rate of 11.4 cents per kWh, cost $1 per year, assuming power usage remains constant.  Also, as a general rule of thumb, every 1W of device power consumption in a data center requires an additional 1 watt for overhead power (Source: Intel IT). So a device that consumes 1W actually consumes 2W of power at a data center level.

Here's the math:  1 Watt power * 8760 hours per year / 1000 * $0.114 electricity rate per kWh = $1 per year.  This math holds the same for any currency, Euro, Yuan, etc.  11.4 cents per kWh is the crossover point…and as electricity rates increase over 11.4 cents, 1 watt will cost more than a $1 per year. 

The datacenter overhead power, often referred to as Power Usage Effectiveness (PUE) is a number which has emerged as the leading metric for data center energy efficiency.

You might say that 1W = $2 annually doesn't sound like much, but start doing the math for 1000 servers that consume 200W in a data center with a PUE of 2.0 which works out to annual electricity cost of ~$400,000 per year.  Now, for every 1 watt the server power consumption is reduced, this would translate into $2000 annual savings.  Note, this is a very rudimentary example, but it is useful to illustrate why customers are really starting to focus on power as one of their key purchase decisions.

If you need energy efficient servers, there are multiple server vendors currently have some exceptional energy efficient products based on Intel(R) Xeon(R) 5500 processors.  And looking forward, we are also actively working on how to reduce power of the processor and at the system level for the upcoming generations of products.

Here’s some good reference on electricity rates:

For United States, state by state electricity rate comparison. 

For Europe, 1st half of 2008 rate comparison by country.

Remember, power is one purchase decision, but it is not the only one.  A rack of servers that consumes less power that does less work isn't an efficient way of deploying servers either.  Ensure that the performance vector is considered.  Intel® Xeon® 5500 processor based servers provid exceptional performance and perf/watt leadership over the competition.

Quick question for you:  How does electricity rate of 11.4 cents per kWh and a data center PUE of 2.0 compare to your data center? 

Non-x86 RISC architectures, Power or SPARC, have been used in high end business critical virtualization solutions for a long while now. These come with a vertical stack of solution including the hardware, software, manageability tools and services provided by one vendor. This often leads to lock-in to the proprietary virtualization solution and services, and can be expensive from an end user perspective.

There are reasons why companies that can afford RISC based solutions have subscribed to it. This has been mainly due to Reliability, Availability and Serviceability (RAS) features, scalability and dedicated resources for quality of service (QoS) and isolation.

The world of virtualization however has significantly changed in the last 5 years. x86 based hardware and software products today offer well accepted and high performance virtualization solution. With the eminent availability of highly scalable and resilient Nehalem-EX products with 16-threads per socket and extensive RAS capabilities in the near future, the line between an expensive RISC solution and x86 based virtualization solution could blur further.

From an end user’s perspective, Nehalem-EX could provide sufficient capabilities that they have come to expect out of a RISC based virtualization infrastructure. Looking at it:

• Hardware partitioning of Nehalem-EX platform would be possible. Along with this OS virtualization and full commercial hypervisor support for logical partitioning already exists on Xeon processors.
• Nehalem-EX hardware infrastructure allows software ecosystem to deliver capacity on demand. For example extra CPU capacity can be dynamically added as needed. Moreover VM migration and policy based load balancing capabilities that already exist in commercial hypervisors complement this and provides IT easy methods to manage capacity at the datacenter level.
• Memory can be dedicated by not oversubscribing the available physical memory.
• CPUs can be dedicated by creating CPU affinity.
• Dedicated I/O assignment is possible using VT for Directed I/O. It can also restrict DMA access from devices to certain areas in memory, increasing isolation and system reliability.
• Single Root IO Virtualization feature would be available as part of Intel VT for connectivity in the networking devices. This allows a single NIC to be shared amongst multiple VMs directly, while isolating the traffic from a NIC queue to a VM for better reliability. Per VM bandwidth allocation can also be supported.
• Nehalem EX adds virtualization feature that could help increase VM performance in a processor oversubscribed environment with high system utilization.
• Nehalem-EX will add new reliability, availability and serviceability (RAS) such as Machine Check Architecture (MCA) Recovery that allows error detection, error recovery and VM isolation.
• Inherent power technologies in the CPU, Turbo mode, and Dynamic Power Node Manager for system wide power capping all deliver IT the essential keys to balance power and performance.

While Nehalem-EX measures up to the infrastructure needs, it also enables horizontal solution that would allow customers to take advantage of best of breed software from the virtualization ecosystem thus reducing lock-in. This could result in faster innovation leading to an array of choices for business critical virtualization.

Based on http://www.itjungle.com/tfh/tfh042808-story03.html, a Power virtualization solution with Power6 based 4 Socket P550 box (~$93,000) and PowerVM Enterprise Edition for large system ($1,969 per core, with $220 per year on the maintenance) will totally cost an enterprise $109,000, just in one server acquisition.

While pricing of NHM-EX 4S system is not available, approximating a cost using current 4-Socket Intel server pricing and commercial VMM software would suggest that Intel based solution could cost at-least 50% less in just infrastructure. Other savings like not requiring specialized RISC based hardware, services, solution and staff would add to the lower cost of ownership in the long run.

Given the economy and Nehalem-EX features, would it not make sense to take RISC out of your investment?

The debate on how to best increase system capacity to accommodate growing applications has raged on for years; “scale up” with more CPU, memory, and I/O, or “scale out” with loosely connected systems.    Scaling out by adding networked systems to increase capacity has been a good economical solution for many IT managers because it allows them to grow by using less expensive, industry standard building blocks.  However, there are some notable exceptions to this line of thought.  One is that the class of applications that require shared memory and large database support are much better suited to run on a single, expandable system that scales up.  These are typically transaction processing, business intelligence and ERP solutions.   Until now, IT managers running applications that require scale-up systems larger than 4 or 8 CPUs have had limited platform choices and most were proprietary and expensive RISC-based servers.

The other problem with the scale out approach is the people, facilities, software and overhead costs and complexity of managing very large numbers of servers, which can grow to a point where the costs outweigh the performance and system cost benefits.  The industry solution to achieving better ROI has been to consolidate multiple scale-out servers onto single industry standard scale-up servers with virtualization solutions.  This is a good solution, but is limited by the number of application loads the IT manager feels comfortable placing on a single server, given the need to maintain peak performance and availability for each application.

Well, it looks like the scale-up, scale-out debate is about to take another turn.  In the server product update Intel gave on May 26th, they talked about new levels of system scalability and choice supported by the upcoming Nehalem-EX processor.  This processor will support systems that scale up to 8 sockets natively (shared memory, without any additional silicon), and up to 16 sockets and higher with node controllers from system manufactures that allow single systems to share memory beyond 8 sockets.   So far there are over 15 different designs from 8 OEMs that offer 8 socket or higher scalability.  But of course, for the class of application where scaling is important, socket count doesn’t tell the whole story of what’s needed for scalable performance.  Thread support, key for transaction processing and virtualization, scales at the rate of 16 threads per socket with 8 cores and Hyper Threading (2 threads per core).  That would be 128 threads for an 8-socket system, and 256 threads for 16 sockets.   And in order to keep those threads fed with data close to the CPU, each processor supports up to 24 MB of shared cache (1.5X current generation Xeon), and an impressive 16 memory slots per socket or 128 DIMMs on an 8-socket system.  In addition, the Scalable Memory Interconnect gives these systems 9 times the memory bandwidth of today’s top Xeon processor.  Finally, four QuickPath interconnect links per socket allow for high-bandwidth sharing of data across the system.

So the net of it is that the industry is going to see a broad selection of highly scalable, next-generation servers that significantly extend the economic advantage of industry standard scale-up solutions for business-critical, large database, and high-end virtualization/consolidation deployments.     I would expect these systems to give IT managers a very cost-effective alternative to the much more expensive and proprietary RISC-based servers they use today.

What are your thoughts?  Mike

Related Topics:

• NHM-EX Press Fact Sheet
• NHM-EX May 26th Press Briefing Video – condensed version
• IBM 8Socket Demo Video
• NHM-EX--A New Standard

When you’re planning a backpacking trip, whether it’s for several hours or several days, space is at premium.  Not only do you need to think about tents, sleeping bags, clothing, first aid, and navigational gear (among other things), but also how to keep yourself properly hydrated and fueled up.  Oh yeah, you have to figure out how to cram all of this gear into your pack…and carrying an additional pack is not an option!

Odds are you’ll be heading into the wilderness and won’t be able to re-supply for a while, so one of the limiting factors will be the amount of food you can carry.  Running out of fuel in the middle of nowhere makes for a potentially disastrous situation.

So let’s look at the nutritional numbers and how best to fuel the trip:

• Fats:  ~9 calories per gram, and typically found in nuts and oils
• Carbohydrates and proteins:  ~4 calories per gram, and typically found in sugars, grains, and meats

If you’re trying to maximize the number of calories you can carry in order to sustain you during your trip, you probably want to pack more foods with a higher fat content (such as peanut butter) than carbs or protein.  More calories per gram à more energy in your pack to get you where you want to go.

You can probably figure out where I’m going with this analogy – low power CPUs are all about helping maximize your performance per rack, just like packing foods with more calories per gram help deliver more energy in a limited amount of backpack space.

Depending on your specific rack power or overall datacenter power / cooling environment, low power SKUs might be a good fit to help maximize your performance per rack.  For the Intel® Xeon® 5500 series, there are two low power CPU options available, both spec’d at a 60W Thermal Design Point (TDP):  Xeon® L5506 (2.13 GHz) and the Xeon® 5520 (2.26 GHz).  These two SKUs have the same features as the corresponding Xeon® E5506 and E5520 SKUs, just lower in power. 

If you’re buying LV Xeon® 5400 CPUs today, such as the L5420, expect a big jump in performance per rack with the Xeon® L55xx SKUs due to lower overall system power and higher performance.  Similar story if you’re evaluating the Xeon® E5506 or E520 SKUs – same performance with L55xx SKUs with lower system power, so higher performance per rack.

Have questions – ask me on this blog or Ask An Expert in the Server Room.

I was thinking about what to write in my next blog and what I could share beyond what I have written previously about Intel Vs RISC in terms of TCO, performance and the customers that are choosing to move.

Luckily I didn't have to think too long on a Friday morning as a a topic came to mind instantly. There are numerous articles flying around this morning that picked up on the Oracle comments yesterday about how SPARC based systems compare to Intel. Thanks for providing me with an appropriate topic.

So in case you missed it, there was a question and answer session with Larry Ellison. When asked about SPARC, this was the reply "SPARC is much more energy efficient than Intel while delivering the same performance on a per socket basis. This is not a green issue, its an economic issue. Today, database centers are paying as much for electricity to run their computers as they pay to buy computers. SPARC machines are much less expensive to run than Intel machines"

1) SPARC more energy efficient than Intel?  Seriously, in what parallel universe does that exists?

SUN continues to use watts per thread as measure of energy efficiency. The recognized industry standard benchmark for measuring energy efficiency is SPECpowerand I don't see any SPARC based results in the 91 results published. The absence of a result certainly says something very clear to me - no story.

These UltraSPARCT2+ systems get loaded with a lot of memory to deliver the their results, so when you look at overall system power (what people care about) they are not as energy efficient as Intel based systems.

SPECpower is effectively based of SPECJbb-2005 so another way of loking at this is to look at the SPECJbb-2005 results for a 4 socket UltraSPARcT2+ system and a Xeon 7400 system. The 4s UltraSPARCT2+ delivers 693k BOPs while Xeon 7400 is 532kBOPs. So you conclude that SPARC is better than Xeon?. That would be the wrong conclusion

UltraSPARCT2+ system would consume 1525 watts Vs Xeon 7400 at 816 watts. If you look at BOPs per watt (another way of looking at energy efficiency and performance) then you would see that Xeon 7400 is 43% more energy efficient. Doing a similar comparison with Xeon 5400 (I haven't even talked about our latest Xeon 5500, Nehalem) would be up to 77% more efficient than UltraSPARCT2+.

And lastly before I forget to mention the 4s UltraSPARCT2+ had 128GB memory and costs over $150,000for the system, while Xeon 7400 based system had 64GB memory and costs around $32,000.

2) SPARC deliver same performance on a per socket basis?

2S Xeon 5500 has performance leadership over 2S UltaSPARCT2+ across a wide range of benchmarks. Up to 70% more performance and up to 60% lower system cost. 4S Xeon 7400 has price/performance leadership over 4S UltraSPARCT2+, UltraSPARCT2+ results achieved with system loaded with lots of memory that drives the cost up to 3-4Xthat of Xeon 7400 system

3) SPARC machine are less expensive to run?. I can't for the life of me work this one out!.

Hardware systems based on Intel have leading price/performance (read cheaper), lower energy needs (so electrivity bill lower) and any software product with a license per core strcuture is less expensive on Xeon system than an 8 core UltraSPARcT2+ (which also has higher multipler per core)

That's all for now folks. I just wanted to share some data on why I know that SPARC machines are much MORE expensive to run than Intel machines

The Intel XEON Processor 5500 is the new world record holder in >30 top performance benchmarks for 2-socket servers. Check out this video with Pat Gelsinger at the launch event in Santa Clara.

You can also check out all the performance results here: Server Performance Summary - Intel® Xeon® Processor