Every day in our personal lives, we’re bombarded with “opportunities” to get a better deal.  At the grocery store, we might be able to buy a single item for $2.50 or 3 for $5.00…which then forces us to go thru the mental gymnastics of figuring out how good of a deal it is, and whether or not we really need three 96 oz. bottles of salad dressing.

But there are some opportunities out there for adding a bunch of compute performance are a bit more straight-forward.

Case in point: Dell recently had Principled Technologies compare the performance for the Intel® Xeon® Processor E5520 and E5506 CPUs each running on a PowerEdge R710 server.  Both are 4 core processors, but the E5520 has many advantages over the E5506: 

• higher frequency (2.26 GHz vs. 2.13 GHz)
• faster QuickPath speeds (5.86 GT/s vs. 4.8 GT/s)
• faster memory support (1066 MHz vs. 800 MHz)
• Turbo Boost
• Hyper-Threading support.

Long story short:  Buying a slightly better processor with a server purchase can drastically increase your performance.  So if you are looking to buy a Dell PowerEdge server configured with Microsoft SQL Server 2008* and an Intel® Xeon® Processor E5506, for an additional $300 you can get up to 75% more performance by upgrading to an E5520 CPU.  More performance headroom in a similar power envelope, faster QuickPath and memory speeds, Hyper-Threading and Turbo Boost functionality – all for $300.  NOW THAT’S A GREAT VALUE!

Check out the summary document for the Dell R710 Principled Technologies performance testing, which also has comparative performance testing for the Xeon® E5540 and X5550 CPUs (also a great value for the money!), along with results for Microsoft Exchange.

NOTE:  System pricing from www.dell.com as of May 13, 2009.  Actual performance will vary based on configuration, usage and manufacturing variability. See the actual Principled Technology report in the following link for complete system configuration

The debate on how to best increase system capacity to accommodate growing applications has raged on for years; “scale up” with more CPU, memory, and I/O, or “scale out” with loosely connected systems.    Scaling out by adding networked systems to increase capacity has been a good economical solution for many IT managers because it allows them to grow by using less expensive, industry standard building blocks.  However, there are some notable exceptions to this line of thought.  One is that the class of applications that require shared memory and large database support are much better suited to run on a single, expandable system that scales up.  These are typically transaction processing, business intelligence and ERP solutions.   Until now, IT managers running applications that require scale-up systems larger than 4 or 8 CPUs have had limited platform choices and most were proprietary and expensive RISC-based servers.

The other problem with the scale out approach is the people, facilities, software and overhead costs and complexity of managing very large numbers of servers, which can grow to a point where the costs outweigh the performance and system cost benefits.  The industry solution to achieving better ROI has been to consolidate multiple scale-out servers onto single industry standard scale-up servers with virtualization solutions.  This is a good solution, but is limited by the number of application loads the IT manager feels comfortable placing on a single server, given the need to maintain peak performance and availability for each application.

Well, it looks like the scale-up, scale-out debate is about to take another turn.  In the server product update Intel gave on May 26th, they talked about new levels of system scalability and choice supported by the upcoming Nehalem-EX processor.  This processor will support systems that scale up to 8 sockets natively (shared memory, without any additional silicon), and up to 16 sockets and higher with node controllers from system manufactures that allow single systems to share memory beyond 8 sockets.   So far there are over 15 different designs from 8 OEMs that offer 8 socket or higher scalability.  But of course, for the class of application where scaling is important, socket count doesn’t tell the whole story of what’s needed for scalable performance.  Thread support, key for transaction processing and virtualization, scales at the rate of 16 threads per socket with 8 cores and Hyper Threading (2 threads per core).  That would be 128 threads for an 8-socket system, and 256 threads for 16 sockets.   And in order to keep those threads fed with data close to the CPU, each processor supports up to 24 MB of shared cache (1.5X current generation Xeon), and an impressive 16 memory slots per socket or 128 DIMMs on an 8-socket system.  In addition, the Scalable Memory Interconnect gives these systems 9 times the memory bandwidth of today’s top Xeon processor.  Finally, four QuickPath interconnect links per socket allow for high-bandwidth sharing of data across the system.

So the net of it is that the industry is going to see a broad selection of highly scalable, next-generation servers that significantly extend the economic advantage of industry standard scale-up solutions for business-critical, large database, and high-end virtualization/consolidation deployments.     I would expect these systems to give IT managers a very cost-effective alternative to the much more expensive and proprietary RISC-based servers they use today.

What are your thoughts?  Mike

Related Topics:

• NHM-EX Press Fact Sheet
• NHM-EX May 26th Press Briefing Video – condensed version
• IBM 8Socket Demo Video
• NHM-EX--A New Standard

The Intel(r) Dynamic Power Node Manager technology allows setting a power consumption target for a server under load as described in a previous article.  This is useful for optimizing the number of servers in a rack when the rack is subject to a power budget.

Higher level software can use this capability to implement sophisticated power management schemes, especially schemes that involve server groups.  The range of control authority for servers in the Nehalem generation is significant.  The power consumption of a fully loaded server consuming 300 watts can be rolled back by roughly 100 watts.  In virtualized utility computing environments additional control authority is possible by migrating the virtual machines out of a host and consolidating them into fewer host.  The power consumption of the power capped host now at 200 watts, can be brought down by another 50 watts, to 150 watts.

The reader might ask about the possibility of constantly running servers in capped mode to save energy.  Unfortunately capping entails a performance tradeoff.  The dynamic is not unlike driving an automobile.  The best mileage is obtained by running the vehicle at a 35 MPH constant speed.  This is not practical in a freeway where the the prevailing speed is 60 MPH.  The vehicle could be rear ended, or perhaps a more mundane motivation, the vehicle driver drives the vehicle at 60 MPH because she wants to get there sooner.  Like a server, the lowest fuel consumption in a running vehicle, at least in gallons per hour, is attained when the vehicle is idling.  No real work is done with an idling engine, but at least the vehicle can start moving in no time.  Continuing with the analogy, turning a server off is equivalent to storing a car in the garage with the engine stopped.

This document provides an example of the performance tradeoff with power capping.  Please look in page 5, Figure 2.

The following example illustrates how group power capping works.  The plot is a screen capture of the Intel(r) Data Center Manager software managing the power consumption in a cluster of four servers.  The four servers are divided in a cluster of two server sub-groups of two servers each, labeled low-priority and high-priority

The light blue band represents the focus of the plot. The focus can be changed with a simple mouse click.  The current focus in the figure is the whole rack.  Hence the power plot is the aggregated power for all four servers in a rack.  If the high priority sub-group were selected, then the power shown would be the power consumed by the two servers in that sub-group.  Finally, if a single server is selected, then the power indicated would be the power for that server only.

There are four lines represented in the graph.  The top line is the plate power.  It represents an upper bound for the server’s power consumption.  For this particular group of servers the plate power is 2600 watts.  The servers are identical, and hence rated at 2600 / 4 = 650 watts. 

The next line down is the derated power.  Most servers will not have every memory slot or every hard drive tray populated. The derated power is the data center’s operator guess about the upper bound for power consumption based on the actual configuration the server.  The derated power is still a conservative guess, considerably higher than the actual power consumption of the server. As a rule of thumb, it is ~70% of the nameplate. The derated power has been set at 1820 watts for the rack or 455 watts per server.

Finally, the gold line represents the actual power consumed by the server.  The dots represent successive samples taken from readings from the instrumented power supplies. 

The servers are running at full power using the SPECpower benchmark.  The rack is collectively consuming a little less than 1300 watts.  At approximately 16:12 a policy is introduced to constrain power consumption to 1200 watts.  DCM instructs individual nodes to reduce power consumption by lowering the set points for Node Manager in each node until the collective power consumption reaches the desired target.

When we instructed Data Center Manager to hold a power cap for the group rack (2), it makes an effort to maintain power at that level, in spite of unavoidable disturbances in the system. 

Note that it takes a few seconds for Data Center Manager to react and to reach the original power level.  Likewise, when the bottom server is brought to idle, it also pulled back the power consumption for the group.  However, the group power went back to the target power consumption after a couple of minutes.  If we look at the plot of the individual servers, we can see Data Center Manager at work maintaining the target power.

The figure above captures the behaviors of the individual servers.  Note how DCM allocates power to individual nodes yet it maintains a global power cap. When the server at the bottom is suddenly idled, there is a temporary dip in power server consumption for the group, but it soon recovers to the target capped level.  Also note that the power not used by the bottom server is reallocated to the remaining three nodes until they get close to the previously unconstrained level.

When you’re planning a backpacking trip, whether it’s for several hours or several days, space is at premium.  Not only do you need to think about tents, sleeping bags, clothing, first aid, and navigational gear (among other things), but also how to keep yourself properly hydrated and fueled up.  Oh yeah, you have to figure out how to cram all of this gear into your pack…and carrying an additional pack is not an option!

Odds are you’ll be heading into the wilderness and won’t be able to re-supply for a while, so one of the limiting factors will be the amount of food you can carry.  Running out of fuel in the middle of nowhere makes for a potentially disastrous situation.

So let’s look at the nutritional numbers and how best to fuel the trip:

• Fats:  ~9 calories per gram, and typically found in nuts and oils
• Carbohydrates and proteins:  ~4 calories per gram, and typically found in sugars, grains, and meats

If you’re trying to maximize the number of calories you can carry in order to sustain you during your trip, you probably want to pack more foods with a higher fat content (such as peanut butter) than carbs or protein.  More calories per gram à more energy in your pack to get you where you want to go.

You can probably figure out where I’m going with this analogy – low power CPUs are all about helping maximize your performance per rack, just like packing foods with more calories per gram help deliver more energy in a limited amount of backpack space.

Depending on your specific rack power or overall datacenter power / cooling environment, low power SKUs might be a good fit to help maximize your performance per rack.  For the Intel® Xeon® 5500 series, there are two low power CPU options available, both spec’d at a 60W Thermal Design Point (TDP):  Xeon® L5506 (2.13 GHz) and the Xeon® 5520 (2.26 GHz).  These two SKUs have the same features as the corresponding Xeon® E5506 and E5520 SKUs, just lower in power. 

If you’re buying LV Xeon® 5400 CPUs today, such as the L5420, expect a big jump in performance per rack with the Xeon® L55xx SKUs due to lower overall system power and higher performance.  Similar story if you’re evaluating the Xeon® E5506 or E520 SKUs – same performance with L55xx SKUs with lower system power, so higher performance per rack.

Have questions – ask me on this blog or Ask An Expert in the Server Room.

Cloud.

I haven’t seen as hyped a term in the data center arena since…um…virtualization.  Everyone is talking cloud, promising cloud, and believing cloud.  But what exactly is this thing called cloud? Is it outsourcing services to a provider, the next generation of virtualization, or something completely different?  There are a lot of definitions, and everyone has opinions…strongly held opinions that led me to my Rolling Stones inspired title to this post (yep, those lyrics...you can sing along now).  Chip Chat decided to get to the bottom of the cloud story, so we were excited to spend some time recently with Intel’s queen of the cloud, Raejeanne Skillern.  Check out my conversation with her here.

In a previous article we explored the implementation mechanisms for monitoring and controlling the power consumed by data center servers.  In this article we'll see that an ability to trim the power consumed by servers at convenient time represents a valuable tool to reduce stranded power and take maximum advantage of the power available under the existing infrastructure.  Let's start with a small example and figure out how to optimize the power utilization in a single rack.

Forecasting the power requirements for a server over the product’s lifetime is not an easy exercise.  Server power consumption is a function of server hardware specifications and the associated software and workloads running on them. Also the server’s configuration may change over time: the machine may be retrofitted with additional memory, new processors and hard drives. This challenge is compounded by more aggressive implementations of power proportional computing: servers of a few years ago exhibited little variability between power consumption at idle and power consumption at full power.

While power proportional computing has brought down the average power consumption, it also has increased its variance significantly, that is, data center administrators can expect wide swings in power consumption during normal operation.

Under-sizing the power infrastructure can lead to operational problems during the equipment’s lifetime: it may become impossible to fully load racks due to supply power limitations or because hot spots start developing.  This extra data center power capacity needs to be allocated for the rare occasion where it might be needed, but in practice and cannot be used because it is held in reserve, leading to the term "stranded power."

One possible strategy is to forecast power consumption using an upper bound.  The most obvious upper bound is to use the plate power, that is, the power in the electrical specifications of the server.  This is a number guaranteed to never be exceeded.  Throwing power at the problem is not unlike the approach of throwing bandwidth at the problem in network design to compensate for lack of bandwidth allocation capability and QoS mechanisms.  This approach is overly conservative because the power infrastructure is designed by adding the assumed peak power for each server over the equipment’s life time, an exceedingly unlikely event.

The picture is even worse when we realize that IT equipment represents only 30 to 40 percent of the power consumption in the data center as depicted in the figure below.  This means that the power forecasting in the data center must not only include the power consumed by the servers proper, but also the power consumed by the ancillary equipment, including cooling, heating and lighting, which can be over twice the power allocated to servers.

Establishing a power forecast and sizing up a data center based on nameplate will lead to gross underestimation of the actual power needed and unnecessary capital expenses[1]. The over-sizing of the power infrastructure is needed as insurance for the future because of the large uncertainty in the actual power consumption forecast.  It does not reflect actual need.

Power allocation in the data center.

A more realistic factor is to de-rate the plate power to a percentage determined by the practices at a particular site.  Typical numbers range between 40 percent and 70 percent.  Unfortunately, these numbers represent a guess representative over a server’s lifetime and are still overly conservative.

Intel(r) Data Center Manager provides a one year history of power consumption that allows a much tighter bound for power consumption forecasting.  At the same time, it is possible to limit power consumption to ensure that group power consumption does not exceed thresholds imposed by the utility power and the power supply infrastructure.

Initial testing performed with Baidu and China Telecom indicates that it is possible to increase rack density by 40 to 60 percent using a pre-existing data center infrastructure.

We will explore other uses in subsequent articles such as managing servers that are overheating and dynamically allocating power to server sub-groups depending on the priority of the applications they run.

"We are virtualizing".  I hear that at every customer, every day.  I am not sure where virtualization is on the hype curve, but i don't think it is anywhere near slowing down.  I am very glad to be past the "Dilbert" and "in flight magazine" era.  Customers seem to have a really solid command of what they want to virtualize and why they want to virtualize. ( not to imply that all the questions have been answered )

The latest Intel servers - Xeon 7400 processor series in the 4 socket family, and the incredible Xeon 5500 (Nehalem) processor series in the 2 socket family - deliver more than sufficient capacity for sweeping data center virtualization.  i.e. very few enterprise applications are to big for a VM on one of these platforms.

I hear three reasons from customers for virtualization. ( in order of emphasis )

1) To improve efficiency.  Most enterprise servers are only about 10% utilized ( and many of these are old, slow, inefficient servers)  Applications are partitioned onto individual servers for archaic historical reasons.  Combining these on powerful modern servers can dramatically reduce footprint, power, server costs and licensing costs.

2) To improve flexibility.  Virtualization allows "servers" ( think VMs ) to be easily moved from one platform to another - for sizing - for maintenance - for almost any reason.  With the Intel Flex Migration technology and recent versions of VMware ESX - customers can pool Intel servers across multiple generations and families.  Live migration from your Xeon 5100 processor based server to a cozy VM on a 4 socket Xeon 7400 based 24 core server.

3) To improve reliability.  Virtualization provides a vehicle for managing hardware failures, allowing near instantaneous fail-over in the event of a server loss.

Virtualization has moved out of the lab and become a "best know method" for doing IT right.

Intel points to three focus areas for servers.  Efficiency, Performance, and Virtualization.  I think virtualizaiton's place in this triad is fleeting.  It only remains  because changes are still being made to the platform to support virtualization.  Soon virtualization will become just another part of the stack- like the operating system.  Nobody claims their processor is optimized for running an operating system...  Even today choosing the best processors for virtualization is more about efficiency and performance than about virtualization features.  Fortunately - as I do work there - Intel has a solid lead on both efficiency and performance.

I'll be up front, I really don't know what Brittany Spears, Miley Cyrus or Susan Boyle would say about moving from RISC to the Xeon 5500 processor!. What I can share is the feedback that I'm getting direct from customers. I'm currently out on the road and have got some real feedback direct from customers on why they are looking at migrating their solutions from RISC  processors to Xeon processors.

Over the past couple of days I have had the opportunity to meet directly with individual customers and hosted a roundtable with several customers to discuss their plans to replace their RISC based infrastructure. The conversation has been very open and frank and has not been about 'should I move' but more focused on 'how do I make the move'. As could be expected the down economy is placing big taxes on the ability of IT organizations to support their business units need for organic growth in a flat to down IT spending environment. A big priority for most of the customers that I spoke with is how to reduce their overall TCO while still meeting the increased demands being placed on IT by their business Partners. Most of the customers are already engaged in active projects to assess moving from RISC or are building their plans to make this migration.

During the roundtable I had opportunity to share the latest Xeon 5500 processor performance comparisons Vs the main SPARC and POWER based solutions out there. There was great rejoicing and joy (ok I'm taking poetic license here) in the roundtable when we share some of the results that we highlighted when we launched the Xeon 5500 processor just over 3 weeks ago. So I want to spread the joy and let you read for yourself the performance and price performance benefits.

We compared the Xeon 5570 processor vs the top UltraSPARCT2+ in a 2 socket configuration. We took best published results on spec.org and sap (so no funny games at play). The results comparing best UltraSPARCT2+ vs best Xeon 5500 with 1 taken as baseline for SPARC redults were amazing

- 20% better on SAP-SD

- 62% better java performance for Specjbb2005

- 69%better for integer performance SPECIntrate-2006

- 75% better for floating point performance SPECfprate-2006

But the best bit was the cost competitiveness of the Xeon 5500 solutions. Comparing both solutions with 32GB memory, the Xeon 5500 based solutions are offered at approx $11,000 whereas the UltraSPARCT2+ is at $36,000.

Compared the Xeon 5570 processor vs the top POWER6 in a 2 socket configuration gave even more staggering results. At the roundtable today customers were amazed. They keep hearing that POWER 6 has leading performance and more GHz so better performance. Right?. Wrong is the answer and I noticed many customers scribbling down the comparisons. Again taking 1 as baseline for POWER results

- 150% better on SAP-SD

- 190% better java performance for Specjbb2005

- 126%better for integer performance SPECIntrate-2006

- 90%better for floating point performance SPECfprate-2006

But the best bit was the cost competitiveness of the Xeon 5500 solutions. Comparing both solutions with 32GB memory, the Xeon 5500 based solutions are 92% less expensive than equivalent POWER 6 offerings.

I only shared the specific comparisons vs RISC and have not gone into the architectural advancements of the Xeon 5500 processor and how it addresses real business needs that have been flagged to us. There have been lots of other blogs out in cyberspace over the last few weeks on improvements in IO, low latency etc. so you don't need my 2 cents.

I think now is the time to make the move from RISC, what do you think?