The debate on how to best increase system capacity to accommodate growing applications has raged on for years; “scale up” with more CPU, memory, and I/O, or “scale out” with loosely connected systems.    Scaling out by adding networked systems to increase capacity has been a good economical solution for many IT managers because it allows them to grow by using less expensive, industry standard building blocks.  However, there are some notable exceptions to this line of thought.  One is that the class of applications that require shared memory and large database support are much better suited to run on a single, expandable system that scales up.  These are typically transaction processing, business intelligence and ERP solutions.   Until now, IT managers running applications that require scale-up systems larger than 4 or 8 CPUs have had limited platform choices and most were proprietary and expensive RISC-based servers.

The other problem with the scale out approach is the people, facilities, software and overhead costs and complexity of managing very large numbers of servers, which can grow to a point where the costs outweigh the performance and system cost benefits.  The industry solution to achieving better ROI has been to consolidate multiple scale-out servers onto single industry standard scale-up servers with virtualization solutions.  This is a good solution, but is limited by the number of application loads the IT manager feels comfortable placing on a single server, given the need to maintain peak performance and availability for each application.

Well, it looks like the scale-up, scale-out debate is about to take another turn.  In the server product update Intel gave on May 26th, they talked about new levels of system scalability and choice supported by the upcoming Nehalem-EX processor.  This processor will support systems that scale up to 8 sockets natively (shared memory, without any additional silicon), and up to 16 sockets and higher with node controllers from system manufactures that allow single systems to share memory beyond 8 sockets.   So far there are over 15 different designs from 8 OEMs that offer 8 socket or higher scalability.  But of course, for the class of application where scaling is important, socket count doesn’t tell the whole story of what’s needed for scalable performance.  Thread support, key for transaction processing and virtualization, scales at the rate of 16 threads per socket with 8 cores and Hyper Threading (2 threads per core).  That would be 128 threads for an 8-socket system, and 256 threads for 16 sockets.   And in order to keep those threads fed with data close to the CPU, each processor supports up to 24 MB of shared cache (1.5X current generation Xeon), and an impressive 16 memory slots per socket or 128 DIMMs on an 8-socket system.  In addition, the Scalable Memory Interconnect gives these systems 9 times the memory bandwidth of today’s top Xeon processor.  Finally, four QuickPath interconnect links per socket allow for high-bandwidth sharing of data across the system.

So the net of it is that the industry is going to see a broad selection of highly scalable, next-generation servers that significantly extend the economic advantage of industry standard scale-up solutions for business-critical, large database, and high-end virtualization/consolidation deployments.     I would expect these systems to give IT managers a very cost-effective alternative to the much more expensive and proprietary RISC-based servers they use today.

What are your thoughts?  Mike

Related Topics:

• NHM-EX Press Fact Sheet
• NHM-EX May 26th Press Briefing Video – condensed version
• IBM 8Socket Demo Video
• NHM-EX--A New Standard

I’ll not debate whether Cloud Computing is a passing fad, marketing hype, a revolution in computing, etc.; what I do know for a fact is that the interest in this model, from equipment vendors, service providers and end users is staying strong.  As much as Intel is reaching out into the industry to learn how people are hoping to take advantage of this phenomenon, what’s exciting to someone like me is that more and more service providers are approaching us on this is topic and seeking our input and guidance.  Service providers of various kinds are asking for Intel’s opinion and advice on how to prepare and evolve their data center architecture and practices to align with the expectations their customers have for cloud computing.  I’m not trying to brag, especially since it is obvious that there’s a ton of things “we” still need to figure out in this area; but when I see some of the giants in this community express appreciation of the contribution Intel is making, I can’t help but feel glad that we have done at least some of our homework right!

So what’s a chip company doing that could be remotely interesting to service provides?  Aren’t these the guys whose job it is to abstract all the hardware?  Absolutely!  But service providers are realizing that their solutions are better delivered and their business models are more competitive when they have a deeper understanding of what the underlying hardware is capable of.  For example, many of the customers I work with tell me that they were unaware of the technologies enabled by our platforms to intelligently manage server power consumption, not just at the individual node level, but for the whole of the data center.  My colleagues at our customers are pleasantly surprised to learn how Intel is pushing the boundaries for virtualization deployment and in collaboration with the leading vendors of virtualization software is making the use of this foundational technology more efficient for cloud computing.

There are many more topics I can add to this list, and service providers have a lot of places to go besides Intel for information.  But what I hear often from the customers I work with is that Intel’s ability to be an impartial (vendor neutral) technology advisor is most appreciated.  Of course not everyone is in a position to take advantage of the latest technology, nor does every new technology we enable serve everyone’s purpose.  But if you are a service provider interested in topics on data center optimization whether that be at: the cpu or chipset, the server, the software or the facilities, I’d encourage you to read up on our products and technologies found in this forum, and in other places on our intel.com sites.  And if there is something you need but can’t find, or need more information feel free to drop me note.

When you’re planning a backpacking trip, whether it’s for several hours or several days, space is at premium.  Not only do you need to think about tents, sleeping bags, clothing, first aid, and navigational gear (among other things), but also how to keep yourself properly hydrated and fueled up.  Oh yeah, you have to figure out how to cram all of this gear into your pack…and carrying an additional pack is not an option!

Odds are you’ll be heading into the wilderness and won’t be able to re-supply for a while, so one of the limiting factors will be the amount of food you can carry.  Running out of fuel in the middle of nowhere makes for a potentially disastrous situation.

So let’s look at the nutritional numbers and how best to fuel the trip:

• Fats:  ~9 calories per gram, and typically found in nuts and oils
• Carbohydrates and proteins:  ~4 calories per gram, and typically found in sugars, grains, and meats

If you’re trying to maximize the number of calories you can carry in order to sustain you during your trip, you probably want to pack more foods with a higher fat content (such as peanut butter) than carbs or protein.  More calories per gram à more energy in your pack to get you where you want to go.

You can probably figure out where I’m going with this analogy – low power CPUs are all about helping maximize your performance per rack, just like packing foods with more calories per gram help deliver more energy in a limited amount of backpack space.

Depending on your specific rack power or overall datacenter power / cooling environment, low power SKUs might be a good fit to help maximize your performance per rack.  For the Intel® Xeon® 5500 series, there are two low power CPU options available, both spec’d at a 60W Thermal Design Point (TDP):  Xeon® L5506 (2.13 GHz) and the Xeon® 5520 (2.26 GHz).  These two SKUs have the same features as the corresponding Xeon® E5506 and E5520 SKUs, just lower in power. 

If you’re buying LV Xeon® 5400 CPUs today, such as the L5420, expect a big jump in performance per rack with the Xeon® L55xx SKUs due to lower overall system power and higher performance.  Similar story if you’re evaluating the Xeon® E5506 or E520 SKUs – same performance with L55xx SKUs with lower system power, so higher performance per rack.

Have questions – ask me on this blog or Ask An Expert in the Server Room.

In this installment on uses of server power management we continue the discussion on using this capability for other uses beyond server rack density.

Intel(r) Data Center Manager (Intel DCM) is a software development kit that can provide real time information to optimize data center operations.  It provides a comprehensive list of publish/subscribe event mechanisms that can form the basis of a sophisticated data center management infrastructure integrating multiple applications where applications get notified of relevant thermal and power events and can apply appropriate policies.

These policies can span a wide range of potential actions:  dialing back power consumption to bring it down below a reference threshold or to reduce thermal stress on the cooling system.  Some actions can be complex, such as migrating workloads across hosts in a virtualized environment, powering down equipment or even performing coordinated actions with building management systems.

Intel DCM also provides inlet temperature or front panel thermals along with a historical record that can be used to identify trouble spots in the data center.  This information provides insights to optimize the thermal design of the data center.  The actions needed to fix trouble spots need not be expensive at all; they may involve no more than relocating a few perforated tiles or installing blanking panels and grommets to minimize air leaks in the raised metal floor.  Traditionally, the hardest part has been identifying the trouble spots, involving time consuming temperature and air flow measurements. Intel Data Center Management provides much of this data ready made from operations. Typically this type of analysis is done by a consulting team and the cost of this exercise is high, anywhere between $50,000 to a $150,000 for a 25,000 square foot data center.  This analysis yields a single snapshot in time which becomes gradually more inaccurate as  the equipment in the data center is refreshed and reconfigured.

Deployment scaling can range from a small business managing a few co-located servers in a shared rack in a multi-tenant environment to organizations managing thousands of servers.

The event handling capability is an software abstraction implemented by the Intel DCM SDK running in a management console.  From an architectural perspective, and the fact that the number of nodes managed can range in the hundreds, it makes more sense to implement this capability as software rather than firmware.  Node Manager is implemented as firmware and it typically controls one server. The choice of SDK over a self-standing management application was also deliberate.  Although Intel DCM comes with a reference GUI to manage a small number of nodes as a self-standing application, it shines when it's used as a building block for higher level management applications.  The integration is done through a Web services interface. Documentation for Intel DCM can be found in http://software.intel.com/sites/datacentermanager/.

I was thinking about what to write in my next blog and what I could share beyond what I have written previously about Intel Vs RISC in terms of TCO, performance and the customers that are choosing to move.

Luckily I didn't have to think too long on a Friday morning as a a topic came to mind instantly. There are numerous articles flying around this morning that picked up on the Oracle comments yesterday about how SPARC based systems compare to Intel. Thanks for providing me with an appropriate topic.

So in case you missed it, there was a question and answer session with Larry Ellison. When asked about SPARC, this was the reply "SPARC is much more energy efficient than Intel while delivering the same performance on a per socket basis. This is not a green issue, its an economic issue. Today, database centers are paying as much for electricity to run their computers as they pay to buy computers. SPARC machines are much less expensive to run than Intel machines"

1) SPARC more energy efficient than Intel?  Seriously, in what parallel universe does that exists?

SUN continues to use watts per thread as measure of energy efficiency. The recognized industry standard benchmark for measuring energy efficiency is SPECpowerand I don't see any SPARC based results in the 91 results published. The absence of a result certainly says something very clear to me - no story.

These UltraSPARCT2+ systems get loaded with a lot of memory to deliver the their results, so when you look at overall system power (what people care about) they are not as energy efficient as Intel based systems.

SPECpower is effectively based of SPECJbb-2005 so another way of loking at this is to look at the SPECJbb-2005 results for a 4 socket UltraSPARcT2+ system and a Xeon 7400 system. The 4s UltraSPARCT2+ delivers 693k BOPs while Xeon 7400 is 532kBOPs. So you conclude that SPARC is better than Xeon?. That would be the wrong conclusion

UltraSPARCT2+ system would consume 1525 watts Vs Xeon 7400 at 816 watts. If you look at BOPs per watt (another way of looking at energy efficiency and performance) then you would see that Xeon 7400 is 43% more energy efficient. Doing a similar comparison with Xeon 5400 (I haven't even talked about our latest Xeon 5500, Nehalem) would be up to 77% more efficient than UltraSPARCT2+.

And lastly before I forget to mention the 4s UltraSPARCT2+ had 128GB memory and costs over $150,000for the system, while Xeon 7400 based system had 64GB memory and costs around $32,000.

2) SPARC deliver same performance on a per socket basis?

2S Xeon 5500 has performance leadership over 2S UltaSPARCT2+ across a wide range of benchmarks. Up to 70% more performance and up to 60% lower system cost. 4S Xeon 7400 has price/performance leadership over 4S UltraSPARCT2+, UltraSPARCT2+ results achieved with system loaded with lots of memory that drives the cost up to 3-4Xthat of Xeon 7400 system

3) SPARC machine are less expensive to run?. I can't for the life of me work this one out!.

Hardware systems based on Intel have leading price/performance (read cheaper), lower energy needs (so electrivity bill lower) and any software product with a license per core strcuture is less expensive on Xeon system than an 8 core UltraSPARcT2+ (which also has higher multipler per core)

That's all for now folks. I just wanted to share some data on why I know that SPARC machines are much MORE expensive to run than Intel machines

I've been in Las Vegas this week for the Blades Systems Insight event talking about data center transformation and data center efficiency (no white tiger sightings...just technology this week in Vegas).  This event draws attendees who are deploying high density compute platforms in their data centers and dealing with the power and cooling challenges that come along with these environments. So I was excited to share some of Intel's thoughts on power and cooling optimization beyond pure system refresh.  If you read the blogs on the server room you know plenty about the compelling financial benefits associated with refresh...and if you haven't seen this yet check out my friend Chris Peters' blog here.

But back to the show and the shower curtains...If you dip a bit deeper into the challenge of data center efficiency, three primary focus areas emerge:

Power: The underlying power cabling and infrastructure into your datacenter.  Ultimately you want the most efficient power delivery possible.

Cooling: The HVAC systems, fans, and ducting installed to remove heat from your datacenter and let you avoid thermal environments that make Las Vegas feel chilly.

Compute: Server, network and storage gear that drive business producitivity for your organization.  This is why you have datacenters to begin with so the ultimate goal is to optimize percentage of power flowing to compute and productivity spent on every kw of power within your compute infrastructure.

At the Blades event we were discussing the impact of high density environments to this fragile ecosystem.  High density environments a) require more power, more than the typical 750W per square foot that an average rack requires and far more than the 75-100W/sq foor that a typical datacenter facility supports.  High density environments also produce a lot of heat that needs to be dealt with by cooling systems that are often close to their cooling capacity.  So how much density is a good thing for datacenters and how do we deal with that gap between power delivered and power required?  I'd like to provide a few concepts but ultimately every datacenter is different...so I'd love to hear from you on how you've dealt with this as well. In this blog I'm going to start with cooling capacity as there are a lot of options to consider:

#1 Warmer datacenters.  ASHRAE recently updated their datacenter temp and humidity recommendations with a range of 18-27 C.  What this means is that server inlet temps can be set higher than what many datacenters are running today...the first step here is to measure your server inlet temp to get a picture of what your facility is operating at, checking with your manufacturers warranty spec, and measuring your power usage difference when altering the datacenter temp - remember to take before and after readings on your cooling power usage.

#2 Cool aisle containment: This is a pretty simple concept - placing barriers to control cool air and confining it to the area where servers need it.  Think about this as constructing a type of wall or ceiling around the cool aisle to control air flow.  So what are these walls made of? I've seen them made of plexiglass and plastic sheeting...and this week at the conference I heard about one of the largest banks in America who is experimenting with the deployment of shower curtains to control air flow and reporting a 15 degree drop in temperature associated with installation.  Now...last time I checked a shower curtain cost a few bucks so we're not talking about a major investment to test this in your datacenter.

#3 Ambient air cooling: Even in Las Vegas datacenters are utilizing outside/filtered ambient air economizers instead of their chillers to deliver cooled air at least part of the year.  This concept is simple - it's like turning on your furnace's fan setting to cool your house instead of your AC and in many regions of the country you can utilize this much of the year at a fraction of the cost of running a chiller.

#4 Liquid cooled cabinets - think of these essentially as a good Sub-Zero for the datacenter and especially applicable for the high density environments that we were focused on at the blade conference.  They basically contain a rack of compute equipment and chill this equipment utilizing liquid cooling.  This is a great way to isolate highly dense racks from your datacenter cooling equation completely and works especially well in heterogeneous environments where cooling requirements vary from rack to rack.

I will be back to you on the power and compute vectors next...in the meantime I'd love to hear if your datacenter has implemented any of these approaches and any results you've been able to measure.

With the introduction of the Intel Xeon processor 5500 series last month, I wrote a blog that discussed that server refresh was an intelligent investment in that it could deliver a rapid payback on investment. For the past few years, I have been working to understand the costs and benefits of server replacement and there are a few conclusions I can draw.

1)      Server Refresh is not new concept.  This approach has existed for decades.  People replace technology as it ages because new software and new technologies enable better business capabilities and as technology ages, the warranty expires and incidence of failure increases. How many of you still have your first mp3 player?

2)      ROI and Refresh Vary. The rate of refresh is a balance of the investment required (purchase, install, removal, validation, etc) and the savings achieved (operational costs, cost avoidance, employee productivity) balanced with the business opportunities available to you (business growth or new business markets, cost of capital, revenue generating investments)

3)      One Size Does not fit all.  Every business looks at financials and opportunities for their business a little differently and calculates their costs and savings differently.

So a few months ago, I embarked with some of my peers, with Intel IT, and industry leading ROI and TCO consultant Alinean, to apply what I have been learning and build an interactive tool to help you model your savings opportunity for server refresh and replacement. 

We identified and were able to model eleven cost and savings categories (both pluses and minuses) in the Server Refresh ROI calculation and make these cost category assumptions able to be included, excluded or modified by you.  You can model and view scenario output real time and print/email reports to share with others.

I invite you to learn more about the tool with this informal how-to-use guide , or better yet, use the tool and estimate how much you could save replacing old servers with new.  Try the new Intel® Xeon® processor-based Server Refresh Savings Estimator today.

You can provide feedback through the tool’s registration process or by responding directly on this blog. I look forward to hearing from you either way.

Thanks, Chris