Found this video about how intel IT converted what was a high volume manafacturing facility to a high performance computing datacenter that now is on the top 500 list.   Watch Tom Greenbaum, Data Center Operations Manager for Intel IT, provide a description of this retro-fit and tour of the new facility.

Some key facts highlighted in the video

• avoided several million $ in facility cost avoidance
• landed traditional enterprise environment in raised floor, hot/cold aisle design in one section of facility
• landed HPC environmet on existing concrete slab floor which enabled higher density deployment of servers
• 6M Watt, 10K server capacity (4.7k today)
• room to grow for future to support data center consolidation

chris

I have been around the supercomputing market for over 25 years and have had an opportunity to see some interesting ideas come and go.  Let me share two that I experienced firsthand. 

·         CDC’s Cyber205 or a Cray 1S.  The CRAY-IS and the CDC Cyber 205 both offered effective vector processing, however, code conversion between them may have required some significant algorithmic changes. Cray of course won the HPC race at that time.  Note, the Cyber 205 was a tremendous performer, when you could keep their extreme.ly long vector pipeline busy. However, one branch or gap in the vector processing pipeline would cause a flush of the vector unit and what performance advantage you appeared to have vs. a Cray 1S was quickly erased.

·         An early day accelerator was Floating Point Systems.  In particular the FPS 164 was an awesome “off load” system where the needs of a few users were satisfied with better throughput than the Cray X-MP and Y-MP of the day. Convex, had a better idea.  It was better at serving the needs of more than an FPS 164 and was simpler to develop, maintain and scale software to next generation systems.

So what are the lessons from history? Perhaps it is that there it is there is a tight connection between application, architectures and algorithms and that it is extremely important to maintain a level of application flexibility and versatility in order to adopt new architectures as they become available in the market.  The old adage still remains true, software will outlive the useful life of hardware.  So it is important to be able to quickly adapt new shifts.

The same questions probably still apply today as they did when Cray, CDC and FPS were around.

When does an accelerator computing strategy work best?

The easiest answer is if your application is extremely data parallel in nature, then it may be well suited for an accelerator strategy. The word extremely is the critical part. 

If your application only performs some level of data parallelism and includes task, thread and cluster level parallelism or contains a small fraction of branching or is host to irregular data sizes, then perhaps an accelerator may not be the best fit.

How much real performance will an accelerator strategy deliver? 

Often times we hear claims of 10X, 20X or even greater than 30X. 

These are great headlines, but as many have noted, you need to understand an accelerators impact on the total execution time of your application.  What may have been 10X to 30X or more on a kernel of the application may only deliver a mere 2X to 3X or even less in terms of total application performance improvement.

Of course the real question is what are we really comparing performance speed ups to?

I have seen well tuned software on accelerators compared to “baseline” code running on one core of an old processor.  However, when you use available software technology and turn compiler flags on and add in a math kernel library call the performance on multi-core solutions can jump by over 10X and in some cases can exceed 30X multiples for total execution time.  This standards based accelerated software will scale forward as newer microarchitectures are made available from Intel.

Why is the difference between the promise and the actual performance so great? 

Always a good questionJ. 

The promise deals with a small part or a kernel of the software that is data parallel and can potentially scale linearly as more compute resources are added.  Again if the application is extremely data parallel, then an accelerator strategy may be the correct approach.

However, when the actual performance result, or total application performance, is significantly different it is often because of several things. 

·         One common reason is that you may be comparing optimized software on multi-core systems to optimized software on an accelerator.  When I compare similarly optimized software on a multi-core system I see that 20 – 30X difference often fades to less than 2X  and in most cases better than hardware accelerators.  This is because optimized software on a multi-core solution accelerates all components of the application.

·         Another situation is the bandwidth imbalance of the attach points of the accelerators, typically the attach speeds do not match the memory bandwidth or the ALU speed on the accelerators and the theoretical peak flops are tough to achieve.  Sometimes, for larger workloads due to limited amount of memory on the accelerator card, performance deteriorates.

·         Another situation may be that your application depends on different forms of parallelism which include task, thread or cluster level parallelism and even in some cases sequential forms of your software

So back to the differences in performance between the Cray 1 and CDC Cyber 205.

While Cyber 205 was great at edges of science the Cray proved to be the workhorse of high performance computing.  It offered better system balance than the Cyber 205.  Here is an example, if you take great care to optimize your software for a particular architecture you will no doubt see tremendous performance gains.  However, like the Cyber 205, if you break that pipeline you need to pay for the overhead to restart the long vector pipeline.  Often times, even with today’s accelerators, that start up cost reduces what appears to be stellar performance gains of the Cyber 205 to being no better than, or sometimes, even slower than the Cray 1.  There were of course examples with the Cyber 205, as there is today with accelerators that demonstrate where select sciences can see tremendous advantages over traditional computing solutions.

What other considerations may weigh in your decision to adopt an accelerator strategy?

Are you constantly refining your software?
Many researchers would probably answer yes.  They are constantly refining their software to improve the results the performance or both.

As I mentioned at the beginning of the blog, the old adage still remains true, software will outlive the useful life of hardware.  So it is important to be able to quickly adapt to new shifts.  One way to simplify these moves is to use standards based tools which can give you the flexibility to create applications that can use the multiple types of parallelism mentioned above via tools, compilers, and libraries.  You may also want to use standards based tools to acquire the versatility you need in order to scale your software across multiple architectures – e.g. large, many and heterogeneous cores. 

The caveat with respect to using non standard tools is that you become locked into a specific architecture.  If that architecture from the same vendor would happen to change, you may be required to make some significant changes (e.g. tuning to grain sizes).

Do you want to maintain, support and update multiple code bases?
I don’t.  I want to invest n the development of parallel algorithms.  The old adage is that software will far out live any hardware implementation still applies and I need the flexibility and versatility to quickly and as painlessly as possible be able to adopt new architectures as they are made available.  I do not want to invest in maintaining, supporting and updating an ever increasing set of code streams as newer architectures are made available.

Our team goal at Intel is to develop software tools and hardware technology that can help you scale-forward your application performance to future platforms without requiring a massive rebuild – just drop-in a new runtime that is optimized for the new platform to experience the improvement (akin to the printer/display driver model, buy a new printer/display, install the respective driver, and your system enjoys improved benefits).  That is the goal.

If you want to learn more about what we are doing to deliver high performing HPC solutions that are both flexible and versatile please visit www.intel.com/go/hpc

I have been watching the social chatter today about the latest Top500 supercomputing list and seeing companies, manufacturers, application vendors and even countries compete for mind share of this most recent list on twitter.

However, as I read about and explored this list, the things that jumped out at me were not the who’s number one, two, three … or who grew what number of spots ... but rather the trends that have occurred over time. These trends have not happened in the last 6 months or the last 6 years but instead over the course of nearly a decade of innovation

1)      Today, the #10 posting (a cluster using the 3-month old Xeon X5570 processor (Nehalem-EP)) delivers the same FLOPS performance capability equal to the entire June 2000 TOP 500 computers list. (see below)

source: http://www.top500.org/lists/2009/06/performance_development

2)      Also, the emergence of multi-core intel-based servers complemented by affordable open-source software solutions have enabled a transformation of how supercomputing performance is delivered. Intel based servers have gone from nearly “0” to nearly “400” over this decade.

source: http://www.intel.com/pressroom/images/IntelTOP500history.jpg

Looks like the Intel® Xeon® processor 5500 series is making lots of noise in HPC.  The QPI and integrated memory controller are really providing the boost necessary to make it an all around performance leader for HPC applications.  With all this performance why did Intel add a third memory channel?

The third memory channel enables the platform to support a boat load of memory.  Matter-of-fact, up to 192GB can be supported in a two socket configuration.  It wasn’t too long ago when only 32GB was supported in a dual socket configuration.  By having the ability to support so much memory you can now meet the needs of almost every HPC application.  The 5500 series is intended for all server markets, but let’s face it, with the design changes Intel made with the new architecture the server segment gaining the most benefit appears to be HPC. 

It seemed like yesterday when the only way to have access to large memory configurations was through expensive, proprietary SMP systems.  The HPC market for large SMP systems is still out there but it is shrinking…fast.  Today, we are clustering low cost solutions to create some of the most powerful systems in the world.  Standard components are leading to lower and lower system costs, delivering a price/performance advantage alternative solutions cannot meet.

Now that a single dual socket node can support up to 192GB’s it is important to understand how to get there.  First, to enable 192GB you need 16GB DIMMs x 12 memory slots.  There will be a premium for a 16GB DIMM.  Knowing the options and determining the best, most cost effective solution is going to be dependent upon your environment.  When a large memory node is required, do you purchase the 16GB DIMM’s or go up to a Multi-socket solution?  If I decide to scale back on the memory (use 4GB or 8GB DIMMs instead of 16GB DIMMs) what is the performance impact to my application?  If I am cost sensitive, will the lower cost outweigh the lack of performance?  Can I use SSD’s (Solid State Disk drives) to compensate for any performance loss due to lower memory capacity?  There are many questions to think about when deciding the right configuration for your application and environment and I certainly can’t answer them here.

Let’s not forget the third memory channel enables a different set of optimal memory configurations.  Think x3 when deciding on how much memory to install into your node; 12GB, 24GB, 48GB, etc.  What happens when you don’t use an optimal configuration?  Well it depends, in most cases the impact is minimal, but let me add a bit of context around minimal:

·         Low bandwidth sensitivity (more dependent upon the processor for performance)

–        E.g. Monte Carlo, Black-Scholes (financial modeling), BLAST (bioinformatics), AMBER (molecular dynamics)

–        Expect less than a 2% difference between memory configurations*

Ÿ  Medium bandwidth sensitivity (somewhat balanced between memory and CPU usage)

–        E.g. CFD, Explicit FEA, Implicit FEA (with robust I/O system)

–        Expect approx. 5% degradation for non-optimal symmetrical configurations*

Ÿ  High bandwidth sensitivity (high access to the system memory)

–        E.g. WRF (weather), POP (climate), MILC (physics), Reservoir Simulation

–        Expect approx. 10% degradation for non-optimal symmetrical configurations*

The results are interesting.  In all three cases above, the degraded performance is always better than the performance you would have with only two memory channels.

When you hear about performance impact of non-optimal memory you can see by the examples above, it is application dependent and will not have a severe impact on your overall system performance.   

The Intel Xeon processor 5500 series offers support for huge memory nodes with the addition of the third memory channel.  Memory configurations in multiples of three are ideal, but if you decide to stay with a power of two configuration the performance should still exceed that of a solution based upon only two memory channels.

*Based upon Intel internal measurements

My name is Steve Thorne, and this is my first blog post in The Server Room. I’m the product line manager for the Intel Xeon processor 5000 family, and I’m based out of our Hillsboro, Oregon facility. I’ve been looking forward to this blog post for quite some time, since I’ve been meeting with a wide variety of customers over the past few weeks.

It’s been just over a month since we introduced the Intel Xeon processor 5500 series (the processor formerly known as “Nehalem-EP”). We are certainly pleased with the response from the industry at this point. Below you will see some of my observations about what has transpired over the first 30 days of release. At the same time, I invite you to share some of your stories about recent installations of the Xeon 5500. Where is it being used? What kind of environments are you using it in? What kind of improvements have you observed in your deployments?

The industry response has been extremely encouraging to me. Our marketing teams spent more than three years diligently preparing for the successful introduction. Some of my observations from the first month include:

·         The list of vendors that support the Xeon 5500 continues to grow. We started with over 70 system manufacturers on March 30, 2009. And on April 14, 2009, Sun Microsystems introduced a new line of x64 blade servers, rack servers and workstations powered by the Intel Xeon processor 5500 series. Of particular interest is the Sun Blade X6275 server module. You can find more info at: http://www.sun.com/solutions/hpc/compute.jsp.

·         I attended our launch event in Santa Clara on March 30, 2009. While at the event, I was pleasantly surprised by the adulation from the customers who were in attendance. In particular, our friends in the Digital Content Creation (DCC) industry are eager to apply the capabilities of the Xeon 5500 for movie special effects and animated features. Being a father of three school age children, I’ve always been fond of our products’ role in the moviemaking process. It’s fun to take your kids to the theater and show them a concrete example of how these incredibly complex processors are used to generate chuckles and special effects in movies ranging from “Cars” to “Monsters vs. Aliens.”

·         Positive recognition has been accorded to the Xeon 5500 from a wide variety of independent press reviewers and articles. A recent internet search revealed almost 875 news references. Recently, George Ou of DailyTech published an interesting article titled “Server roundup: Intel “Nehalem” Xeon versus AMD “Shanghai” Opteron”. You can read the entire article at: http://www.dailytech.com/article.aspx?newsid=15036

·         On May 4, 2009 two independent financial analysts upgraded Intel Corp. stock. Both analysts attributed part of their positive outlook to the introduction and ramp of Xeon 5500 servers.

·         On April 8, 2009 the new Xeon 5500 was a centerpiece of our IDF event in Beijing. In his enterprise key note, Pat Gelsinger said the “Nehalem” microarchitecture has received worldwide acclaim.

·         Customer deployments are underway at leading data centers around the globe – particularly in High Performance Computing (HPC) applications. The HPC accounts encompass university research labs, commercial research and development and large scale clusters. These HPC customers are pushing the outer limits of scientific discovery and innovation, and the best examples are yet to come!

Personally, I was proud to be a part of the introduction of the Xeon 5500. There is a strong sense of satisfaction when the silicon is deployed in real-world environments. And in case you hadn’t heard, we are busy getting ready for the next addition to the Xeon family, codenamed “Westmere-EP.” We expect this new 32nm processor to be socket and pin-compatible with the Xeon 5500, and it will stretch the processor to support six individual CPU cores per socket. Stay tuned for this release in 2010!

That’s right, now you can buy a supercomputer that fits right under your desk. The PSC’s (Personal supercomputer) of today would have been #1 on the Top500 http://www.top500.org in November of 1996. ASCI Red would have been the first system to overtake the performance these small supercomputers can provide today.

So why do you need one of these high performance bad boys? Well, if you are trying to keep up with technology, beat out your competition, and do it at the lowest cost possible then you just better think about buying one. Whether you are in manufacturing, engineering, financial services or life sciences, the benefits offered are huge. You can now simulate vs constructing expensive prototypes, you can do more work at your desk vs. waiting to schedule the job on the oversubscribed cluster and most importantly, it provides the competitive advantage you most dearly need to keep up with your customer’s demands for lower pricing.

There are a couple of very interesting solutions on the market right now that should be considered. One is the Cray CX1. This little monster can support up to 16 quad core processors! With Nehalem soon to launch, that is one heck of a lot of performance.

http://www.cray.com/products/CX1.aspx

The CX1 is also ICR (Intel Cluster Ready) certified. This certification helps to ensure end users the system will provide a positive out-of-box experience. When you install the system and turn it on, it just works. You are maximizing your investment. The last thing a small business needs is to make the investment and then spend days getting the system up and running.

http://software.intel.com/en-us/cluster-ready/

With today’s economy, many businesses are reducing cost and putting off capital expenditures. You have decided to be like many other businesses and wait just one more year to upgrade your current computing system. Your competition decided not to wait. Instead, they purchased a PSC that was Intel Cluster Ready Certified and are now more productive than ever before. They made the investment and are now turning out new designs faster and at a lower cost than ever before. You scratch your head wondering how they do it. As you try to save your business they are growing and winning new business. Sometimes, being aggressive in a difficult time is the prudent thing to do…

Still wondering if this is right for me? Concerned you don’t have the IT staff to support such as beast? You don’t have the budget? Is there software out there I can use? All are good questions/concerns, but the ICR certified PSC minimizes if not eliminates the need for an IT staff. The PSC is one of the most affordable cluster solutions on the market today. It plugs right into your wall socket…you have tamed the beast! As for software, if you are purchasing an ICR certified system, then there are numerous applications available and most likely, one you are already familiar with.

When you are getting ready to make your next workstation or high-end PC purchase, I strongly recommend you consider one of the new kids on the block, the PSC. If it is Intel Cluster Ready Certified, you can rest assured the solution you do buy will just work.

I just got back from Supercomputing 2007. I remember a conversation 10 or 12 years ago with someone I really respect. That was just after Intel's Supercomputing Division had folded and HPC was in one of its cyclical downturns. Our conversation was roughly about there being no demand for large supercomputers anymore (outside of govt). We surmised that what seemed to be needed most was a cheap gigaflop. To some extent we were right, but mostly we were wrong. Right in that since then, basic engineering analysis has been a driver in the growth of HPC (thinking clusters) . Very wrong in thinking that demand for compute cycles would not continue to increase and a whole host of other things. Big, big miss on that one. If you believe the current market survey numbers, the high end of HPC is mostly stagnant (dollar wise, but not innovation wise), but the low end, small clusters, is growing by leaps and bounds.

One of the things I wanted to get a read on at the conference is 10GbE adoption. While a high performance interconnect can be important, especially if you are paying significant amounts for a SW license, so is convenience & ease of use. Particularly if the user base is increasingly non HPC geeks, but mechanical / electrical / aero type engineers who just need to get some work done. Plus, 20 Gb Infiniband / Myrinet / Quadrics might be overkill for small jobs (4 - 16 cluster nodes). My impression is that we still aren't there on 10GbE. I was hoping to see 10GBaseT but it was rare. A couple of vendors had it & could actually show me a switch, but that was it. CX4 really does give me the hives.

And there is the question of the day - accelerators. What I wanted to understand is the details of how people are programming these things to get an idea of whether the PCIe interface is going to be a bottleneck or not. Are people 'blocking' real codes at coarse enough granularity to avoid a PCIe bottleneck? I mostly struck out. I did have a good chat with the Clearspeed folks. Their programmability looked much better than I expected, but I wonder if it will be too labor intensive for all but the highest ROI situations.

Another item for me was small form factor boards & density in general. Supermicro, Tyan, Fujitsu, Intel EPSD all had small form factor / high density stuff - for rack n' stack configurations. SGI and Appro showed off what I considered complete systems based on small form factors. There were several more exotic options, but they tend to be outside my customer base.

The Sun / Rackable 'datacenter in a shipping container' seemed to get a good amount of attention. I'll be very curious to hear why end users like them (assuming they do). Is it reduced CapEx? Is it shorter time to datacenter implementation / expansion?

Going back to the conversation of a decade ago. We've gotten to the cheap *flop - I'll claim its clusters, or something close to them. Now the focus seems to be on making them 'user friendly' enough for the small industrial cluster crowd. Intel Cluster Ready is one example. WinCCS (or whatever they are calling it today) is another. But there were also a lot of booths emphasizing out of the box experience (SGI & Appro come to mind) or smaller players emphasizing custom configuration (per the application) & hand holding / down the street throat to choke type service levels.