Nehalem-EX: Big Memory for Big Science

I was at SuperComputing’09 last week in Portland, Oregon. I talked with some brilliant people, and saw some fantastic stuff.

It was good timing on my part because last week Intel also announced that it would offer a 6-core, frequency-optimized version of its Nehalem-EX product due out next year. This part is intended for use in tackling some of the types of high performance computing (HPC) workloads prominently displayed at SC’09.

Most people know that the majority of HPC workloads today are based on clusters of relatively small-memory, 2-socket systems. That is because most HPC workloads may be broken into smaller, discrete units of work that can be efficiently processed using such clusters. For these workloads the primary hardware capability selection criterion is typically a balance of both memory bandwidth and compute FLOPs (floating point operations per second).

But there are other types of HPC workloads. Specifically, those that deal with very large datasets (some as large as a terabyte) or those that have to deal with non-sequential memory access. This means the workloads simply aren’t easily divisible--or it is inefficient to do so-- into the relatively small memory footprints used in traditional clustered 2-socket HPC solutions. Examples of these types of bigger memory applications can be found in a variety of fields such as weather prediction, manufacturing structure analysis, and financial services.

The high-speed processing requirements and size of these workloads put a greater premium on system memory capacity/bandwidth than on compute FLOPs.

If the larger dataset won’t fit into available memory, and dividing up the dataset to spread across multiple nodes cannot easily be done, then data has to be moved in and out memory to hard disk.  But using hard disk drives (which are many times slower than RAM memory) can drastically impair performance.

There are now two better alternatives to the use of hard drivers. One is SSDs and the other is having a larger memory footprint. Solid State Drives have fairly high data density vs RAM, but much faster access than hard-disk drives--albeit still markedly slower than RAM. Another solution is to simply have more capacity of the faster RAM. This last one is what the Nehalem-EX HPC part is aimed at.

Nehalem-EX is the Expandable Class of Nehalem. The Expandable Class brings all the goodness of the Nehalem architecture (Xeon 5500 product line) to the HPC market, but in the form of a “super node” that has greater: a) core/thread count, b) socket scaling (up to 256), c) I/O and memory capacity (up to 1 terabyte in a 4 socket system) and bandwidth at capacity, d) reliability features, e) and other features.

The 6-core frequency-optimized Nehalem-EX part has also been tuned to offer the highest core frequency possible for this chip.   In creating this part, Intel is meeting the needs of the HPC community that want higher scalar performance along with the benefit of large memory capacity and bandwidth per core.

Of course the 8-core version of NHM-EX is still an option for those HPC workloads that scale well with more cores while still looking for the high memory capacity of the expandable class.

By having both 8-core and frequency optimized 6-core versions of the NHM-EX class of processors means HPC researchers have greater choice in selecting the processor best suited for their specific workloads.

After talking with some of the researchers at SC’09 last week I’m really excited to see how the Nehalem-EX “super node” will deliver the necessary compute and memory capabilities to help those researchers solve some of their biggest challenges.

In order to deliver to the continued promise of Moore’s Law, Intel’s Information Technology team needs to enable Intel’s Silicon designers with the tools, capabilities and streamlined processes to bring higher performing processors to market every year.  The latest generation of 45nm products (ie the Intel microarchitecture, codenamed Nehalem) was an especially challenging project for us. 

With Intel design computing demand growing an average of 45% year over year coupled with the rich technology capabilities in the 45nm based Nehalem micro-architecture, the computational requirements of silicon tape-out (the last stage of design before manufacturing) represented an approximate 13x increase in increase in demand from prior 65nm processors. Staring at this demand (1.2 million hours of compute demand per day) plus a need to bring products to market faster and more efficiently, our IT team realized we needed to do something different - our standard grid computing solution that was sufficient for earlier stage design work was insufficient for tape-out. 

Solution: Intel IT built a High Performance Computing (HPC) solution that currently rank in the Top 500 list of supercomputers (#261, #308, Nov 09) and feature a new parallel storage environment to support our 45nm Silicon tape-out process.  The details of this effort are captured in this whitepaper. 

In summary, the Intel IT HPC solution employs two of the world’s fastest supercomputers to create the fastest microprocessors helping Intel achieve the following results.

·         Completed 45nm tape-out in 10day, less than HALF the time of prior products

·         Delivered an estimated incremental value of $44M to Intel

I can’t wait for what tomorrow will bring as Intel IT is already upgrading and evolving this HPC solution to support our future generations of micro-processor designs. Tune in tomorrow at SuperComputing 2009 in Portland where Shesha Krishnapura from Intel IT will present more details on our HPC environment or join us December 8th, 2009 from 10-12am PST for a live chat with Intel IT experts in the Server Room

Chris (twitter)

Interactive Modeling and Simulation – Come on you are kidding!!

Recent advancements in mathematical modeling, computational algorithms, and the speed of computers based on technologies like the Intel® Xeon® processor 5500 series have brought the field of computer simulation to the threshold of a new era.  While not quite interactive, simulation and analysis can now occur at a pace that impacts decisions further upstream in the design process. 

Simulation and analysis tools are also no longer the domain of the expert.  Organizations can now potentially achieve a more efficient product development process by considering a reordering of product design activities and empowering engineers with easy-to-use and powerful 3D conceptual design tools and early access to CAE applications.

Why consider reordering your product development process?

This is not new news. Manufacturers know that when product analysis or simulation results trail the detailed design process that product changes are become significantly expensive and will most likely negatively impact new product release schedules. Worse yet, they also realize that changes made downstream in any design cycle are often “last minute” and almost always imply compromises on original design goals. This, of course, cuts into the product performance and profits of the new or updated product.

By reordering product design activities, manufactures may be able to achieve a more efficient product development process and reduce overall product development cost, time and risk.

No experts needed.

Don’t be fooled.  While ISV’s from ANSYS, ALTAIR, MSC, PTC, Siemens PLM, SIMLUIA, SolidWorks and others have made tremendous strides in making their simulation products easier to use, you probably still need an expert.  However, their collective advancements in tools, wrappers, and easy-to-use frameworks that guide the engineers through complex multi-physics analyses with drag-and-drop simplicity make it easier to move analysis further upstream. 

That means your expert can now focus on the really hard problems.

Workgroup Computing – Bringing “Real” HPC Computing To Your Department

Using analysis and simulation to get results before the detailed design process begins will help ensure the CAD models meet performance requirements and will almost always mitigate last-minute and expensive design changes.

Large scale compute intensive jobs used to require investments and/or access to a divisionally shared, large scale cluster housed in a controlled Data Center environment …supporting hundreds of users.

While this may have been true a few years ago, the advancements in mathematical modeling, computational algorithms, and the speed of computers based on technologies like the Intel® Xeon® processor 5500 series now makes it possible to quickly and efficiently solve large scale problems closer to the engineers responsible for dealing with them, on compute clusters supporting small workgroups or departments of engineers vs. large scale clusters shared by hundreds of engineers.

As an example let’s look at the Cray CX1™ deskside personal supercomputer.  Like others in this new usage category, it presents an organization with a solution that is the "right size" in performance, functionality, and cost for individuals and departmental workgroups who want to harness HPC without the complexity of traditional clusters.  Equipped with powerful Intel Xeon 5500 series processors the Cray CX1 delivers the power of a high performance cluster with the ease-of-use and seamless integration of a workstation.

OK, You Can Give Me The Performance, But The Support Can Be A Nightmare

Intel® Cluster Ready makes HPC simpler.  It boosts productivity and solves new problems. The Intel® Cluster Ready program makes it simpler to experience the power of high-performance computing. 

Intel Cluster Ready presents HPC users a certification program that is designed to establish a common specification among original equipment manufacturers, independent software vendors (ISVs) and others for designing, programming and deploying high performance clusters built with Intel components.

For users, this certification means that these certified HPC systems will run a wide range of Intel Cluster Ready ISV applications right of the box.  Tested, validated and simple.

By selecting a certified Intel Cluster Ready system for your registered Intel Cluster Ready applications you can be confident that hardware and software components will work together, right out of the box. Software tools such as Intel® Cluster Checker help ensure that those components continue to work together, delivering a high level of quality and a low total cost of ownership over the course of the cluster’s lifetime.

To learn more about Intel HPC Technology visit www.intel.com/go/HPC

Found this video about how intel IT converted what was a high volume manafacturing facility to a high performance computing datacenter that now is on the top 500 list.   Watch Tom Greenbaum, Data Center Operations Manager for Intel IT, provide a description of this retro-fit and tour of the new facility.

Some key facts highlighted in the video

• avoided several million $ in facility cost avoidance
• landed traditional enterprise environment in raised floor, hot/cold aisle design in one section of facility
• landed HPC environmet on existing concrete slab floor which enabled higher density deployment of servers
• 6M Watt, 10K server capacity (4.7k today)
• room to grow for future to support data center consolidation

chris

I have been around the supercomputing market for over 25 years and have had an opportunity to see some interesting ideas come and go.  Let me share two that I experienced firsthand. 

·         CDC’s Cyber205 or a Cray 1S.  The CRAY-IS and the CDC Cyber 205 both offered effective vector processing, however, code conversion between them may have required some significant algorithmic changes. Cray of course won the HPC race at that time.  Note, the Cyber 205 was a tremendous performer, when you could keep their extreme.ly long vector pipeline busy. However, one branch or gap in the vector processing pipeline would cause a flush of the vector unit and what performance advantage you appeared to have vs. a Cray 1S was quickly erased.

·         An early day accelerator was Floating Point Systems.  In particular the FPS 164 was an awesome “off load” system where the needs of a few users were satisfied with better throughput than the Cray X-MP and Y-MP of the day. Convex, had a better idea.  It was better at serving the needs of more than an FPS 164 and was simpler to develop, maintain and scale software to next generation systems.

So what are the lessons from history? Perhaps it is that there it is there is a tight connection between application, architectures and algorithms and that it is extremely important to maintain a level of application flexibility and versatility in order to adopt new architectures as they become available in the market.  The old adage still remains true, software will outlive the useful life of hardware.  So it is important to be able to quickly adapt new shifts.

The same questions probably still apply today as they did when Cray, CDC and FPS were around.

When does an accelerator computing strategy work best?

The easiest answer is if your application is extremely data parallel in nature, then it may be well suited for an accelerator strategy. The word extremely is the critical part. 

If your application only performs some level of data parallelism and includes task, thread and cluster level parallelism or contains a small fraction of branching or is host to irregular data sizes, then perhaps an accelerator may not be the best fit.

How much real performance will an accelerator strategy deliver? 

Often times we hear claims of 10X, 20X or even greater than 30X. 

These are great headlines, but as many have noted, you need to understand an accelerators impact on the total execution time of your application.  What may have been 10X to 30X or more on a kernel of the application may only deliver a mere 2X to 3X or even less in terms of total application performance improvement.

Of course the real question is what are we really comparing performance speed ups to?

I have seen well tuned software on accelerators compared to “baseline” code running on one core of an old processor.  However, when you use available software technology and turn compiler flags on and add in a math kernel library call the performance on multi-core solutions can jump by over 10X and in some cases can exceed 30X multiples for total execution time.  This standards based accelerated software will scale forward as newer microarchitectures are made available from Intel.

Why is the difference between the promise and the actual performance so great? 

Always a good questionJ. 

The promise deals with a small part or a kernel of the software that is data parallel and can potentially scale linearly as more compute resources are added.  Again if the application is extremely data parallel, then an accelerator strategy may be the correct approach.

However, when the actual performance result, or total application performance, is significantly different it is often because of several things. 

·         One common reason is that you may be comparing optimized software on multi-core systems to optimized software on an accelerator.  When I compare similarly optimized software on a multi-core system I see that 20 – 30X difference often fades to less than 2X  and in most cases better than hardware accelerators.  This is because optimized software on a multi-core solution accelerates all components of the application.

·         Another situation is the bandwidth imbalance of the attach points of the accelerators, typically the attach speeds do not match the memory bandwidth or the ALU speed on the accelerators and the theoretical peak flops are tough to achieve.  Sometimes, for larger workloads due to limited amount of memory on the accelerator card, performance deteriorates.

·         Another situation may be that your application depends on different forms of parallelism which include task, thread or cluster level parallelism and even in some cases sequential forms of your software

So back to the differences in performance between the Cray 1 and CDC Cyber 205.

While Cyber 205 was great at edges of science the Cray proved to be the workhorse of high performance computing.  It offered better system balance than the Cyber 205.  Here is an example, if you take great care to optimize your software for a particular architecture you will no doubt see tremendous performance gains.  However, like the Cyber 205, if you break that pipeline you need to pay for the overhead to restart the long vector pipeline.  Often times, even with today’s accelerators, that start up cost reduces what appears to be stellar performance gains of the Cyber 205 to being no better than, or sometimes, even slower than the Cray 1.  There were of course examples with the Cyber 205, as there is today with accelerators that demonstrate where select sciences can see tremendous advantages over traditional computing solutions.

What other considerations may weigh in your decision to adopt an accelerator strategy?

Are you constantly refining your software?
Many researchers would probably answer yes.  They are constantly refining their software to improve the results the performance or both.

As I mentioned at the beginning of the blog, the old adage still remains true, software will outlive the useful life of hardware.  So it is important to be able to quickly adapt to new shifts.  One way to simplify these moves is to use standards based tools which can give you the flexibility to create applications that can use the multiple types of parallelism mentioned above via tools, compilers, and libraries.  You may also want to use standards based tools to acquire the versatility you need in order to scale your software across multiple architectures – e.g. large, many and heterogeneous cores. 

The caveat with respect to using non standard tools is that you become locked into a specific architecture.  If that architecture from the same vendor would happen to change, you may be required to make some significant changes (e.g. tuning to grain sizes).

Do you want to maintain, support and update multiple code bases?
I don’t.  I want to invest n the development of parallel algorithms.  The old adage is that software will far out live any hardware implementation still applies and I need the flexibility and versatility to quickly and as painlessly as possible be able to adopt new architectures as they are made available.  I do not want to invest in maintaining, supporting and updating an ever increasing set of code streams as newer architectures are made available.

Our team goal at Intel is to develop software tools and hardware technology that can help you scale-forward your application performance to future platforms without requiring a massive rebuild – just drop-in a new runtime that is optimized for the new platform to experience the improvement (akin to the printer/display driver model, buy a new printer/display, install the respective driver, and your system enjoys improved benefits).  That is the goal.

If you want to learn more about what we are doing to deliver high performing HPC solutions that are both flexible and versatile please visit www.intel.com/go/hpc

I have been watching the social chatter today about the latest Top500 supercomputing list and seeing companies, manufacturers, application vendors and even countries compete for mind share of this most recent list on twitter.

However, as I read about and explored this list, the things that jumped out at me were not the who’s number one, two, three … or who grew what number of spots ... but rather the trends that have occurred over time. These trends have not happened in the last 6 months or the last 6 years but instead over the course of nearly a decade of innovation

1)      Today, the #10 posting (a cluster using the 3-month old Xeon X5570 processor (Nehalem-EP)) delivers the same FLOPS performance capability equal to the entire June 2000 TOP 500 computers list. (see below)

source: http://www.top500.org/lists/2009/06/performance_development

2)      Also, the emergence of multi-core intel-based servers complemented by affordable open-source software solutions have enabled a transformation of how supercomputing performance is delivered. Intel based servers have gone from nearly “0” to nearly “400” over this decade.

source: http://www.intel.com/pressroom/images/IntelTOP500history.jpg

Looks like the Intel® Xeon® processor 5500 series is making lots of noise in HPC.  The QPI and integrated memory controller are really providing the boost necessary to make it an all around performance leader for HPC applications.  With all this performance why did Intel add a third memory channel?

The third memory channel enables the platform to support a boat load of memory.  Matter-of-fact, up to 192GB can be supported in a two socket configuration.  It wasn’t too long ago when only 32GB was supported in a dual socket configuration.  By having the ability to support so much memory you can now meet the needs of almost every HPC application.  The 5500 series is intended for all server markets, but let’s face it, with the design changes Intel made with the new architecture the server segment gaining the most benefit appears to be HPC. 

It seemed like yesterday when the only way to have access to large memory configurations was through expensive, proprietary SMP systems.  The HPC market for large SMP systems is still out there but it is shrinking…fast.  Today, we are clustering low cost solutions to create some of the most powerful systems in the world.  Standard components are leading to lower and lower system costs, delivering a price/performance advantage alternative solutions cannot meet.

Now that a single dual socket node can support up to 192GB’s it is important to understand how to get there.  First, to enable 192GB you need 16GB DIMMs x 12 memory slots.  There will be a premium for a 16GB DIMM.  Knowing the options and determining the best, most cost effective solution is going to be dependent upon your environment.  When a large memory node is required, do you purchase the 16GB DIMM’s or go up to a Multi-socket solution?  If I decide to scale back on the memory (use 4GB or 8GB DIMMs instead of 16GB DIMMs) what is the performance impact to my application?  If I am cost sensitive, will the lower cost outweigh the lack of performance?  Can I use SSD’s (Solid State Disk drives) to compensate for any performance loss due to lower memory capacity?  There are many questions to think about when deciding the right configuration for your application and environment and I certainly can’t answer them here.

Let’s not forget the third memory channel enables a different set of optimal memory configurations.  Think x3 when deciding on how much memory to install into your node; 12GB, 24GB, 48GB, etc.  What happens when you don’t use an optimal configuration?  Well it depends, in most cases the impact is minimal, but let me add a bit of context around minimal:

·         Low bandwidth sensitivity (more dependent upon the processor for performance)

–        E.g. Monte Carlo, Black-Scholes (financial modeling), BLAST (bioinformatics), AMBER (molecular dynamics)

–        Expect less than a 2% difference between memory configurations*

Ÿ  Medium bandwidth sensitivity (somewhat balanced between memory and CPU usage)

–        E.g. CFD, Explicit FEA, Implicit FEA (with robust I/O system)

–        Expect approx. 5% degradation for non-optimal symmetrical configurations*

Ÿ  High bandwidth sensitivity (high access to the system memory)

–        E.g. WRF (weather), POP (climate), MILC (physics), Reservoir Simulation

–        Expect approx. 10% degradation for non-optimal symmetrical configurations*

The results are interesting.  In all three cases above, the degraded performance is always better than the performance you would have with only two memory channels.

When you hear about performance impact of non-optimal memory you can see by the examples above, it is application dependent and will not have a severe impact on your overall system performance.   

The Intel Xeon processor 5500 series offers support for huge memory nodes with the addition of the third memory channel.  Memory configurations in multiples of three are ideal, but if you decide to stay with a power of two configuration the performance should still exceed that of a solution based upon only two memory channels.

*Based upon Intel internal measurements

My name is Steve Thorne, and this is my first blog post in The Server Room. I’m the product line manager for the Intel Xeon processor 5000 family, and I’m based out of our Hillsboro, Oregon facility. I’ve been looking forward to this blog post for quite some time, since I’ve been meeting with a wide variety of customers over the past few weeks.

It’s been just over a month since we introduced the Intel Xeon processor 5500 series (the processor formerly known as “Nehalem-EP”). We are certainly pleased with the response from the industry at this point. Below you will see some of my observations about what has transpired over the first 30 days of release. At the same time, I invite you to share some of your stories about recent installations of the Xeon 5500. Where is it being used? What kind of environments are you using it in? What kind of improvements have you observed in your deployments?

The industry response has been extremely encouraging to me. Our marketing teams spent more than three years diligently preparing for the successful introduction. Some of my observations from the first month include:

·         The list of vendors that support the Xeon 5500 continues to grow. We started with over 70 system manufacturers on March 30, 2009. And on April 14, 2009, Sun Microsystems introduced a new line of x64 blade servers, rack servers and workstations powered by the Intel Xeon processor 5500 series. Of particular interest is the Sun Blade X6275 server module. You can find more info at: http://www.sun.com/solutions/hpc/compute.jsp.

·         I attended our launch event in Santa Clara on March 30, 2009. While at the event, I was pleasantly surprised by the adulation from the customers who were in attendance. In particular, our friends in the Digital Content Creation (DCC) industry are eager to apply the capabilities of the Xeon 5500 for movie special effects and animated features. Being a father of three school age children, I’ve always been fond of our products’ role in the moviemaking process. It’s fun to take your kids to the theater and show them a concrete example of how these incredibly complex processors are used to generate chuckles and special effects in movies ranging from “Cars” to “Monsters vs. Aliens.”

·         Positive recognition has been accorded to the Xeon 5500 from a wide variety of independent press reviewers and articles. A recent internet search revealed almost 875 news references. Recently, George Ou of DailyTech published an interesting article titled “Server roundup: Intel “Nehalem” Xeon versus AMD “Shanghai” Opteron”. You can read the entire article at: http://www.dailytech.com/article.aspx?newsid=15036

·         On May 4, 2009 two independent financial analysts upgraded Intel Corp. stock. Both analysts attributed part of their positive outlook to the introduction and ramp of Xeon 5500 servers.

·         On April 8, 2009 the new Xeon 5500 was a centerpiece of our IDF event in Beijing. In his enterprise key note, Pat Gelsinger said the “Nehalem” microarchitecture has received worldwide acclaim.

·         Customer deployments are underway at leading data centers around the globe – particularly in High Performance Computing (HPC) applications. The HPC accounts encompass university research labs, commercial research and development and large scale clusters. These HPC customers are pushing the outer limits of scientific discovery and innovation, and the best examples are yet to come!

Personally, I was proud to be a part of the introduction of the Xeon 5500. There is a strong sense of satisfaction when the silicon is deployed in real-world environments. And in case you hadn’t heard, we are busy getting ready for the next addition to the Xeon family, codenamed “Westmere-EP.” We expect this new 32nm processor to be socket and pin-compatible with the Xeon 5500, and it will stretch the processor to support six individual CPU cores per socket. Stay tuned for this release in 2010!