Just wrapped up Oracle Open World…sitting at SFO, waiting for a flight back home. 

The event from a Nehalem-EX perspective was a success. Hit important points and accomplished what we had to deliver. 

Hit #1:  Michael Dell, in his key note, delivered Nehalem-EX message beautifully.  2.5x performance improvements coming from 9x memory bandwidth…compared to currently sold technology.  Thank you, Michael. 

Hit #2:  Dell placed Nehalem-EX demo at its Exhibit at Moscone West.  I missed seeing it in person but the Dell friends came to me reporting that the demo attracted a lot of attention from the audience.  Thank you again, Dell. 

Hit #3:  My Nehalem-EX demo at Intel booth was also a success.  The pre-production system ran throughout the event with 64 logical processors fully active with 1TB of Samsung DDR3 memory, running SPECjbb, stressing all the CPUs, cores, and threads.  Occasionally, I injected double-bit error to show off the MCA-Recovery function.  Windows 2008 R2 reported nicely that the system encountered a critical error but the system still running at full speed.  If not with MCA-Recovery function, I would have had blue screen each time I ran that error injection script and would have had to wait for few minutes to have the server come back up online. 

Also, I really liked the demographics of the audience this time.  Compared to the other events I went to this year, I had more conversations with the folks who actually purchase equipments, those who test new equipments at IT shops, and those from Oracle starting to realize that hardware choice does matter when selling Oracle software.  Many people specifically asked when Intel starts shipping Nehalem-EX and which specific OEM models would use Nehalem-EX.  I hope my responses to those folks were legitimate.  ;-)  I also hope Oracle sales folks now have true confidence that the Oracle software stack runs best on Intel, specifically, Nehalem-EX. 

Oracle Open World is said to be the largest IT event.  I believe that.  You don’t get to have lunch at the middle Mission St tarmac very often.  You don’t get to see four digit hotel bills very often for just couple night stay. Despite the fall storm hitting the peninsula dumping loads of water and gust knocking trees down on Tuesday, Intel booth continuously had heavy flow of traffic.  I admire the Intel team putting together our presence and admire the whole industry supporting the event. I also personally learned a lot from the event, meeting people, exchanging knowledge.  Three day booth duty is a tough one but worth it. 

Oh, and to wrap the whole trip up…

Hit #4:  cleared the wait list and getting home earlier with an earlier flight… 

AND…I wish today was Friday… 

With the Intel Xeon 5500 series (Nehalem) based processors, the X5500 chipset and instrumented power supplies, you can start with the most basic use case for Intel Node Manager - monitoring the power usage of your servers.

As you can see in the Intel Datacenter Manager (DCM) screen below - there are multiple servers configured into logical units:  HF2-EIL is the lab that these servers are located in.  Rack 1 and Rack 2 are the physical location of these servers, and each Rack contains 2 servers each.

When you highlight one server (as above in DCM)- you can see the power characteristics over a certain time period.  The time period shown gives you the idle power, max power, and thermal measurement.  The 'hump' in the graph is a SQL workload which creates 'work' for the server and the process runs for about 5 1/2 minutes with no power capping.

Here's a graph of the 2nd server in that rack, performing a similar workload.  As you can see, the 2nd server power usage is different than the first.

The Intel Datacenter Manager SDK console can monitor multiple systems as well.  The next graph, is both of those servers in the rack, which accounts for both servers power usage during the same timeframe.

Finally, here is the final graph, showing the accumulation of all 4 servers, in both Rack #1 and Rack #2.  This shows the maximum power utilized during the workload, the minimum power (idle) and the inlet thermal temperature in the lab.  Something that hasn't been able to be done before without expensive equipment in the datacenter.

My next power based blog will show how power-capping can give you more effience use of your workload power while using Xeon 5500 series platforms.

Nehalem-EX has been in the news quite a bit over the past several months. 

First, in May, Intel described how Nehalem-EX will be at the heart of the next generation of intelligent and expandable high-end Intel server platforms, delivering a number of new technical advancements (Intel Nehalem Architecture, Quick Path Interconnects, 16 threads, 24MB cache, new RAS features like MCA-Recovery, 16 DIMM slots per socket, 128 threads on 8 Socket systems) and boost enterprise computing performance (the greatest gain in generational performance ever seen at Intel.)

Next at IDF in September Intel described how Nehalem-EX would deliver a bigger generational performance improvement than that delivered by the Intel Xeon 5500 processor (including a 3X Nehalem-EX gain in database performance); a large shift in Xeon scalability with over 15 >8S systems anticipated and expandability for the most data demanding enterprise applications, the addition of about 20 RAS capabilities traditionally found in the Intel® Itanium processor family – along with a demonstration of MCA-Recovery. IBM announced their upcoming BladeCenter products that will support 4S Nehalem-EX blades and Super-Micro announced a 1U box, specifically targeted at HPC.  Staying on the HPC theme, Mark Seager from the Lawrence Livermore National Laboratory was also quoted with stating that “Nehalem-EX allows us to invest in science, not the computer science of porting and adapting software to new architectures, but real science.  Nehalem EX is an innovative SMP on a chip solution that provides us access to a “super node” … The result is an astonishing new level of performance.”

And Oracle Open World on October 13^th, the drumbeat for Nehalem-EX continued.  Michael Dell in his Oracle Open World Keynote today discussed how Nehalem-EX will provide a true leap in performance, with up to 9x the memory bandwidth and 3x the database performance vs. prior generation.  And he mentioned that Dell’s unique implementation of the memory architecture will allow the most cost effective scaling, with 4S systems up to 1TB of DRAM (64 Dimms x 16GB Memory sticks) enabling customers to run their entire database in system memory.  He also mentioned that standard based systems are driving new efficiencies with applications like Oracle, where Dell’s data shows Oracle apps run better on x86 vs. proprietary architectures, up to 200% better.  Check out this short video from the keynote and watch what Michael Dell had to say. 

Keep your eyes on the Server Room for more Nehalem-EX news as it comes between now and launch.  And visit the Intel booth at South Moscone Booth #1621 to learn more.

Bryce

If you are attending Oracle Openworld 2009 and are interested in learining more about Oracle Database performance on Linux on Intel platforms then I will be talking on this topic in my session on Monday 12th Oct as follows:

Presenter:  Steve Shaw

ID#: S312645
Title: Oracle Database Performance on Linux: Tips, Tools, and Tuning for Intel Platforms
Track: Database
Date: 12-OCT-09
Time: 11:30 - 12:30
Venue: Moscone South
Room: Room 303

I will talking about the latest platform features, how you can get the best out of them on Linux  and also showing worked and customer examples of how these can benefit you with 'real world' Oracle installations.

Steve

Sun has recently published a whitepaper that discusses how the Solaris OS will take advantage of the next generation Intel Xeon processor (codename Nehalem-EX) for expandable servers (4 sockets & greater).  Sun with over 20 years of experience in larger socket, core & threading capabilities is working to have the Solaris OS be ready to take advantage of the features & new capabilities of “Nehalem-EX”.  The three areas of collaboration for Solaris & Nehalem-EX are around  scalable performance; advanced reliability and energy efficiency between the specific features in Solaris and the next generation Intel Xeon processor.  Read this recently published whitepaper

When it comes to your car or fixing something around the house, you know there’s a tool for every job right? Well, it’s no different when you’re considering transitioning from a desktop-based server to a real server. That’s why we created the Server Transition tool for all small businesses.  It’s easy to use (a few clicks), and gives you what you need to make a sound business decision.

Now you have a comprehensive tool to understand your server options and how your current system measures up. All you do is select the year you purchased your current system and the tool makes an initial guess at the configuration you have right now. You can make adjustments so it better matches your actual set-up and then, PRESTO! You’ll see how your system stacks up to a real server based on the processor, memory, storage, business applications and form factor. This way, you’re making an apples to apples comparison with quantifiable data.  Well, I guess since it’s a desktop versus a server it’s more of an apples to apple pie comparison and maybe even apple pie a la mode….

You can even take the comparison one step further by answering a few usage questions to find a more customized configuration for your business needs. It’s really that simple – a few clicks and you have a recommended server configuration that will deliver greater dependability, productivity and performance to meet your needs today and tomorrow. Now, if only there were tools like that to help my sister find the right guy to date.  Maybe that would be more of a tool avoidance tool, if you know what I mean.

Don’t wait, check it out now.

I’m a bit late in relaying my thoughts from Intel’s Developer Forum (IDF), but there was definitely some excitement around virtualization and high performance networking that I wanted to get the word out about!

In the past I’ve shared some details about SR-IOV and the advantages you can gain by being able to present virtual LAN hardware to each Virtual Machine (VM), effectively avoiding the Hypervisor when presenting virtual devices to each VM.  The advantage of being able to do this is clear:  The less interaction in the networking stack there is from the hypervisor, the less processing overhead is required for the system process the data.

That’s all good because if you have a dual 10 Gigabit adapter, you can segregate those two physical pipes into perhaps 16 virtual pipes that get exposed to 16 VMs.  By segregating these LAN pipes at the hardware level with SR-IOV instead of using Hypervisor switching, the performance gains in both CPU utilization as well as maximum total throughput can be very large.  There were several demos at IDF with various configurations, but reductions in CPU utilization of 40% were possible coupled with dramatic improvement in throughput!

But there is unfortunately one minor complication that I didn’t mention in my last post on the topic of SR-IOV.  There is the little fact that when VMs move between physical boxes (a usage that is highly desired and commonplace these days) you run into some problems with this SR-IOV capability.  When the hypervisor owned the network hardware abstraction, the performance was worse, but the functionality was better because you could seamlessly migrate from one box to another and the virtualization application would handle the details.  But with SR-IOV, a new layer needs to be added so that the direct hardware connection between the VM and the LAN hardware can be moved to a new box.

The really exciting part of IDF demos that I saw was the demonstration not just of the SR-IOV functionality on multiple hardware and virtualization configurations, but that these demonstrations also showed updated software from two virtualization vendors allowing mobility of the VMs while supporting SR-IOV! 

There was a demo on Dell systems showing this fully functional SR-IOV implementation with Citrix’s Virtualization suite.  There were two separate demonstrations on Dell systems, with VMWare displaying their new Network Plug-In Architecture (NPIA) solution that allows for the migration of SR-IOV connected VMs seamlessly between servers.

For those hungry for more detail, I’ve included the three SR-IOV demonstration videos here:

The first is the Citrix demonstration on Dell and Intel hardware of SR-IOV with VM mobility:

These next two are two videos are demos on Dell and Intel hardware with VMWare and their NPIA software implementation.

Each virtualization demo shows the massive performance benefits under various workloads when moving from Hypervisor based LAN segregation to SR-IOV implementation.  But most importantly, each demonstration proves out the capability to migrate VMs between physical hardware.  The only system hardware requirement is that the server itself supports VT-d.  If the networking hardware in the newly migrated-to box supports SR-IOV you get better performance, and if not, the solution falls back on the legacy Hypervisor virtualization.  Backwards compatibility is maintained!

I didn’t get firm details on when this full support for SR-IOV and migration will be available in Citrix and VMWare’s releases, but the demos looked pretty clean, and hopefully these suites will be available soon with this new functionality.  The LAN and Server hardware ecosystems are ready today, and it looks like the software vendors are just around the corner.  Virtualization momentum continues!

While virtualization was the big takeaway for me from IDF, there were also several other interesting demos for us networking hounds.  I’ve linked a couple videos of them below for anyone still thirsting for more of the latest networking technology and performance details!

The first video is a demonstration of Intel’s 82599 10 Gigabit Ethernet-based adapter card with Fiber Channel over Ethernet (FCoE) support.  Storage and Ethernet together at last!

The second video is a demonstration of Intel’s NetEffect 10 Gigabit Ethernet card publishing 1 million messages per second in a simulated NYSE floor trading scenario.  Oh yeah, only 35uS of latency.  That is fast.

So although I am two weeks after IDF, I hope some of you got a little taste of the networking excitement that took place.   Industry wide, hardware and software vendors alike are delivering ultra high performance low latency applications for the financial services industry, as well as mainstream performance increases for virtualization.  The performance and technology beat moves forward.  Exciting times!

--

Ben Hacker

I wanted to follow-up on the pre-IDF blog I wrote and what I and Sean conveyed regarding comprehensive IO optimization for enterprise cloud (based on virtualization infrastructure). A blog I owed to those who could not attend the IDF session.

In the last blog we identified 4 important vectors that drive I/O evolution.

1)      Balanced system that maps to the increases in CPU performance

2)      Scalability

3)      Unified fabric

4)      Security

In my view I state it an evolution as I feel that is the natural state things will head towards in the (near) future.

In a cloud environment you would anticipate automation and policies determine the consolidation possible on a system. If SSDs get broader adoption and virtualization performance increases due to hardware assists, I/O and fabric could become the bottleneck for the degree of consolidation and efficiency as it cannot map to the increased data rates from the storage and CPU performance.

Ways to address this is either to reduce or eliminate the overheads in the I/O stack caused by software emulation of devices in the VMM. VMDq is an example of a technology that can reduce the overhead or offload some of the VMM tasks through hardware assists in the NIC. Direct assignment with PCI-SIG SR-IOV support is a way to eliminate the overheads by by-passing the VMM. With SR-IOV, a single device can be divided into many logical devices known as Virtual Functions (like a pair or independent transmit receive queue). Each virtual function can be directly assigned to a VM using Intel VT-d thereby bypassing the VMM. This can work with Live VM Migration too. At IDF we showcased 4 demos of prototype SR-IOV software solutions running on Intel Xeon 5500 based hardware with prominent VMM vendors like VMware, Citrix and Redhat that have different hypervisor technology. The networking demos showcased working live migration with SR-IOV and VT-d based direct assignment. Direct assigned VMs could be even moved to an emulated mode and brought back to direct assigned mode. Intel has not only been working with software providers but also with other hardware vendors like LSI to demonstrate this capability. These technologies are as important to storage as networking particularly as SSDs gather steam. You can learn more from these blogs below on the demos.

Demo with Intel Xeon 5500 based Dell servers

An analyst view on the LSI solution demonstrated

If those multi 1GbE cables (that make your fabric look like pasta) be replaced by 10GbE and if SR-IOV and VT-d be used for performance, then it answers both the I/O performance and scalability requirements for a flexible datacenter.

Beyond those VT-d provides greater protection by allowing I/O devices to access only the memory regions allocated to them, and SR-IOV allows VMs to access only their portion of the device and restricts access to other Virtual Functions (owned by other VMs) on the I/O device or the entire device itself. Better security through better isolation.

Last but not the least of the requirements is the unified fabric. When IT can use a single I/O device for storage or for LAN traffic, the rigidity associated with provisioning of servers with some number of HBAs and some number of NICs is reduced. The I/O capacity becomes fungible and flexible. FCoE and iSCSI are key technologies in this direction. Adding capability to monitor QoS and shape the traffic makes it a good match for flexible datacenter.

Many of the technologies I discussed above (VT-d, SRIOV, FCoE, iSCSI) are here today… software such as Red Hat Enterprise Linux is already delivering the solution. In my view just a matter of time that ecosystem builds further and hardware is well tuned.

With these in perspective how do you see your datacenter shaping up?