I’m a bit late in relaying my thoughts from Intel’s Developer Forum (IDF), but there was definitely some excitement around virtualization and high performance networking that I wanted to get the word out about!

In the past I’ve shared some details about SR-IOV and the advantages you can gain by being able to present virtual LAN hardware to each Virtual Machine (VM), effectively avoiding the Hypervisor when presenting virtual devices to each VM.  The advantage of being able to do this is clear:  The less interaction in the networking stack there is from the hypervisor, the less processing overhead is required for the system process the data.

That’s all good because if you have a dual 10 Gigabit adapter, you can segregate those two physical pipes into perhaps 16 virtual pipes that get exposed to 16 VMs.  By segregating these LAN pipes at the hardware level with SR-IOV instead of using Hypervisor switching, the performance gains in both CPU utilization as well as maximum total throughput can be very large.  There were several demos at IDF with various configurations, but reductions in CPU utilization of 40% were possible coupled with dramatic improvement in throughput!

But there is unfortunately one minor complication that I didn’t mention in my last post on the topic of SR-IOV.  There is the little fact that when VMs move between physical boxes (a usage that is highly desired and commonplace these days) you run into some problems with this SR-IOV capability.  When the hypervisor owned the network hardware abstraction, the performance was worse, but the functionality was better because you could seamlessly migrate from one box to another and the virtualization application would handle the details.  But with SR-IOV, a new layer needs to be added so that the direct hardware connection between the VM and the LAN hardware can be moved to a new box.

The really exciting part of IDF demos that I saw was the demonstration not just of the SR-IOV functionality on multiple hardware and virtualization configurations, but that these demonstrations also showed updated software from two virtualization vendors allowing mobility of the VMs while supporting SR-IOV! 

There was a demo on Dell systems showing this fully functional SR-IOV implementation with Citrix’s Virtualization suite.  There were two separate demonstrations on Dell systems, with VMWare displaying their new Network Plug-In Architecture (NPIA) solution that allows for the migration of SR-IOV connected VMs seamlessly between servers.

For those hungry for more detail, I’ve included the three SR-IOV demonstration videos here:

The first is the Citrix demonstration on Dell and Intel hardware of SR-IOV with VM mobility:

These next two are two videos are demos on Dell and Intel hardware with VMWare and their NPIA software implementation.

Each virtualization demo shows the massive performance benefits under various workloads when moving from Hypervisor based LAN segregation to SR-IOV implementation.  But most importantly, each demonstration proves out the capability to migrate VMs between physical hardware.  The only system hardware requirement is that the server itself supports VT-d.  If the networking hardware in the newly migrated-to box supports SR-IOV you get better performance, and if not, the solution falls back on the legacy Hypervisor virtualization.  Backwards compatibility is maintained!

I didn’t get firm details on when this full support for SR-IOV and migration will be available in Citrix and VMWare’s releases, but the demos looked pretty clean, and hopefully these suites will be available soon with this new functionality.  The LAN and Server hardware ecosystems are ready today, and it looks like the software vendors are just around the corner.  Virtualization momentum continues!

While virtualization was the big takeaway for me from IDF, there were also several other interesting demos for us networking hounds.  I’ve linked a couple videos of them below for anyone still thirsting for more of the latest networking technology and performance details!

The first video is a demonstration of Intel’s 82599 10 Gigabit Ethernet-based adapter card with Fiber Channel over Ethernet (FCoE) support.  Storage and Ethernet together at last!

The second video is a demonstration of Intel’s NetEffect 10 Gigabit Ethernet card publishing 1 million messages per second in a simulated NYSE floor trading scenario.  Oh yeah, only 35uS of latency.  That is fast.

So although I am two weeks after IDF, I hope some of you got a little taste of the networking excitement that took place.   Industry wide, hardware and software vendors alike are delivering ultra high performance low latency applications for the financial services industry, as well as mainstream performance increases for virtualization.  The performance and technology beat moves forward.  Exciting times!

--

Ben Hacker

I’ve spent a fair number of words in the past on the benefits of 10 Gigabit and what it means for the server market.  Through the addition of FCoE and DataCenter Ethernet as well as advanced virtualization features 10 Gigabit seems likely to have its big day in the sun here pretty soon.  But the question is still “When”?

While the proof is ultimately in the raw volumes of 10 Gigabit that ship, and the number of IT users who utilize the higher performance, there are some key reasons to think that 10 Gigabit momentum is accelerating beyond just the numbers* below:

Over the past year, there has been a raft of new 10 Gigabit switch announcements** from Cisco (Nexus 5k/7k), Arista (7100, 7124, and 7148), BNT (G8100), Extreme Networks (Summit X650) Juniper (EX8200), Voltaire (8500) and many others that have increased the choice, and the density of 10 Gigabit switches in the marketplace.   There are now many 48+ port 10 Gigabit switches available and even a few 200+ port models.  Also, the improved density and feature set of certain switches (such as Voltaire’s 280+ port 8500 series switch) provide a path for 10 Gigabit’s ascent into the clustering market by improving port density and latency for clustering applications.

Broad acceptance of SFP+ has also helped to drive a rapid improvement in price, density, and power.  SFP+ provides a smaller form factor standard for optics, as well as a standard connection methodology to connect directly from switch to NIC via a Twin-Ax copper (read: ‘low cost’) cabling solution inside the rack (up to 10m).  The widespread adoption of SFP+ form factors has dramatically reduced the entry level price points for switches, and through the ‘direct attach’ copper connection capability it has also reduced the overall cost for initial and ongoing deployments of 10 Gigabit by providing a lower cost bridge to optical or full 10GBase-T support.

There are also a few data points to suggest that the Server side cost for 10 Gigabit will also be dropping fast going forward.  As power for 10GBase-T continues to drop quickly, more and more Server vendors are looking at the options available to embedded 10 Gigabit directly into their systems.  This will not likely be a 2009 story, but it is approaching quickly.  Additionally, the acceptance of SFP+ form factors for optics/direct attach cabling has provided a path that some Server vendors may use to design 10 Gigabit down on motherboards without adding the extra cost and power of a 10GBase-T solution.  This looks like a likely near term given that the solution power and design are robust and ready for motherboard based designs today.

Finally, the continued cost reduction provides an attractive long term value of standards based 10 Gigabit Ethernet.  There is clear indication downward pressure on 10GbE prices already present today.  We will see 10 Gigabit pricing follow a similar price curve as we saw with Single Gigabit.  This is evidenced in the recent pricing announcement where Intel reduced the cost of single port 10GBASE-T adapter 40% from $999 to $599.  The competitive economics of standards based hardware will continue to drive down 10 Gigabit prices even further and we will see 10GBASE-T pricing below the $500 / port price in the near future. Once it gets on the motherboard, prices will drop even further.

Overall, the power, density, latency, and cost of 10 Gigabit are all improving at a rapid rate.  Form factor flexibility coupled with a wide array of switch and NIC vendors in the marketplace will provide choice and low cost for IT departments while virtualization and convergence in the datacenter and elsewhere continue to provide demands for ever greater I/O bandwidth and performance.

Sometimes the next step up is a big one. The Intel® Xeon® processor 5500 series (formerly codenamed “Nehalem”) is one of those kinds of steps.

Over the last few years 10 Gigabit has started to take off, but there have always been some negative mutterings: “Why do I need 10 Gigabit?”, “Why do we need this much bandwidth?” or “My server can’t support 10 Gigabit per second bidirectional traffic anyway.” Despite the volume of 10 Gigabit products shipped, there is still the reality that if you intend to use the entire 20 Gbps (10G both directions) or heaven forbid you try to use 40 Gbps with a dual port product; you will likely be sorely disappointed with the results.

The reason for this is simple. Most current mainstream servers and 10 Gigabit products don’t support the intense usage models needed to drive that much network I/O and they also don’t have the memory architecture to unleash the full potential of dual 10 Gigabit links.

Luckily, that all just changed with Intel® Xeon® processor 5500 series.

In addition to the great processing improvements that the Intel® Xeon® processor 5500 series brings to the table, Intel has also introduced our third generation 10 Gigabit product, the Intel® 82599 10 Gigabit Ethernet Controller which provides two ports, and new capabilities and enhancement to the 10 Gigabit product landscape that help unshackle the new processor from its predecessor’s network I/O handcuffs and unleashes blazing performance in a variety of usage models. These improvements, coupled with the new architecture of the Xeon 5500 provide a symbiotic processor-networking combination that makes new usages possible and expands server and datacenter computing by a big leap… not just a baby step.

One of the key changes with Intel® Xeon® processor 5500 series architecture is a step function improvement in the internal system I/O. The new local memory controller design, faster cache architecture, and support for DDR3 help push Xeon 5500 to be able to support peak memory bandwidth of ~32 Gigabytes, per socket. In a dual socket system this provides for ~64 Gigabytes of bandwidth which is dramatically more than the previous generation server configuration. In addition, the new Intel® QuickPath Interconnect (Intel® QPI) improves the speed both for inter-Processor communication as well as a faster path to the I/O hub. Finally, PCI Express* 2.0 I/O Bus support has been added to improve the entire data path from Processor to the 10 Gigabit Ethernet link.

Taken together, the above improvements are a performance game changer for 10 Gigabit Ethernet.

The chart below** shows the previous generation Intel® Xeon® paired with the previous generation Intel 10 Gigabit Ethernet Controller compared to the latest platform using the newest Intel silicon for both processor and networking. Not only is the performance better in 1-4 port configurations, but the performance scales dramatically better to above 50 Gigabits per second of total LAN throughput in a four port configuration vs. *just* 17 Gigabits on the previous generation! A complete platform architecture solution makes this huge improvement possible.

Now, it’s great that Intel® Xeon® processor 5500 series coupled with the Intel® 82599 10 Gigabit Ethernet can deliver such raw performance, but there is the forever nagging question of usage model. Luckily, the new headroom breathes new life into both Virtualization and storage over Ethernet usages (both of which I’ve talked about here and here) and provides new opportunities to more efficiently utilize your network link.

Intel® Xeon® processor 5500 series allows the vision of consolidation in the datacenter to scale new heights, increasing the number of Virtual Machines (VM) that can effectively live inside a single system enclosure. Each incremental VM will add additional network I/O that is already starting to exhaust a 4 or 8 port single gigabit interface configuration with today’s server capabilities. As more VMs are added in the Xeon 5500 generation, 10 Gigabit will no longer be seen as optional; it will be required. For its part, the Intel® 82599 10 Gigabit Ethernet Controller supports Intel® Virtualization Technology for Connectivity (Intel® VT-c) to improve overall system performance in virtualized server environments. Intel VT- c includes hardware optimizations that help reduce I/O bottlenecks, boost throughput and reduce latency. Components of Intel VT-c include VMDq, and VMDc. VMDc consists of SR-IOV which I’ve covered before, and the ability to support VM mobility; a critical usage model for modern a IT deployment. All together, server systems can support more VMs, more throughput, more flexibility and better performance in a datacenter environment.

Finally, the additional capabilities of the Intel® 82599 10 Gigabit Ethernet Controller product surrounding support for FCoE offloads and full support for the new Data Center Bridging (DCB) standards provide an opportunity for storage convergence over Ethernet in either a datacenter using a Fiber Channel SAN environment or an IT environment more focused on iSCSI. On the performance side of things, iSCSI acceleration along with FCoE data path offloads are supported in the Ethernet controller, and on the processor there is support for the CRC instruction set which insures iSCSI data integrity while minimizing processor overhead.

The ability to converge at least part of the additional storage infrastructure onto Ethernet is just another factor driving massively increased data rates over Ethernet… luckily, the Intel® Xeon® processor 5500 series and the Intel® 82599 10 Gigabit Ethernet Controller solutions are up to the task.

Over the past few days, there has been a lot of noise around Intel® Xeon® processor 5500 series and the many other platform components that help it shine its brightest. Improved processing power, memory controller bandwidth, faster and redesigned FSB, and improved 10 Gigabit networking all converge together to provide a fantastic performance, convergence, scalability and power story. Intel’s strong history in the server and processor markets, coupled with over 25 years in Ethernet makes this latest release a natural evolution of technology. Together these capabilities, along with the improved 10 Gigabit features and performance, are helping to transform the datacenter. It will be denser, more power efficient, more performant, and more consolidated with capabilities like FCoE and iSCSI.

As for “Why do I need 10 Gigabit?” We have the answer, and it’s the new Xeon®.

Ben Hacker

-----

** Source. Intel. Mar 2009. Up to 2.5x performance compared to Intel® Xeon® processor 5300 series. Performance result of a bandwidth intensive network benchmark (IxChariot). Network throughput was measured on 64KB I/O size transfers between the test system and multiple network targets. Intel pre-production system with two Quad-Core Intel® Xeon® processor 5500 series CPUs (2.93 GHz), 12 GB memory (6 x2GB DDR3 - 1066MHz) vs. Intel Production system with two Intel® Xeon® processors X5365 (3.0GHz, 1333MHz FSB), 8 GB memory (8 x 1 GB DDR 2 - 667). Windows Server 2008, stock unmodified installation.

Day 1: I'm live from VMWorld this week experiencing the virtualization event of the year. I'll be updating this blog with happenings from the Intel booth and around the show floor. Some really cool video interviews with Intel Partners who are making a big impact in the virtualization world and giving IT managers real advantages over previous generation solutions. Here's a video showing the XEON 7400 near perfect scalability from 8 to 24 to 48 cores. Wow, 48 cores, that's cool!

If you liked the first video, check out this one where Jon Markee is talking about Intel Virtualization Technology (VT) and how flex priority improves performance and reduces boot time in your virtualized environment.

Day 2: Here's another video from the Intel Booth showing more examples of Intel Virtualization technology.