I’m a bit late in relaying my thoughts from Intel’s Developer Forum (IDF), but there was definitely some excitement around virtualization and high performance networking that I wanted to get the word out about!

In the past I’ve shared some details about SR-IOV and the advantages you can gain by being able to present virtual LAN hardware to each Virtual Machine (VM), effectively avoiding the Hypervisor when presenting virtual devices to each VM.  The advantage of being able to do this is clear:  The less interaction in the networking stack there is from the hypervisor, the less processing overhead is required for the system process the data.

That’s all good because if you have a dual 10 Gigabit adapter, you can segregate those two physical pipes into perhaps 16 virtual pipes that get exposed to 16 VMs.  By segregating these LAN pipes at the hardware level with SR-IOV instead of using Hypervisor switching, the performance gains in both CPU utilization as well as maximum total throughput can be very large.  There were several demos at IDF with various configurations, but reductions in CPU utilization of 40% were possible coupled with dramatic improvement in throughput!

But there is unfortunately one minor complication that I didn’t mention in my last post on the topic of SR-IOV.  There is the little fact that when VMs move between physical boxes (a usage that is highly desired and commonplace these days) you run into some problems with this SR-IOV capability.  When the hypervisor owned the network hardware abstraction, the performance was worse, but the functionality was better because you could seamlessly migrate from one box to another and the virtualization application would handle the details.  But with SR-IOV, a new layer needs to be added so that the direct hardware connection between the VM and the LAN hardware can be moved to a new box.

The really exciting part of IDF demos that I saw was the demonstration not just of the SR-IOV functionality on multiple hardware and virtualization configurations, but that these demonstrations also showed updated software from two virtualization vendors allowing mobility of the VMs while supporting SR-IOV! 

There was a demo on Dell systems showing this fully functional SR-IOV implementation with Citrix’s Virtualization suite.  There were two separate demonstrations on Dell systems, with VMWare displaying their new Network Plug-In Architecture (NPIA) solution that allows for the migration of SR-IOV connected VMs seamlessly between servers.

For those hungry for more detail, I’ve included the three SR-IOV demonstration videos here:

The first is the Citrix demonstration on Dell and Intel hardware of SR-IOV with VM mobility:

These next two are two videos are demos on Dell and Intel hardware with VMWare and their NPIA software implementation.

Each virtualization demo shows the massive performance benefits under various workloads when moving from Hypervisor based LAN segregation to SR-IOV implementation.  But most importantly, each demonstration proves out the capability to migrate VMs between physical hardware.  The only system hardware requirement is that the server itself supports VT-d.  If the networking hardware in the newly migrated-to box supports SR-IOV you get better performance, and if not, the solution falls back on the legacy Hypervisor virtualization.  Backwards compatibility is maintained!

I didn’t get firm details on when this full support for SR-IOV and migration will be available in Citrix and VMWare’s releases, but the demos looked pretty clean, and hopefully these suites will be available soon with this new functionality.  The LAN and Server hardware ecosystems are ready today, and it looks like the software vendors are just around the corner.  Virtualization momentum continues!

While virtualization was the big takeaway for me from IDF, there were also several other interesting demos for us networking hounds.  I’ve linked a couple videos of them below for anyone still thirsting for more of the latest networking technology and performance details!

The first video is a demonstration of Intel’s 82599 10 Gigabit Ethernet-based adapter card with Fiber Channel over Ethernet (FCoE) support.  Storage and Ethernet together at last!

The second video is a demonstration of Intel’s NetEffect 10 Gigabit Ethernet card publishing 1 million messages per second in a simulated NYSE floor trading scenario.  Oh yeah, only 35uS of latency.  That is fast.

So although I am two weeks after IDF, I hope some of you got a little taste of the networking excitement that took place.   Industry wide, hardware and software vendors alike are delivering ultra high performance low latency applications for the financial services industry, as well as mainstream performance increases for virtualization.  The performance and technology beat moves forward.  Exciting times!

--

Ben Hacker

I’ve spent a fair number of words in the past on the benefits of 10 Gigabit and what it means for the server market.  Through the addition of FCoE and DataCenter Ethernet as well as advanced virtualization features 10 Gigabit seems likely to have its big day in the sun here pretty soon.  But the question is still “When”?

While the proof is ultimately in the raw volumes of 10 Gigabit that ship, and the number of IT users who utilize the higher performance, there are some key reasons to think that 10 Gigabit momentum is accelerating beyond just the numbers* below:

Over the past year, there has been a raft of new 10 Gigabit switch announcements** from Cisco (Nexus 5k/7k), Arista (7100, 7124, and 7148), BNT (G8100), Extreme Networks (Summit X650) Juniper (EX8200), Voltaire (8500) and many others that have increased the choice, and the density of 10 Gigabit switches in the marketplace.   There are now many 48+ port 10 Gigabit switches available and even a few 200+ port models.  Also, the improved density and feature set of certain switches (such as Voltaire’s 280+ port 8500 series switch) provide a path for 10 Gigabit’s ascent into the clustering market by improving port density and latency for clustering applications.

Broad acceptance of SFP+ has also helped to drive a rapid improvement in price, density, and power.  SFP+ provides a smaller form factor standard for optics, as well as a standard connection methodology to connect directly from switch to NIC via a Twin-Ax copper (read: ‘low cost’) cabling solution inside the rack (up to 10m).  The widespread adoption of SFP+ form factors has dramatically reduced the entry level price points for switches, and through the ‘direct attach’ copper connection capability it has also reduced the overall cost for initial and ongoing deployments of 10 Gigabit by providing a lower cost bridge to optical or full 10GBase-T support.

There are also a few data points to suggest that the Server side cost for 10 Gigabit will also be dropping fast going forward.  As power for 10GBase-T continues to drop quickly, more and more Server vendors are looking at the options available to embedded 10 Gigabit directly into their systems.  This will not likely be a 2009 story, but it is approaching quickly.  Additionally, the acceptance of SFP+ form factors for optics/direct attach cabling has provided a path that some Server vendors may use to design 10 Gigabit down on motherboards without adding the extra cost and power of a 10GBase-T solution.  This looks like a likely near term given that the solution power and design are robust and ready for motherboard based designs today.

Finally, the continued cost reduction provides an attractive long term value of standards based 10 Gigabit Ethernet.  There is clear indication downward pressure on 10GbE prices already present today.  We will see 10 Gigabit pricing follow a similar price curve as we saw with Single Gigabit.  This is evidenced in the recent pricing announcement where Intel reduced the cost of single port 10GBASE-T adapter 40% from $999 to $599.  The competitive economics of standards based hardware will continue to drive down 10 Gigabit prices even further and we will see 10GBASE-T pricing below the $500 / port price in the near future. Once it gets on the motherboard, prices will drop even further.

Overall, the power, density, latency, and cost of 10 Gigabit are all improving at a rapid rate.  Form factor flexibility coupled with a wide array of switch and NIC vendors in the marketplace will provide choice and low cost for IT departments while virtualization and convergence in the datacenter and elsewhere continue to provide demands for ever greater I/O bandwidth and performance.

Sometimes the next step up is a big one. The Intel® Xeon® processor 5500 series (formerly codenamed “Nehalem”) is one of those kinds of steps.

Over the last few years 10 Gigabit has started to take off, but there have always been some negative mutterings: “Why do I need 10 Gigabit?”, “Why do we need this much bandwidth?” or “My server can’t support 10 Gigabit per second bidirectional traffic anyway.” Despite the volume of 10 Gigabit products shipped, there is still the reality that if you intend to use the entire 20 Gbps (10G both directions) or heaven forbid you try to use 40 Gbps with a dual port product; you will likely be sorely disappointed with the results.

The reason for this is simple. Most current mainstream servers and 10 Gigabit products don’t support the intense usage models needed to drive that much network I/O and they also don’t have the memory architecture to unleash the full potential of dual 10 Gigabit links.

Luckily, that all just changed with Intel® Xeon® processor 5500 series.

In addition to the great processing improvements that the Intel® Xeon® processor 5500 series brings to the table, Intel has also introduced our third generation 10 Gigabit product, the Intel® 82599 10 Gigabit Ethernet Controller which provides two ports, and new capabilities and enhancement to the 10 Gigabit product landscape that help unshackle the new processor from its predecessor’s network I/O handcuffs and unleashes blazing performance in a variety of usage models. These improvements, coupled with the new architecture of the Xeon 5500 provide a symbiotic processor-networking combination that makes new usages possible and expands server and datacenter computing by a big leap… not just a baby step.

One of the key changes with Intel® Xeon® processor 5500 series architecture is a step function improvement in the internal system I/O. The new local memory controller design, faster cache architecture, and support for DDR3 help push Xeon 5500 to be able to support peak memory bandwidth of ~32 Gigabytes, per socket. In a dual socket system this provides for ~64 Gigabytes of bandwidth which is dramatically more than the previous generation server configuration. In addition, the new Intel® QuickPath Interconnect (Intel® QPI) improves the speed both for inter-Processor communication as well as a faster path to the I/O hub. Finally, PCI Express* 2.0 I/O Bus support has been added to improve the entire data path from Processor to the 10 Gigabit Ethernet link.

Taken together, the above improvements are a performance game changer for 10 Gigabit Ethernet.

The chart below** shows the previous generation Intel® Xeon® paired with the previous generation Intel 10 Gigabit Ethernet Controller compared to the latest platform using the newest Intel silicon for both processor and networking. Not only is the performance better in 1-4 port configurations, but the performance scales dramatically better to above 50 Gigabits per second of total LAN throughput in a four port configuration vs. *just* 17 Gigabits on the previous generation! A complete platform architecture solution makes this huge improvement possible.

Now, it’s great that Intel® Xeon® processor 5500 series coupled with the Intel® 82599 10 Gigabit Ethernet can deliver such raw performance, but there is the forever nagging question of usage model. Luckily, the new headroom breathes new life into both Virtualization and storage over Ethernet usages (both of which I’ve talked about here and here) and provides new opportunities to more efficiently utilize your network link.

Intel® Xeon® processor 5500 series allows the vision of consolidation in the datacenter to scale new heights, increasing the number of Virtual Machines (VM) that can effectively live inside a single system enclosure. Each incremental VM will add additional network I/O that is already starting to exhaust a 4 or 8 port single gigabit interface configuration with today’s server capabilities. As more VMs are added in the Xeon 5500 generation, 10 Gigabit will no longer be seen as optional; it will be required. For its part, the Intel® 82599 10 Gigabit Ethernet Controller supports Intel® Virtualization Technology for Connectivity (Intel® VT-c) to improve overall system performance in virtualized server environments. Intel VT- c includes hardware optimizations that help reduce I/O bottlenecks, boost throughput and reduce latency. Components of Intel VT-c include VMDq, and VMDc. VMDc consists of SR-IOV which I’ve covered before, and the ability to support VM mobility; a critical usage model for modern a IT deployment. All together, server systems can support more VMs, more throughput, more flexibility and better performance in a datacenter environment.

Finally, the additional capabilities of the Intel® 82599 10 Gigabit Ethernet Controller product surrounding support for FCoE offloads and full support for the new Data Center Bridging (DCB) standards provide an opportunity for storage convergence over Ethernet in either a datacenter using a Fiber Channel SAN environment or an IT environment more focused on iSCSI. On the performance side of things, iSCSI acceleration along with FCoE data path offloads are supported in the Ethernet controller, and on the processor there is support for the CRC instruction set which insures iSCSI data integrity while minimizing processor overhead.

The ability to converge at least part of the additional storage infrastructure onto Ethernet is just another factor driving massively increased data rates over Ethernet… luckily, the Intel® Xeon® processor 5500 series and the Intel® 82599 10 Gigabit Ethernet Controller solutions are up to the task.

Over the past few days, there has been a lot of noise around Intel® Xeon® processor 5500 series and the many other platform components that help it shine its brightest. Improved processing power, memory controller bandwidth, faster and redesigned FSB, and improved 10 Gigabit networking all converge together to provide a fantastic performance, convergence, scalability and power story. Intel’s strong history in the server and processor markets, coupled with over 25 years in Ethernet makes this latest release a natural evolution of technology. Together these capabilities, along with the improved 10 Gigabit features and performance, are helping to transform the datacenter. It will be denser, more power efficient, more performant, and more consolidated with capabilities like FCoE and iSCSI.

As for “Why do I need 10 Gigabit?” We have the answer, and it’s the new Xeon®.

Ben Hacker

-----

** Source. Intel. Mar 2009. Up to 2.5x performance compared to Intel® Xeon® processor 5300 series. Performance result of a bandwidth intensive network benchmark (IxChariot). Network throughput was measured on 64KB I/O size transfers between the test system and multiple network targets. Intel pre-production system with two Quad-Core Intel® Xeon® processor 5500 series CPUs (2.93 GHz), 12 GB memory (6 x2GB DDR3 - 1066MHz) vs. Intel Production system with two Intel® Xeon® processors X5365 (3.0GHz, 1333MHz FSB), 8 GB memory (8 x 1 GB DDR 2 - 667). Windows Server 2008, stock unmodified installation.

It is inevitable that as processing power increases, and cores multiply on servers, that new bottlenecks will be found. If you make a processor that can do more, you will increase your system performance until you find where the next area of focus is. I’ve talked on several occasions about the system bottlenecks that are now apparent by using the latest Xeon processors or when a many virtual machines are loaded onto a single physical box, and what can be done to improve them. As you may have noticed, a key laggard that will frequently be found in modern multi-processor servers is the network connection. Moving to 10 Gigabit clearly alleviates this issue in many high horsepower systems, and the addition of advanced features like VMDq and SR-IOV will continue to push the envelope on Network I/O in virtualized environments.

However, it is clear that for some applications, especially in the HPC market and certain applications in the Financial Service industry, an even more performing I/O solution than standard 10 Gigabit will be needed when latency is of the utmost importance. A solution that has existed in this space for a few years, but is gaining momentum is what is referred to as Remote Direct Memory Access (RDMA), which lets one server directly place information into the memory of another server; essentially bypassing the kernel and networking software stack.

At first blush, this solution may sound odd. Why bypass the entire stack; doesn’t this complicate things? Well, it certainly does add some complications by requiring a modified OS stack and support on both sides of the network link, but there are some telling details about where real world latencies come from that make this methodology attractive in certain circumstances.

If you look at the typical breakdown of CPU utilization in the context of processing networking traffic, you see the workload consumed by buffer copies, application context switching and some TCP/IP processing. If you look at this (albeit, in a simplified way) visually, there is a vertical stack of tasks that need to take place in the server:

There are application buffer copies from the app to kernel which then get handled by the NIC. Additionally, there is the TCP/IP processing that takes place within the OS and is a large consumer of CPU cycles, and there are also I/O commands that add additional latencies into the communication process.

So the question is what to do to help reduce these latencies? RDMA has been adapted for standard Ethernet via the IETF Internet Wide Area RDMA Protocol (iWARP). Until the iWARP specification, RDMA had been a capability only seen using Infiniband networking. By porting the goodness of RDMA to Ethernet, iWARP offers the promise of ultra low latency, but with all the benefits of standard Ethernet.

With Intel’s acquisition in October of NetEffect, we now have one of the leading iWARP product lines for 10 Gigabit Ethernet. Within these products, the iWARP processing engine can help eliminate some of the key bottlenecks described above, and provide very low latency and high performance networking for even the most demanding HPC applications.

The first item that an iWARP engine can address is to help offload some of the TCP processing task which can bog down processing power as bandwidth loads increase. The Intel NetEffect iWARP solution can offload this TCP processing by handling the sequencing, payload reassembly, and buffer management in dedicated hardware.

The next item that iWARP addresses is the extra copies that need to be done by the system when transferring data. iWARP extensions for RDMA and Direct Data Placement (DDP) allow the iWARP engine to tag the data with the necessary application buffer information and place the payload directly in the target Server’s memory. This eliminates the delays associated with memory copies by moving to a so called ‘zero copy’ model.

Finally, iWARP extensions also implement user-level direct access which allows a user-space application to post commands directly to the network adapter without having to make latency consuming calls to the OS for requests. This along with the other pieces of iWARP provides dramatically reduced latency and increased performance.

The diagram below summarizes what the new system stack looks like after the implementation of iWARP. Much simpler and which much lower latency.

One obvious issue that is raised after thinking about the above diagram is what modifications need to be made at the OS or application level. Clearly applications need to be modified to be iWARP compatible and this can be a time consuming process. This is one of the reasons that this solution has been slow to gain adoption. However, there is an Open Fabrics Alliance (OFA) which is working on a unified stack for RDMA for the open source community. The OFA has an Open Fabrics Enterprise Distribution (OFED) release which is a set of libraries and code that can unify different solutions that use RDMA. There have been several OFED releases so far and further plans are coming to align and expand various RDMA capabilities. In this way, applications that run using the OFED stack under Infiniband can be run without any changes over iWARP and Ethernet.

As more applications get modified to support the feature set of iWARP RDMA, there will be a wider understanding and acceptance in the HPC community of the incremental performance, cost, and standards advantages of using Ethernet with RDMA for the most performance sensitive applications. Moving from standard non-iWARP Ethernet to iWARP enabled Ethernet provide more bounded latency reduction from ~14 us to <<10 us… now that is fast.

We live in exciting times!

Ben Hacker

--

For those looking for some more detail, there is a nice whitepaper on iWARP located here.

For those interested in learning more about the Open Fabrics Alliance (OFA) please see here.

If there is one thing that has stayed consistent in the computing industry over time, it's that performance doesn't stand still. As our computing platform processing, I/O, and memory speeds continue to accelerate, it is important to remember a little thing called latency.

Often in the Ethernet world throughput is the 1^st and last performance metric of choice. 1 Gigabit and 10 Gigabit are the numbers that inspire thoughts of increased performance, and improved computing power. However, it's important to note that, in many applications, the transaction latency over the wire is really the key to unlocking high performance at the system level. One of the primary reasons that some organizations have turned to Infiniband and other I/O technologies for HPC and clustering in the past has to do with their desire to achieve very low latencies, not necessarily increased throughput. If you look at a historical standard Gigabit Ethernet connection, you may see latencies that are around 125μs. This may have been ok in the past, but as improvements at the application level as well in the system hardware and CPU take hold, legacy Ethernet won't be good enough for HPC and clustering environments.

The interesting, and often overlooked fact with Ethernet is that the latency characteristics are improving as the industry moves from 1 Gigabit to 10 Gigabit. The faster throughput on the wire comes along with lower latency to some extent, but in addition, there have been several improvements in interrupt handling that drastically improve overall latencies when comparing legacy 1Gigabit to 10Gigabit. With a basic 1^st generation Intel® 10Gigabit CX4 card you can now see latencies approach 25μs without any special tuning.

What's even better is that Intel's 10 Gigabit networking silicon also has further enhancements for improving latency by introducing some new specialized Low Latency Interrupt (LLI) filters in the silicon. These filters provide the hardware with a quicker reaction time to network packets that meet certain customizable criteria. The filters can be tuned to have a rapid response to certain packet and traffic types. With these kinds of LLI filters in place, latencies can be reduced further by another ~50% to ~14μs.

Going forward with 10 Gigabit there are new technologies and designs that can help push latency even lower to the sub-10μs threshold to keep Ethernet very competitive as a fabric not only from a cost and throughput perspective, but also from the perspective of latency.

And while lower latency is certainly important, the last piece that was really missing from the Ethernet performance puzzle was not just low latency, but deterministically low latency. The key is that the worst case packet latencies for many applications are relevant and very important. By application thread affinitization, the individual data thread can be piped directly between a network queue and a CPU core. By more evenly distributing the networking workload between CPU cores in a predictable fashion, you get a deterministic kind of latency that does not stray far from the average assuming CPU cores do not get oversubscribed. Average latency of ~14μs is good, but the fact that you can get this with reasonable determinism is a key for many applications and usages.

Now, lower, deterministic latency is not just a theoretical benefit for certain niche applications. Decreasing latency and improving overall latency characteristics while increasing throughput directly benefits the transaction rates that can be achieved with real world applications. As an example of the improved performance is the latest Reuter Market Data Systems (RMDS) benchmarks done by STACResearch on the 4-way Intel® Xeon E7450 (Dunnington) using the Intel® 82598EB 10 Gigabit AT Dual Port networking adapter. The testing showed the Highest Point-to-Point Server throughput to date on a single server in testing done by STAC. And total updates per second reached over 15 million. Financial Service industry administrators: I can see you drooling...

Latency and throughput numbers are great to talk about, but at the end of the day, real world application performance on real systems is the key. While there will always be a small subset of the high end server market that needs the absolute lowest latencies provided by Infiniband; 10 Gigabit Ethernet is gaining ground while maintaining its place as the default fabric of choice for multiple applications and traffic types. I believe the best is yet to come as newer, faster, and more responsive technologies continue to roll out.

Ben Hacker

Sometimes you get so deep into something that you don't realize how crazy it is until you take a step back. Like most technology companies, Intel has an inherent love for acronyms. The cacophony of standards bodies, advanced technologies, and intense rates of change in our industry necessitates the use of abbreviation just to be able to communicate clearly and briefly. However, while I am at least as much of a techno-phyliac as most of the folks in the technology jungle, even I sometimes run into an acronym wall. I thought to help myself and others it might be a good idea to decode one of the newer sets of network technologies that I work closely with and to decipher some of the associated names and acronyms that come along with it.

10 Gigabit Ethernet: It's here, it's real, and it's growing fast_._

Ethernet (IEEE 802.x) has evolved over the years from a new standard linking computers together at slow rates and has moved from 10 Megabit per second (Mbps), to 100Mbps, to 1 Gigabit per second (Gbps), and a few years ago to 10GbE unidirectional throughput. Over time there have been several physical connection types for Ethernet. The most common is copper (Cat 3/4/5/6/7 cabling is used as the physical medium) but Fiber has also been prevalent as well as some other more esoteric (such as BNC Coax) physical media types. The most common 10GbE adapter (until very recently) has been Optical only due the difficulty of making 10GbE function properly over copper cabling.

But this post isn't meant to discuss the past, but more to decode the present and future as it relates to 10Gig Ethernet and the variety of flavors that are available. Below I'll cover a number of acronyms for 10GbE IEEE standards that are often lumped together as '10 Gigabit' and discuss some of the differences and usages for each. After that, I'll also try to clear up some of the confusion about ‘form factor' standards for optical modules (which are separate from IEEE) and some of terms and technologies in that realm:

10GBase-T (aka: IEEE 802.3an):

This is a 10GbE standard for copper-based networking deployments. Networking silicon and adapters that follow this specification are designed to communicate over CAT6 (or 6a/7) copper cabling up to 100 meters in length. To enable this capability, a 10GbE MAC (media access controller) and a PHY (Physical Layer) designed for copper connections work in tandem.

10GBase-T is viewed as the holy grail for 10GbE because it will work within the most prevalent Cat 6/7 based infrastructure that is already in place. For this flexibility, 10GBase-T trades off higher power, and higher latency.

10Gbase-KX4 (aka: IEEE 802.3ap):

This is a pair of standards that are targeted toward the use of 10GbE silicon in backplane applications (such as a blade design). The specifically is designed for an environment where lower power is required and shorter distances (up to only 40 inches) are sufficient.

10GBase-SR (aka: IEEE 802.3ae):

This specification is for 10GbE with optical cabling over short ranges (SR = Short Range) with multi-mode fiber. Depending on the kinds of fiber, SR in this instance can mean anything between 26 - 82 meters on older fiber (50-62um fiber). On the latest fiber technology, SR can reach distances of 300m. To be able to physically support a connection of the cable, any network silicon or adapter that support 10GBase-SR would need to have a 10GbE MAC connected to an Optics module designed for multi-mode fiber. (We'll discuss optics modules in more depth further down in this post.)

10GBase-SR is often the standard of choice to use inside the datacenters where fiber is already deployed and widely used.

10GBase-LR (aka: IEEE 802.3ae, Clause 49):

LR is very similar to the SR specification except that it is for Long Range connections over single-mode fiber. Long Range in this spec is defined as 10km, but distances above that (as much as 25km) can often be obtained.

10GBase-LR is used sparsely and really only deployed where ultra long distances are absolutely required.

10GBase-LRM (aka: IEEE 802.3aq):

LRM stands for Long Range over Multimode and allows distances of up to 220 meters on older standard (50-62um) multi-mode fiber.

10GBase-LRM is targeted for those customers who have older fiber already in place but need extra reach for their network.

10GBase-CX4 +(aka: IEEE 802.3ak):+

This standard of 10GbE connection uses the CX4 connector/cabling that is used in Inifinband^TM^* networks. CX4 is a lower power standard that can be supported without a standalone PHY or optics module (the signals can be routed directly from a CX4 capable 10GbE MAC to the CX4 connector). Due to the physical specification for CX4 based 10 Gigabit, it provides a lower latency than comparable 10GBase-T copper PHY solutions. With the use of CX4 passive (copper) cables, the nominal distance you can expect between your 10GbE links is ~10-15m. There are also amplified 'active' (but still copper) cables with nominal distances up near 30m.

Below is an image of a standard CX4 based socket that would be on a 10GBase-CX4 NIC:

There are also what referred to as ‘active optical' cables are for CX4, which actually have an optics module in the termination of the cable, and the cable body is fiber. This kind of active design increases cable reach and improves flexibility (fiber is smaller than copper pairs) but also increases cost. These active cables can increase reach up to 100m.

Intel has recently released our own series of active optical CX4 cables.

For short distances (such as inside the rack in a datacenter), CX4 offers one of the lowest cost ways to deploy 10GbE from switch to server. Because of its design, CX4 also achieves very low latencies as well.

&lt;/end of IEEE standards ramble&gt;

Ok, so we've summarized the majority of the IEEE 10GbE standards. But the immediate question arises: "Why are there so many?" Is the IEEE standards body for 10GbE just throwing out all these standards for every possible niche application? The answer is no. For any new standard IEEE phy interface standard to be approved, it must pass on several criteria including "distinct identity" and "broad market potential". While all of these standards certainly won't apply to any given institution's network, they all do all meet real market needs.

X2, XFP, SFP+... say what?

A final mystery that I've alluded to above has to do with the various optical module form factors that are available for 10GbE. XENPAK, X2, XPAK, XFP and SFP+ are standard optics module form factors that are used by both switch and NIC vendors in the industry. These modules that go along with the 10GbE networking products are an interesting beast. They are not specified by IEEE, but are standardized by a group of industry participants through what is known as a Multi-Source Agreement (MSA).

XENPAK, XPAK and X2 are the older module standards originally used for 1GbE, followed by XFP which shrunk the form factor of the actual module as well as the fiber cable pairs. SFP+ is a newer form factor that is now gaining momentum with switch and NIC vendors. An SFP+ optics module can use the same fiber pairs used with XFP (no new fiber cable needed), but the form factor of the cage in the switch or NIC as well as the optics module itself are smaller. The key advantage of using SFP+ is the new form factor can drive lower costs, lower thermals, and higher densities at the switch.

Here is an image of an older X2 optics module:

And here is a comparison of the size of XFP (right) relative to SFP+ (left):

The optics modules are driven by a low power interface from the 10GbE MAC. The interfaces are XAUI (for X2 modules), XFI (for XFP modules), and SFI (for SFP+ modules). These interfaces generally are supplied directly from the 10GbE based MAC to the module cage. One of the things the module MSA standards bodies agree on is not only a form factor for the module itself but also the electrical specifications of the driver interface that can be accepted from the MAC.

The key thing I want to hammer home here is that IEEE specification (such as 10GBase-SR) is separate from the module form factor used.

For example, you can have a Short Range optical NIC that uses X2, XFP, or SFP. So asking for an "SFP NIC" isn't actually specific enough, because that could mean a lot of different things. You'd have to specify a 10GBase-SR NIC, with SFP+ optics.

SFP+Direct Attach:

Now that I've thoroughly confused everyone, I'll take it one step further. Not only can each module form factor be used with different IEEE MAC specifications, but each module doesn't even need to be used for a fiber connection at all. An interesting example of using an ‘optics' module form factor for a non-optical design is SFP+Direct Attach.

SFPDA is similar in concept to CX4 but provides a bit more flexibility. Normally, you may have a switch or NIC that is designed to be able to support the addition of SFP based optics modules for a 10GbE fiber connection. Direct Attach allows for passive Twin-Axial (2 pair copper) cables to be plugged directly into the SFP+ cage (in place of an optical module) to carry the serial signal from the MAC directly over the cable to another SFP+ form factor enabled NIC or switch.

Again, the downside is that without either a standalone PHY, or optics module to send the signal over a long distance, a passive cable with SFPDA has a reach in the ~10-15m range. The real advantage for SPFDA over CX4 is that on the switch side the SFP+ module design allows higher density switches than CX4 can provide.

For a top of the rack switch, SFP+DA will likely provide excellent cost, power and latency characteristic and still have enough reach to be very feasible inside the rack.

10GbE - The Infrastructure is Ready!

I hope that I've lifted a little bit of the fog that surrounds the 10GbE market and the related technologies. The last thing I want to leave you with is the fact that 10GbE infrastructure is now starting to roll into the mainstream. CX4 switches are available broadly in the market today and SFP+ type designs for both optical modules as well as Direct Attach connections have been demonstrated and will be getting rolled out very soon by various vendors.

Intel is already selling a wide variety of NICs and silicon to meet the various form factors and standards based market needs I listed above along with other vendors in the market place.

After years of anticipation, 10GbE is finally hitting its stride. Next stop... 10_0_GbE...

While Intel is certainly most widely known for manufacturing our extremely complicated CPUs that are the brain of many computing platforms worldwide; there are several other products and technologies that people at Intel have been involved in for many years which are critical to computing environments everywhere. As a person who has been working in various networking and manageability roles at Intel since 2001, I'd like to take a little time to focus on Intel's history in the Ethernet market since its inception more than 25 years ago and focus a little on where the market might be going in the future.

Below is an image that tries to capture the key highlights of Intel's specific involvement in the Ethernet market over the last 3 decades:

As you can see the Ethernet market has come along way from clunky multi-chip 10Mpbs solutions from more than 25 years ago all the way to Quad Port Gigabit and Dual Port 10 Gigabit designs that are prevalent today.

Moving into the future the Ethernet market is growing increasingly more complicated by the year with new capabilities and features targeted specifically to support server virtualization, infrastructure convergence, enhanced storage technologies, and the continued importance of power efficiency of the overall compute infrastructure. Each of these innovations and changes will have a big impact on the overall structure and design what servers and datacenters will look like in the future. My colleague Ken Lloyd gave his thoughts on how 10 Gigabit technologies will provide I/O convergence and overall cost savings for computer networks in the future and there are clearly lots of interesting things going on right now.

Over the next several months I plan to try to go more in depth on many of the exciting developments taking place in the Ethernet market and to hopefully shed some light on some of the changes that are coming our way.

Stay tuned in the coming weeks!

- Ben