The Server Room Blog

10 Gigabit iWARP @ SC’09!

tom.l.stachura@intel.com — Thu, 19 Nov 2009 18:48:09 GMT

What is iWARP? (Click here to find out)

Ethernet Tom here. I’ve recently come into the Intel Ethernet group – marketing Ethernet products in the HPC, Financial, and Cloud verticals.

And very HPC relevant - I’ve been very busy getting things lined up for Super Computing ’09 and wanted to share what we have happening:

· 3 demos:

o Intel Booth: Two 6-node clusters (96 total cores!) running NYSE’s Data Fabric* middleware – showing iWARP vs. non-iWARP

o Supermicro Booth: 4-node cluster running Fluent

o EA Booth: 4-node cluster running Linpack in a converged Ethernet environment

· 5 presentations:

o Data Transportation at iWARP Speed – Feargal O’Sullivan – NYSE Technologies

o Memory Virtualization over iWARP for Radical Application Acceleration – Tom Matson – RNA Networks:

o A 10Gigabit Ethernet iWARP Storage Appliance – Paul Grun – System Fabric Works

o CD-adapco Benchmarking Performance using Intel iWARP – William Meigs – Intel Corporation

o iWARP – What & Why? – Tom Stachura – Intel Corporation

If you are here today (final day of SC’09!), stop by and check it out. If not, I’ll come back later with some video links.

What if every server in your data center generated 10Gbps of network traffic?

jim.r.blakley@intel.com — Thu, 30 Apr 2009 22:29:30 GMT

What if every server in your virtualized data center was driving 10Gbps of traffic?

My team just completed a test with an end user where we drove nearly 10Gbps of traffic over Ethernet through a single Xeon 5500 based server running ESX4.0. The workload was secure FTP. Our results will be published in the next 30 days. We’ve seen 10Gbps through a server in several other cases (notably, video streaming and network security workloads) but this is first time we’ve really tried to do a 10GB “enterprise” workload in a virtualized environment. It took a fair amount of work to get the network and the solutions stack to work (we had to get a highly threaded open source SSH driver from the Pittsburgh Supercomputer Center, for example, to make it scale). We also found some good value for some of our specialized network virtualization technologies (i.e., the VT-c feature known as VMDQ). But, regardless, by working at it moderately diligently, we got it to work at 10Gbps and don’t see any real barriers to doing that in real production environments.

We also found that the solution throughput is not particularly CPU-bound, it’s “solution stack bound”. That means that workloads that are more “interesting” than virtualized secure FTP and video streaming are likely to be able to source and sync more than 10Gbps/server, too. And, when we get to converged fabrics like iSCSI and FCOE that put the storage traffic on the same network path (or at least the same medium) as the LAN traffic, we’d expect that the application needs for higher Ethernet throughput will increase.

So what? Well, if you buy the fact that virtualized servers can do interesting things and still drive 10GB/s of Ethernet traffic, you have to wonder what’s going to happen to the data center backbone network. If you have racks with 20 servers each, putting out a nominal 6Gbps of Ethernet traffic, each rack will have a flow of 120Gbps and a row of 10 racks will need to handle 1.2 Tbps. I’m not sure what backbone data center network architecture will be able to handle that kind of throughput. Fat tree architectures help especially if there are lots of flows between servers in close proximity to each other in the same data center. But, fat tree networks are very new and not widely deployed. Thoughts?

Data Center Fabric

ken.r.lloyd@intel.com — Wed, 13 Feb 2008 20:56:39 GMT

Whenever I see the word fabric, I immediately think of cloth, which immediately takes my mind to "helping" pick out curtains, which puts me in immanent boredom mode. Pardon the offense to those of you that actually know what color curtains you own, but there are some very cool things happening in data center communication.

Today the boxes in the data center (servers, storage, switches, ...) communicate over some combination of medium and protocols. While some protocols have become less common, (DECnet, Token Ring) there is still a bunch of InfiniBand, Fiber Channel, and Ethernet. Guess what, Ethernet is going to win. Ok, that was an unsupported prognostication,,, but Ethernet has won in every other arena it has entered.

Ethernet is what I really want to talk about today. There are a series of changes happening that allow Ethernet to be cast in the role of data center fabric. The first is simple throughput - 10GB. 10GB has the capacity to support the needs of detached storage. The thing that really makes this possible is a QOS feature called "priority pause". This extension to the Ethernet standard enables Ethernet to support QOS for differentiated services and to minimize or eliminate packet drops.

This new "Ethernet" enables rich SOE ( Storage over Ethernet) beyond iSCSI to FCOE( fiber channel over Ethernet). Intel has open[-sourced|http://open-fcoe.org/] FCOE software, and the network community is actively discussing the future of Fiber Channel.

Consolidating on 10gb reduces required port counts, and a single protocol reduces server hardware and switch infrastructure. All of this saves energy and simplifies data center wire management. These are good things. The extensions to the Ethernet specs were the result of collaboration between Intel and other industry leaders. This new spec should make it simpler to choose which curtains will go best in the data center

Intel is the leader in the add on server NIC business has great products available in the 10gb NIC space. The Intel NICS, when used on and Intel based platform, also support VMDQ - part of Intel's "Virtualization Technology for Connectivity". At VMworld Intel demonstrated that VMDQ technology in Intel Network Adapters boosted max throughput on a 10gb virtualized connection from 4gb to ~9gb - nearly max theoretical capacity.

You can buy the bits today! There are about 30 different vendors with products in the 10gb space. Clearly this is a ripe area for innovation and entry. The question remains glass or copper - optical or electrical? There are pros and cons for each, and both are supported by this next generation Ethernet. I would love to hear which you are choosing and why.