Blog Posts From Wired Ethernet Tagged With bandwidth

Taking the Wraps off of Intel’s New SDN Products at IDF

webadmin@intel.com — Tue, 18 Sep 2012 22:56:32 GMT

Last week at IDF, we took the wraps off of two exciting new products that OEMs and ODMs can use to develop switch systems for the emerging market for software-defined networks (SDN) in the data center.

At the chip level, we launched the new Intel^® Ethernet Switch FM6700 series, which is a 10G/40G SDN-optimized switch family that provides up to 64 10GbE or up to 16 40GbE ports.

The FM6700 series can support both SDN and legacy networks. Thus it can be used in top-of-rack switch SDN applications in the data center, or in network appliances or video distribution switches (thanks to its built in load balancing features).

For all applications, the switch features a pioneering low-latency architecture, built on the programmable Intel^® FlexPipe™ frame-processing pipeline and single output queued shared memory architecture. Both of these technologies combine to deliver highly deterministic packet forwarding with a maximum layer 3 latency of about 400nS.

The switch supports NAT and IP tunneling features for use in both IP and SDN applications. For the SDN networks, the FlexPipe frame processor can be used to parse and process SDN packets. The switch also supports 4,000 complete OpenFlow 12-tuple table entries that can be searched in a single pass for added performance. There are also flexible tagging and tunneling options, including the ability to provide both an SDN and tunneling proxy for connected hosts.

At the platform level, we’ve introduced Seacliff Trail, a top-of-rack switch network reference platform for OEMs and ODMs that is based on the FM6764. It offers 48 SFP+ 10GBE ports and four QSFP+ 40GBE ports, and can drive up to 7m of direct attach copper without the need for additional PHY chips on the board.

It’s an all-Intel platform as well with a control plane based on the Crystal Forest AMC module that features an Intel^® Xeon^® processor, Intel communications chipset and Intel^® 82599 10GBE controller. Intel’s Wind River subsidiary provides the open and extensible software framework based on its Linux OS. This provides both easy SDN integration and also direct API access to add third-party apps for rapid innovation.

Seacliff Trail is a major step forward in fulfilling Intel’s SDN vision of the next-generation of networking. That vision combines standardized, high-volume hardware with an open and extensible software framework that allows OEMS/ODMs to add their own value-added functionality.

Last week we had good crowds coming to see these products at our IDF booth along with two sessions where we presented both the FM6700 series SDN features along with its features for server load balancing..

All you ever wanted to know about Intel SR-IOV and Flexible Port Partitioning

webadmin@intel.com — Fri, 10 Aug 2012 23:11:23 GMT

OK, maybe not ALL you ever wanted to know, but all we've shared with you up until now, all in one place!

It has been nearly 2½ years since I posted my first ever Blog, it was about

Setting up Red Hat 5.4 Xen* for SR-IOV using the Intel® 82576 GbE. Since that time I’ve managed to post a blog now and then, usually pointing you to a new paper or video I’ve published.

The very smart fellow who is now responsible for the Intel Ethernet Virtualization technology has continued to write great documents on how to configure things such as SR-IOV in different environments.

So we have managed to accumulate a fairly nice list of documents and videos that talk about SR-IOV and one usage of it, Flexible Port Partitioning.

To make it easier for you, I’ve decided to compile a list of all these documents, videos and blogs in one place as a nice reference. So without any more fanfare from me, here are the various docs that we have written:

SR-IOV Background, Introduction

· SR-IOV Primer [Document]

o http://www.intel.com/content/www/us/en/pci-express/pci-sig-sr-iov-primer-sr-iov-technology-paper.html

· SR-IOV Explanation [Video]

o http://www.youtube.com/watch?v=hRHsk8Nycdg

SR-IOV Configuration

· Setting up Red Hat 5.4 Xen for SR-IOV using the Intel 82576 GbE [Blog]

o http://communities.intel.com/community/wired/blog/2010/03/01/setting-up-red-hat-54-xen-for-sr-iov-using-the-intel-82576-gbe

· Using Intel® Ethernet and the PCISIG* Single Root I/O Virtualization(SR-IOV) and Sharing Specification on Red Hat* Enterprise Linux* Technical Brief [Document]

o http://www.intel.com/content/www/us/en/network-adapters/10-gigabit-network-adapters/ethernet-x520-sr-iov-red-hat-tech-brief.html

· How to Configure Intel® X520 Ethernet Server Adapter Based Virtual Functions on SuSE*Enterprise Linux Server* using Xen* [Document]

o http://www.intel.com/content/www/us/en/network-adapters/10-gigabit-network-adapters/ethernet-x520-suse-linux-xen-tech-brief.html

How to Configure Intel® X520 Ethernet Server Adapter Based Virtual Functions on Citrix*XenServer 6.0* [Document]
- http://www.intel.com/content/www/us/en/ethernet-controllers/ethernet-x520-citrix-xenserver-tech-brief.html

Intel Flexible Port Partitioning

· Intel Flexible Port Partitioning using SR-IOV Demonstration

o http://www.youtube.com/watch?v=bOMB9RsQfo4 [Video]

· An Introduction to Intel Flexible Port Partitioning Using SR-IOV Technology Technical Brief [Document]

o http://www.intel.com/content/www/us/en/network-adapters/10-gigabit-network-adapters/10-gbe-ethernet-flexible-port-partitioning-brief.html

· Configure QoS with Intel® Flexible Port Partitioning [Document]

o http://www.intel.com/content/www/us/en/network-adapters/10-gigabit-network-adapters/config-qos-with-flexible-port-partitioning.html

High Quality Built-in PHYs Simplify ToR Switch Designs

webadmin@intel.com — Tue, 31 Jul 2012 20:59:31 GMT

Most top-of-rack (ToR) data center switches are installed where their name says, on the top of the server rack. This means their 10G downlink ports are connected to servers that are within a few meters of the switch. In data center applications where low latency is critical, SFP+ direct attach (DA) copper cables can be used to connect servers at up to 7 meters from the switch. But this requires high quality PHYs inside the ToR switch.

Switching ASICs used in ToR switches can have up to 72 10G ports on a single piece of silicon. But high quality 10G SerDes are difficult to design, and many of these large chips are designed with the assumption that the SerDes only needs to drive a locally connected PHY chip that will take on the burden of driving DA copper cables or backplanes.

At Intel, we took a different approach. With the knowledge that many of our customers are designing low-latency ToR switches using DA copper cabling, we chose to embed high quality 10G PHYs within our Intel Ethernet FM6000 series switch silicon. These PHYs can drive up to 7m of SFP+ DA copper on 10GbE ports or up to 5m of QSFP DA copper on 40GbE ports. With up to 72 10G SerDes on the FM6000, this eliminates up to 18 external quad PHY chips that must be used when lower quality SerDes are used within the switch ASIC. Elimination of these external PHYs saves cost, power and board area, which are critical in today’s large, flat data center installations.

Understanding 10 Gigabit BASE-T: Noise to Signal Ratio

webadmin@intel.com — Thu, 31 May 2012 20:46:55 GMT

Noise to signal? Isn’t it supposed to be signal to noise ratio? When you’re talking 10 Gigabit BASE-T, you’re talking noise to signal. 10 Gigabit BASE-T has been described to me as “It’s not like looking for a needle in a haystack, it’s like looking for the right snowflake in a snow storm. At night.” Where does all that noise come from? It comes from the cable next to the cable your data is trying to travel across, it comes from wireless signals, and it comes from outer space. It even comes from the other wires in the same cable. With all this noise, how does all that analog chaos get transformed into digital clarity of 1s and 0s?

Lots and lots of processing. In our current generation of 10 Gigabit BASE-T products we use PHYs that have 5 channels of DSPs, and tons of specific analog processing silicon to turn that snow storm of waveforms into nice clear digital Ethernet frames. Far End Cross Talk (FEXT), disturbers, Near End Cross Talk (NEXT), 8/10 encoding, it all gets to be a jumble. FEXT is signal coupled from one channel to a neighboring channel, measured at the far end. NEXT is the same as FEXT, except that it is a measure of how much of that coupled signal is reflected back onto a neighboring channel on the same end. Disturbers are other noise sources outside of the cable. The 8/10 signaling is how the data is encoded to help make sure there is a roughly even mix of 1’s and 0’s in the data stream. It also helps to ensure the quality of the data by periodically transmitting a checksum that the link partner can use to validate the data that it has received. If your data stream is highly biased, like all 1’s or all 0’s, the signal processing can get “tone deaf”, limiting what it thinks is a 1, making all the data a 0. Unless the average voltage is centered you can get signal droop which makes it hard to tell a weak 1 from a strong 0. The extra bits of data do take up some space on the wire, but things like clock recovery from the extra bits make it worth the overhead. Just means that to get to your advertised speed, your baud rate must be higher. For example, a 64/66 10G serial signal has a clock rate of 10.3125 G/sec, which transmits data at 10G/sec. You can’t transfer 10 Gigabits of data. There will always be space on the wire taken up by other things (TCP/IP headers is a big one). But those extra pieces help keep 10G copper stable.

Time to dispel a couple of 10 Gigabit BASE-T rumors I’ve heard over the years.

First up, bending the cable. I was at a trade show and a gentleman comes up and says to me “That 10 Gigabit stuff is for the birds, you can’t even bend the cables without dropping link!” I was covering for another employee that had taken sick, so I wasn’t warned of the rowdiness of this show. “I think you’re mistaken, sir.” I said. He smiles and reaches behind my demo and grabs the cable. He bends it over on itself, making the straight run into an O shape. He pushed with his thumb and made into more of an I shape. He let it go and walked over to the console screen, expecting error message after error message of disconnected link. To his shock, (and my amusement) nothing happened. “Wow.” He mumbled. He took my card and left the show floor, all the while being heckled by his travel buddies. Unless you damage the cable, I’ve never seen a bent cable effect link. And I’m not gentle to my cables.

Second, cell phones. I have heard concerns about answering your cell phone in a datacenter and taking your whole network offline. There is some kernel of truth to this one since the signal bands of cell phones and 10G cables do get dangerously close to one another. But it’s just another disturber source, albeit a powerful one. Modern cell phones and modern 10 Gigabit BASE-T are both designed to use as little power as needed to reach the end station so you would have to be pretty close with a very powerful cell phone to put a ton of noise onto the wire. And we do test cell phone interference, but with good cables you shouldn’t see any issues. Use bad cables and things could get ugly. Good cables include shielding to protect signal wires from disturbers. Those DSPs can only filter out so much; an investment in quality cables is an investment in the quality of your data.

I bet you a least half of you will now go to your 10 Gigabit BASE-T installs and make a call from right next to your servers to see if it drops link. If you have Intel® Ethernet and good cables, I think you’ll have a phone call and link all at the same time.

Thanks for using Intel® Ethernet, and special thanks to Sam J for his help double checking my technical details.

How Intel Delivers Low Latency Ethernet Switching

webadmin@intel.com — Tue, 22 May 2012 16:44:45 GMT

Up until recently, Ethernet has grown to dominate enterprise and Internet networking applications without much consideration for packet latency. In applications like email or video delivery, an extra microsecond or two doesn’t have that much impact. But many emerging datacenter applications need low latency networks. These include financial trading, real-time data analytics and cloud-based high-performance computing. In addition, many web service applications can spawn a large amount of east-west traffic, where the latency of these multiple transactions can build up to unacceptable levels.

The Intel^® Ethernet switch product family can deliver the industry’s lowest layer 3 forwarding latencies while providing a large number of 10GbE and 40GbE interfaces. Intel latency numbers are less half of traditional layer 3 Ethernet switches, and to deliver this performance, Intel uses several key technologies:

Cut-Through Switching: A switch in cut-through mode will start transmitting a data packet before it has completely received that packet. This is compared to store-and-forward switching, where a packet is completely received by the switch before it is forwarded to its next destination. Store-and-forward switches can’t deliver low enough latency for the data center.

Terabit Crossbar Switch: In many cases, latency increases as a result of internal switch congestion. This causes the packets to be queued until others are sent through, causing delays that can be unacceptable for real-time applications. Intel uses a matrix-based crossbar switch that’s unique because of its capacity – 1 terabit-per-second. This amount of crossbar over-speed can greatly reduce internal switch congestion.

Single Output Queued Shared Memory: Intel uses a proprietary SRAM technology that is fast enough to allow every input port to write into the same output queue simultaneously. Without this technology, there may be insufficient on-chip bandwidth for simultaneous output queue access, so chip architects have traditionally relied upon combined input/output queued (CIOQ) memory architectures that build in a set of virtual output queues at every switch input. This is a complex solution that is very difficult to scale to large port count switches, and adds blocking that impacts packet latency through the chip.

High Packet-Rate Frame Processing: It doesn’t matter how fast you can enqueue and dequeue packets from shared memory if your frame forwarding pipeline can’t keep up. Intel employs special circuit technology that allows a single L2/L3/L4 frame processing pipeline to forward packets at full line rate even if they are minimum size packets arriving back-to-back on all ports simultaneously. It does this while maintaining extremely low processing latencies under all forwarding conditions.

By adding a high-performance frame processing pipeline, terabit crossbar data path and very fast packet memory to a low latency cut-through switch, the Intel Ethernet Switch family chips deliver the latency that meets the needs of the new mesh, cloud and financial networking applications.

What’s a Cloud without Full Virtualization using Unified Networking?

webadmin@intel.com — Fri, 18 May 2012 16:17:45 GMT

http://www.youtube.com/embed/oYrlRWMbpO0In a cloud environment, virtualization that stops at the server edge stops short of the potential of cloud computing. To operate a robust cloud environment, you also need to not only virtualize but unify your network and storage connections, so those resources can scale up and down with the needs of dynamic workloads. It’s this fully unified and virtualized combination that gives cloud greater flexibility and agility. This is a key point made in a new Unified Networking video now appearing on YouTube. This lively animation—produced on behalf of Intel—provides a high-level look at the benefits of Unified Networking solutions combining network and storage connections to enable better use of resources and better cloud performance. In animated graphics, the video shows that static network and storage connections can’t reallocate themselves to meet the dynamic needs of cloud workloads, and you end up wasting resources. The video also provides a look at a data center built around Unified Networking over 10 Gigabit Ethernet. The Intel® Ethernet Converged Network Adapter used in this example provides more than just a wider pipe; it’s a smarter pipe. You don’t have to break a 10GbE pipe into smaller static connections when using Intel® Ethernet. It gives you the ability to dynamically reallocate bandwidth to provide the correct ratio of storage and network resources to meet the changing needs of individual workloads. The result? “You never have to over-engineer your data center to power your cloud,” the video tells us. If you can spare 4 minutes to watch this video, you can potentially see a new way to look at Unified Networking and how it can help you avoid inefficient hardware allocation in your cloud environment.

To get further details on this new way of looking at using 10GbE connections that the video outlines, check out the Unified Networking content on our Cloud Usage Models site.

Intel® Data Direct IO Technology (Intel® DDIO) explained (Video)

webadmin@intel.com — Mon, 30 Apr 2012 20:17:34 GMT

I helped David film this short explaining how Intel® DDIO works. The content is all David, the video production (or the lack thereof) is all mine. David explains the value of this new technology.

http://www.youtube.com/watch?v=CvNzX8FGdKA

Introducing the Intel® Ethernet Controller X540

webadmin@intel.com — Tue, 06 Mar 2012 21:06:22 GMT

Today we announce the arrival of our newest product: the Intel® Ethernet Controller X540

What is it?

The Intel® Ethernet Controller X540 is the next generation of 10 Gigabit Ethernet (10GbE) controllers from Intel. Using a 40 nm manufacturing process, the X540 is the first 40 nm, dual-port integrated Media Access Controller/Physical Layer (MAC/PHY) single chip designed for reduced power and package size that is designed for use as a LAN on Motherboard (LOM) network controller. It also powers the Intel® Ethernet Converged Network Adapter X540.

Key Features

The world’s first fully integrated single chip 10GBASE-T Ethernet Controller specifically optimized to bring 10 GbE networking to server boards as a LOM.
Designed for LOM in mainstream rack and tower servers. – Support up to 100m over widely implemented Cat 6A cables
Backwards compatible with existing 1 GbE infrastructure, providing a seamless (or easy) upgrade path to 10GbE
Provides industry leading features for I/O Virtualization and Storage over Ethernet, including iSCSI and FCoE.
Low power <12.5 W
Small package 25 x 25 mm,
Designed using the latest 40 nm PHY technology with industry-leading integrated Electrical Mechanical Interference/Remedial Frequency Interference (EMI/RFI) filters.

Intel® Ethernet Unified Networking Principles

Intel has delivered high quality Ethernet products for over 30 years, and our Unified Networking solutions are built on the original principles that made us successful in Ethernet:

Open architecture integrates networking with the server, enabling Information Technology (IT) managers to reduce complexity and overhead while enabling a flexible and scalable data center network.
Intelligent offloads lower cost and power while delivering the application performance that customers expect.
Proven Ethernet unified networking is built on trusted Intel Ethernet technology, enabling customers to deploy Fiber Channel over Ethernet (FCoE) or Internet Small Computer System Interface (iSCSI) while maintaining the quality of their traditional Ethernet networks.

Intel’s unified networking solutions are enabled through a combination of standard Intel® Ethernet products along with trusted network protocols integrated in the operating systems. Thus, unified networking is available on every server either through LOM implementation or via an add-in Converged Network Adapter or Network Interface Card (NIC).

Go with the Flow Control

webadmin@intel.com — Fri, 13 Jan 2012 23:21:17 GMT

Flow control is a key part of keeping your 1 Gigabit and faster network running smoothly. Somewhere along the line some websites started telling people to turn off flow control so their network would go faster. In the short term this might be fine; in the long term you’re going to see bigger problems and probably drop more packets than you’ll make up by being able to send as needed. The problem people would say is that Flow Control stops the traffic, and this costs performance. Absolutely it stops traffic. But it stops the traffic the receiver doesn’t have room for! The flow control is like a stop light controlling access to the highway. Instead of letting them all in at once, when there is no room for them and gumming up the works even further, flow control gives protection to the receiver. This protection allows for long term speed and less dropped packets.

Consider the following data set that was gathered using ethtool -S eth0 from a real system.

NIC statistics:

     rx_packets: 329461
      tx_packets: 302120
      rx_bytes: 34897969
      tx_bytes: 32293428
      rx_no_buffer_count: 39147
      rx_missed_errors: 1097931
      rx_flow_control_xon: 0
      rx_flow_control_xoff: 0
      tx_flow_control_xon: 228
      tx_flow_control_xoff: 1098233

Let’s look at it in detail. Tx Flow control XOFF is the NIC telling the link partner, “I’m overwhelmed, stop the packets”, Rx Flow Control is the Link partner telling the NIC “I’m overwhelmed, stop the packets”. Note the difference. TX FC is transmitting TO the partner, RX FC is receiving FROM the partner. In this case, the NIC is basically screaming, I’m overwhelmed (More XOFF than packets), and the rx_no_buffer_count and rx_missed_error confirms it. What this means is the NIC has no resources and is actively dropping packets. But FC is on! Why are we still dropping/missing packets? The link partner is not honoring the flow control packets! In this case, the link partner has sent 1.4 million frames, but only 300K got through because the link partner didn’t care about flow control. With flow control the packets might take a little longer to get there, but they will get there.

Looking at the data, see the 228 XON? The NIC only caught up 228 times. That’s not so good. So what was causing all these missed packets? Most likely cause is a slow PCI express and/or slow memory implementation. Packets come all the time and memory slowness and getting combined with another busy device on a few narrow lanes can mean not enough PCI Express bus between something like the ESB or a switch cascaded off a switch.

Moving to 10 Gigabit it is, well, ten times worse. You have 1/10^th the time and ripple effect of delaying a packet moves faster. It was so problematic that Data Center Bridging (DCB) and DCBx came out to make flow control end to end. Instead of just link partner to link partner, DCBx allows one overwhelmed end point to tell the overwhelming source to chill out. This moves the delay caused by flow control to the point most able to deal with it. While some backplanes of switches can temporarily store terabits of data, having the starting node just not send it right now is the best result. We’ll do a deeper dive on DCBx another time, but with it you get effectively lossless Ethernet with DCBx and that lets you do FCoE and other storage technologies.

Thanks for using Intel Ethernet and turn on your Flow Control!!

Where's the queues?

webadmin@intel.com — Tue, 20 Dec 2011 21:35:08 GMT

A food vendor in the US has started using their old slogan from the 80s again directing people to wonder where the main point of the product is. I’ve started to see some of the same questions around our Intel® Ethernet Controller I350 product and queues. Our Intel® 82576 GbE Controller product had 16 queues per port, and as a virtualization product, it really excelled. Now our I350 part is out, and people are looking at it having only 8 queues per port, and exclaiming “Where’s the queues!”

An easy reaction to have. You have to look at the math and typical usage models to see why we went in the direction we did. The I350 is a quad port product, which means the total number of available queues is the same as the 82576 product. But by spreading that load across four physical ports it allows for more total throughput and efficiency. A modern CPU can do 1 Gigabit of traffic on a single queue with very little CPU overhead, something that is very hard to do on a 10 Gigabit port. So having extra queues at 1 Gigabit allows for oversubscription, which means more applications or processes using the same maxed out port. By spreading the same number of queues across four physical ports it avoids the oversubscription. I’ve heard people talking about some O/S vendors have been recommending one physical port per virtual machine! I’d recommend one queue per VM. One queue per CPU core has been enough at 1G for a while. So with a single I350 quad port, that means 32 CPU cores, which is hard to do, or 32 VMs which is easier to do. But 32 VMs is a lot of them. With the quad port density you can scale quickly without having to add switch chips which add to latency. Four quad port I350 cards would give you 128 VMs or 128 cores, all at a modest amount of slots. With our Virtual Machine Device Queues (VMDq) and SR-IOV technologies you don’t have to do one core one queue, or one VM per queue, you can share queues efficiently and effectively. But with that many why share? All this applies to all the other queue using technologies. With RSS, for example, the cores are the thing, and most machines don’t have 32 cores. Yet.

We do make a dual port I350 product, and there will be a decrease since there are not enough ports to make up the loss of the queues per port. But that product is designed to allow design compatibility with our new 10 Gigabit BASE-T product, so it’s designed to fulfill a different market role. This new product has over a hundred queues, so the I350 and even the 82576 can’t keep up. The dual I350 was designed to allow a 1G LOM today that can be upgraded to a 10 Gigabit LOM without a redesign (if designed right to start with). This dual build strategy is the point of the I350 dual, so in the interest of bringing the new features of the I350 to market, the tradeoffs of queues was made. And leaving the dual port I350 internally the same as the quad port I350 allowed us to be timely in our product offerings.

Thanks for using Intel® Ethernet!

Letting the Cables Speak

webadmin@intel.com — Fri, 02 Dec 2011 22:58:46 GMT

Here at the wired blog we talk endlessly about our adapters and Ethernet silicon. But there is more to a network than just an end node. There is more to it than the link partner. There is the cable that makes it all work together. Without the cable you have just nothing (we don’t do wireless here on the wired blog) and without a network you have a static end point. A network is dynamic, and the cable brings that rich content to you.

I was lucky enough to spend some time with the guys at Panduit*, a worldwide leader in cable manufacturing, at their HQ near Chicago, Illinois. They have been doing electrical interconnects for ages. They got started with power lines and power interconnects; it makes sense that they would eventually head into Ethernet cables. I learned a ton from them about the specifics of the language around the cable, like the RJ45 connector is actually the female part that the cable plugs into, and the jack is what most people would call the RJ45, the end of the cable. It reminds me of how specific language is developed around an industry. I was at Panduit to shoot a video (it’s still under construction), but once it’s done, hopefully you’ll learn about 10 Gigabit BASE-T like I did during my visit. I considered myself an expert, but they taught me a thing or two about the interconnect, the cable.

I asked Tom K from Panduit a few questions about Ethernet cables and here is how it went:

Q1: What is the most common cable problem?

Believe it or not, Panduit has found the most common problem to be cable management. As mundane as that sounds, poor cable management can quickly lead to wasted time making moves, adds, and changes as well as drops in network performance.

For example, a patch field or cross connect may start out nice and neat, everything is groomed and looks great. Over time, with the pressure to get operational changes done quickly, cable management drops off the list of priorities. People have the best of intentions, thinking “I’ll get back to it later,” and before you know it, the patch field has shifted from being neatly dressed and identified to a disorganized sprawl.

The ironic part is that, in the long term, one is not saving any time at all. The messier the patch field gets, the more time it takes to make changes because one has to double and triple check to make sure they are making the right move. It also makes airflow through the cabinet a problem. Finally, from an installation or permanent link standpoint, we often find that poorly managed cables coming out of the jack have too tight of a bend radius, which could cause the link to fail.

Q2: How much of an impact on a network’s performance do the wrong cables have?

In the context of moving towards 10GBASE-T, a lot. As an example, let’s say that for whatever reason, a 10GBASE-T link is deployed over CAT5e cabling, rather than CAT6A cabling, which is designed to handle 10 Gig traffic. The net effect of this decision would be a higher bit error rate than the 10GBASE-T standard calls for, as CAT5e cannot support the cabling requirements for 10GBASE-T. There also would be more internal crosstalk such a Near End Crosstalk (NEXT), Far End Crosstalk (FEXT), and Alien Crosstalk (AXT). (NEXT is cross talk between the cable pairs that occurs near the 10GBASE-T PHY, FEXT is crosstalk that occurs at the far end of the link away from the PHY, and AXT is cross talk that occurs to signals coming from another cable laying next to the one in question.) Overall, if the wrong cabling is used, the link will drop packets and overall network throughput and performance will be compromised.

Q3: What are some tips on moving from CAT5e to CAT6a?

First tip would be to use proper installation techniques. CAT5e and 1000BASE-T are a little more forgiving than 10GBASE-T given the higher frequencies used in 10GBASE-T, and the impact of noise is more troublesome too. An example would be un-twisting too much of the copper pair when terminating CAT6a in a RJ-45 jack. For 1000BASE-T or CAT5e, excessive un-twist may not impact the performance of the link. However, with the tighter tolerances of 10GBASE-T and CAT6a cabling, the balance of the twisted pair might be impacted enough by too much un-twist to the point that crosstalk and external noise sources may impact the performance of the link, which then would adversely affect throughput and overall network performance. This problem can be solved by using the proper tools from the connector manufacturer.

Second tip would be to stress the importance of link testing, or of using cables that are designed to reduce the time and types of testing required. Testing the usual parameters, such as Near End Crosstalk (NEXT) and return loss, to name just two, are important. However, because of the frequencies involved and the modulation levels that are used for 10GBASE-T, people are most concerned with Alien Crosstalk (AXT). This effect can occur when twisted pair cables are laid down right next to each other in cable raceways, increasing the chance of signal coupling. Testing for Alien Crosstalk can be a time consuming, and therefore, expensive process. Fortunately, Panduit’s standard CAT6a and CAT6a-SD (small diameter) cables use a patent pending technology that suppresses Alien Crosstalk. This means one does not need to perform testing for AXT with Panduit’s CAT6a solutions, which saves the time and expense of testing for it.

Here is a link to the Tom’s Hardware* article that showed how we test for exactly the kind of stuff Tom is talking about and more.

If you have questions for the Panduit guys, let me know in the comments.

Thanks for using Intel Ethernet.

(update: There was an error in the link to Tom's Hardware)

Understanding Performance

webadmin@intel.com — Thu, 13 Oct 2011 18:23:18 GMT

   Performance is why you buy 10 Gigabit and 1 Gigabit products. Performance doesn’t always just happen. When should you make changes? There is no magic formula for it. Let’s look at the problem as a system from network to application. The first level of buffering is at the switch. Using flow control - or better yet Data Center Bridging with its end point to end point flow control - will help keep down the number of dropped packets by making sure the network is aware of the resources level of system. With flow control you should only get what you have room for. Once a packet gets to the system, it is first buffer is in our First in, First out (FIFO) storage. If the bus is busy, or things in general are delaying the processing of packets in the host system, the FIFO can store both TX and RX packets. The descriptors and host memory buffers are the next queuing place. Next is the driver interface, since almost every driver this day is a miniport driver, and above that is the protocol stack. Today that is mostly TCP/IP. Finally things end up in the application. Now this is a slightly simplified description and leaves out components like our intermediate driver(ANS) which can do its own queuing but shouldn’t be holding any resource long enough to influence our discussion.   To tune the applications network performance, you need to tune all the layers in between. For some O/Ses that is easy, some not so much. Most modern operating systems will be dynamic and try to keep up, but some settings - like Maximum TCP Buffer - can be static and may need some manual tweaking.
     If you have a large number of packets incoming and they are being processed slowly, you might want to turn up the number of buffers. This will increase the entry buffers, but the real problem is why the packets are being processed slowly above it. In the older Windows days you could set some parameters to make it allocate more TCP/IP and NET buffers, but that seems to have gone away. If you know some good ones, post them in the comments. Having said that more can be better, here is why you shouldn’t always just put in more. There is a cache of descriptors on chip that limits how many of the bigger list can be used at any given moment. While the whole list is 2048, it will only fetch and put them in bunches of 64. This is mostly for cache-line alignment and internal architecture reasons, but it can make the 2048 get consumed in fits and starts instead of a linear flow. What this means is while 2048 buffers might be ready, only 64 will be consumed before it has to write those back and get new ones. It isn’t always exactly 64, since it would be wasteful to wait for that many in a low traffic environment. But what that does mean is while 64 are being processed, the other 1984 are sitting idle. Idle is of course relative, since at 1G speed all 2048 descriptors can be read, used and put back in less than a millisecond. But the point is still there. If the O/S can service and return the buffer back to the driver in under a millisecond, you don’t need more buffers since the data is moving fast enough. If the buffers go to an upper layer like the stack or the application and sit, then more buffers is just going to treat a symptom and not the problem. You are better off to keep digging for the root cause rather than just slapping on more buffers. The stack is an important part of the equation, so check out its statistics and errors to see if your network is underperforming because of slowness there.
     Other than more descriptors, what can I do? Interrupt Throttle Rate can help. There will be a separate article on that. Make sure the data is going to the core that is going to be doing the work. Use RSS and MSI-X to make sure that you’re not moving your data several times. If you’re sending your traffic to a core that is saturated, consider moving some of what is running on that core to another core. Process affinity is pretty easy to use and can make sure you’re keeping all those cores working evenly. You might also consider updating the O/S as an option.   This is not always a very attractive option, but modern O/Ses are very aware of the loads that a network can bring, and the vendors listen to our suggestions like never before. We saw major improvements moving from one O/S to a newer version from the same vendor. I won’t name names since we all have our off days, but since the driver for our stuff didn’t change, it clearly pointed at the cause. The application can also be a good source of tuning, so scour the apps support site for tuning ideas.
Let’s look at each bottleneck one at a time. There will be hints on things to do and questions that need to be answered.
1.    Packet creation. In the perfect performance model, zero clocks are spent on creating the packets. Pre-existing packets will be faster than packets that require the CPU to touch them in memory, or worse yet move the data from ring 3 to ring 0. Minimize this to maximize performance. Understanding whether the data is static (created once) or dynamic (created at use) can influence how long it takes to create.
2.    Maximize the bus utilization. Even though there is enough bandwidth on the bus for one port doing bidirectional traffic, some things can take up bandwidth which can cost performance. Statistics are slow registers and there are a lot of them. When trying to maximize performance, don’t access any statistics. Just like the Stats registers, there are other “Slow” registers. When maximizing performance, leave all registers out of it. Tail registers should be the only registers used. Registers like VLAN, RAR, MCAST, TxCW, RxCW and all PHY registers are slow enough through the internal logic to cost packets.
3.    Data Locality. Make sure the data (packet buffers and descriptors) are running on the same CPU as any work being done on them. If work has to be scheduled between processors it will slow things down a great deal. Single misses can cause impacts. This is one of the best things you can do to improve your network performance, but most of the work isn’t in the network area. If you can only do one thing to your network, do this first, then the rest of the paper.
4.    Time. Full line rate with 64 byte packets is a packet every 69 nanoseconds. A delay of a microsecond will slow 15 packets for a loss of 10K of performance. Breakdown the CPU time, analyze the bus time, and watch the inter-packet rates coming out of the port. If there is a gap of more than the IPG time, then that is slowing performance. If you’re getting 9.2Gbps, that works out to roughly 3747 of “missing” usecs. By reviewing the bus data you should be able to work up the stack to find the trouble.

Performance isn’t just a case of making a call to the vendor to make the magic happen. Every part of the chain needs to be analyzed to improve the overall performance of the system.

Thanks for using Intel® Ethernet.

Maximum Teaming

webadmin@intel.com — Fri, 16 Sep 2011 20:52:10 GMT

A question I get on occasion is how to make my teaming go faster. This webpage outlines a TON about teaming. It's a great “go to” reference and lists abilities the infrastructure must have.

In terms of the throughput, I'm looking at a test pass report in which we ran some teaming testing. In a two Intel® Ethernet Server Adapter X520 team under Windows* in our testing we saw 18Gb doing just TX, almost 19Gb doing just RX, and almost 32Gb bi-direction(BX). That's 18 out of 20, 19 out of 20 and 32 out 40 (10TX1 + 10TX2 + 10RX1 + 10RX2). Where did the 8Gb go? Overhead! Each time somebody has to act on that packet, it takes time, CPU and trips to memory. Each step and packet touch slows things down enough for it to add up. Also remember that the usable data throughput of 10GbE unidirectional is about 9.49Gb. Headers, checksums, consume the rest.

We used 16 1Gb clients per port for this test. That's 32Gb RX and 32Gb TX so that should provide saturation. With 10Gb clients you can lower the number needed. The ultimate goal with over-saturating the traffic load is to ensure that traffic generation isn’t the bottleneck for your benchmark.

Make sure your switch is up to the task. Most people assume the switch can handle it, but some switches don’t have a lot of backplane throughput to handle the saturation traffic levels. And for some teaming options, like 802.3ad, you will need specific switch configurations. These may need additional license. Consult your switch vendor for more details.

On the system side, in the BIOS we get aggressive with the Power Management related setting. On high performance runs, we don’t care about power. We usually turn off processor C-states and Speed Step off so the processor doesn't try to sleep during the test. The time lost coming out of the sleep states will cost you performance. Performance at its core isn’t about bandwidth; it truly is about time. The processor guys are epic in keeping these transitions as fast as possible, but in the Ethernet performance land, we can’t spare time for anything. At 10Gb, that can be 11 million packets per second. Even with a 4 Gigahertz CPU, nanoseconds lost can mean packets lost.

A dynamic team seems faster than static ones since we use dynamic more for our testing. It also keeps you away from proprietary solutions. Make sure your server has LOTS of RAM. Ours have 12GB. Obviously you’ll need an O/S that can address all that memory. Clients had only 2GB each.

Sorry I can't share the test report (it includes can’t-share info on 3^rd party’s adapters) but I think I captured the high points to help you get your teams tuned. Let me know if you have questions in comment section.

And, as always, thanks for using Intel® Ethernet

From the VMworld* show floor

webadmin@intel.com — Tue, 30 Aug 2011 23:01:01 GMT

Trade shows are always fun. You make new friends, you see old friends (like EMC’s Pat Gelsinger) and you get to show people what your product can really do. For the demo at the booth today, I’ve been using vMotion* on vSphere5* to move six vm’s from one machine to another machine using Intel® Ethernet. The 10 Gigabit pipe of the Intel® Ethernet Server Adapter x520-DA2 makes it happen so fast. To make it more fun, I make booth visitors guess how long it will take to move them. All six. At the same time. I get guesses than run from ten minutes to one minute.

It takes on average 17 seconds. To move all six.

They can’t believe it. But it’s real.

Stop by the Intel booth at VMworld and I’ll show you too. Booth number 738.

Intel and Red Hat* to talk “Free Fibre Channel”

webadmin@intel.com — Mon, 01 Aug 2011 22:04:17 GMT

On Tuesday August 2, 2011 at 1800 UTC, 2pm Eastern Daylight Time, David Fair from Intel and Russell Doty from Red Hat will be hosting a webinar exploring the use of 10Gb Ethernet as a universal fabric. You’ll learn if Fibre Channel over Ethernet (FCoE) is a technology that you should deploy in your network. The talk will cover how FCoE is implemented, and they will cover other storage protocols like iscsi and file services.

https://engage.redhat.com/forms/20110802FibreChannel