Blog Posts From Wired Ethernet Tagged With driver

All you ever wanted to know about Intel SR-IOV and Flexible Port Partitioning

webadmin@intel.com — Fri, 10 Aug 2012 23:11:23 GMT

OK, maybe not ALL you ever wanted to know, but all we've shared with you up until now, all in one place!

It has been nearly 2½ years since I posted my first ever Blog, it was about

Setting up Red Hat 5.4 Xen* for SR-IOV using the Intel® 82576 GbE. Since that time I’ve managed to post a blog now and then, usually pointing you to a new paper or video I’ve published.

The very smart fellow who is now responsible for the Intel Ethernet Virtualization technology has continued to write great documents on how to configure things such as SR-IOV in different environments.

So we have managed to accumulate a fairly nice list of documents and videos that talk about SR-IOV and one usage of it, Flexible Port Partitioning.

To make it easier for you, I’ve decided to compile a list of all these documents, videos and blogs in one place as a nice reference. So without any more fanfare from me, here are the various docs that we have written:

SR-IOV Background, Introduction

· SR-IOV Primer [Document]

o http://www.intel.com/content/www/us/en/pci-express/pci-sig-sr-iov-primer-sr-iov-technology-paper.html

· SR-IOV Explanation [Video]

o http://www.youtube.com/watch?v=hRHsk8Nycdg

SR-IOV Configuration

· Setting up Red Hat 5.4 Xen for SR-IOV using the Intel 82576 GbE [Blog]

o http://communities.intel.com/community/wired/blog/2010/03/01/setting-up-red-hat-54-xen-for-sr-iov-using-the-intel-82576-gbe

· Using Intel® Ethernet and the PCISIG* Single Root I/O Virtualization(SR-IOV) and Sharing Specification on Red Hat* Enterprise Linux* Technical Brief [Document]

o http://www.intel.com/content/www/us/en/network-adapters/10-gigabit-network-adapters/ethernet-x520-sr-iov-red-hat-tech-brief.html

· How to Configure Intel® X520 Ethernet Server Adapter Based Virtual Functions on SuSE*Enterprise Linux Server* using Xen* [Document]

o http://www.intel.com/content/www/us/en/network-adapters/10-gigabit-network-adapters/ethernet-x520-suse-linux-xen-tech-brief.html

How to Configure Intel® X520 Ethernet Server Adapter Based Virtual Functions on Citrix*XenServer 6.0* [Document]
- http://www.intel.com/content/www/us/en/ethernet-controllers/ethernet-x520-citrix-xenserver-tech-brief.html

Intel Flexible Port Partitioning

· Intel Flexible Port Partitioning using SR-IOV Demonstration

o http://www.youtube.com/watch?v=bOMB9RsQfo4 [Video]

· An Introduction to Intel Flexible Port Partitioning Using SR-IOV Technology Technical Brief [Document]

o http://www.intel.com/content/www/us/en/network-adapters/10-gigabit-network-adapters/10-gbe-ethernet-flexible-port-partitioning-brief.html

· Configure QoS with Intel® Flexible Port Partitioning [Document]

o http://www.intel.com/content/www/us/en/network-adapters/10-gigabit-network-adapters/config-qos-with-flexible-port-partitioning.html

Even more ECC updates

webadmin@intel.com — Sat, 17 Mar 2012 00:09:52 GMT

ECC has many real "definitions" - error correcting circuits, error correcting code, or error correction code - but they all do the same thing. It helps keep data intact within the chip memory. ECC uses a special algorithm to encode information in a block of bits that contains sufficient detail to permit the recovery of a single bit error in the safeguarded data. This protocol will not only detect single bit errors, but will transparently correct them on the fly. Double errors will be flagged as an error and the device will try to get software’s attention about it. Related to ECC is parity. Parity will keep track of the quantity of bits in total and track them as either even or odd. Should this parity change while it is in the chip memory, it will be flagged as an error. Since you can’t tell which bit went rogue, this is a poor man's protection. Also, if more than 1 bit changes, parity check can miss it.

(Warning HTML table!)

Product	Packet Buffer (In band Traffic)	Managability (out of band traffic)	Datasheet Info
X540	ECC	ECC	7.14.1
82599	ECC	ECC	7.14.1
82598	ECC	ECC	Look for DHER/PBUR
I350	ECC	ECC	7.6
82580	ECC	ECC	7.23
82575 and 82576	ECC	ECC	7.6
82571	ECC	Parity	13.7
82573 / 82574 / 82583	None	None	n/a
82546	None	None	n/a

Both ECC and parity have a basic limitation in that if the error is large enough, it will look okay. ECC is far more resistant to this. We try to make sure bad things don't happen to your data, but it still might happen. And while it will try to tell you when it does go bad, sometimes it still won't notice. That's why our lawyers care about articles like this. Multiple bit errors are very rare and probably will cause other problems to the machine. Data integrity isn't made with a single point safety net. If you want to guarantee your data, use a multiple layered approach since it’s unlikely that all of them will fail.

Any Requests from the Audience?

webadmin@intel.com — Fri, 27 Jan 2012 21:27:03 GMT

The Wired blog will be adding videos this year! In the spirit of helping this meet the needs of the community, I figured I’d open it up to requests. (No Free Bird requests please!). We can have explanations of technologies (like FCoE, or Flow control), how-tos (like teaming setup) or things like video datasheets or even interviews with the people that help shape Intel® Ethernet products.

But unless you comment, I can’t supply the videos you need to help you understand and deploy Ethernet.

Thanks for visiting!

Dword on avoiding getting bit by the bytes

webadmin@intel.com — Thu, 19 Jan 2012 18:58:08 GMT

Our 1 Gigabit and 10 Gigabit products have some specific access rules. The manuals outline the details, but they are all located in the definitions chapter and I’ve skipped over my fair share of those types of chapters. This blog will cover the when, where and why of these rules and where to watch out in your implementation.

Why is usually a late question to ask, but we’re starting with it. All the MAC registers are 32 bit, dword accesses. Some registers (like the VLAN and multicast hash tables) are much bigger, but you still access them one dword at a time. Outw, UINT32, use what you must, but make sure it does an atomic 32 bit read of the register. Anything bigger or smaller will not latch the internal logic and it won’t complete. Why 32 bits? It makes the access engine very straightforward and it keeps the MAC logic discreet. By not having byte, or qword accesses available, we can keep the MAC register logic down. That means fewer gates and that means smaller products and less heat to dissipate.

So when doing accesses to registers that “span” across two 32 bit registers (like RAR and some stats registers) read the lower dword first. This will help avoid carries from the lower register to the upper register during the time between the two reads. When writing to a register that spans, write the latching part last. In the case of the RAR, there is an “Address valid” bit. If you write that first, there will be a timeframe (albeit small) that half right addresses will be valid.

That’s for the MAC registers. Now to deal with the NVM/EEPROM. This is where bits, bytes and words really get messy. Most NVM devices are expressed as size X bits. Most of our documentation uses words. Tools like ethtool use accesses in bytes. Outch! Most engineers can do the math in their head, but with some of our features (iSCSI, management) requiring certain sizes of NVM, missing the size means missing the feature. Not pretty. To make matters worse, there are byte swapping issues when you start talking about word vs byte. X Y become YX when reading words. Since the manuals are in words, you need to keep track. The Intel-provided tools do it all in words, but ethtool and the Linux guys like it in bytes. A script should be able to keep track of it for you.

Being aware of the access size requirements the Intel® Ethernet devices have will keep you from misreading the NVM or not being able to get data out of the device. If you have any tips for the NVM or the registers let me know in the comments!

Thanks for using Intel® Ethernet.

Go with the Flow Control

webadmin@intel.com — Fri, 13 Jan 2012 23:21:17 GMT

Flow control is a key part of keeping your 1 Gigabit and faster network running smoothly. Somewhere along the line some websites started telling people to turn off flow control so their network would go faster. In the short term this might be fine; in the long term you’re going to see bigger problems and probably drop more packets than you’ll make up by being able to send as needed. The problem people would say is that Flow Control stops the traffic, and this costs performance. Absolutely it stops traffic. But it stops the traffic the receiver doesn’t have room for! The flow control is like a stop light controlling access to the highway. Instead of letting them all in at once, when there is no room for them and gumming up the works even further, flow control gives protection to the receiver. This protection allows for long term speed and less dropped packets.

Consider the following data set that was gathered using ethtool -S eth0 from a real system.

NIC statistics:

     rx_packets: 329461
      tx_packets: 302120
      rx_bytes: 34897969
      tx_bytes: 32293428
      rx_no_buffer_count: 39147
      rx_missed_errors: 1097931
      rx_flow_control_xon: 0
      rx_flow_control_xoff: 0
      tx_flow_control_xon: 228
      tx_flow_control_xoff: 1098233

Let’s look at it in detail. Tx Flow control XOFF is the NIC telling the link partner, “I’m overwhelmed, stop the packets”, Rx Flow Control is the Link partner telling the NIC “I’m overwhelmed, stop the packets”. Note the difference. TX FC is transmitting TO the partner, RX FC is receiving FROM the partner. In this case, the NIC is basically screaming, I’m overwhelmed (More XOFF than packets), and the rx_no_buffer_count and rx_missed_error confirms it. What this means is the NIC has no resources and is actively dropping packets. But FC is on! Why are we still dropping/missing packets? The link partner is not honoring the flow control packets! In this case, the link partner has sent 1.4 million frames, but only 300K got through because the link partner didn’t care about flow control. With flow control the packets might take a little longer to get there, but they will get there.

Looking at the data, see the 228 XON? The NIC only caught up 228 times. That’s not so good. So what was causing all these missed packets? Most likely cause is a slow PCI express and/or slow memory implementation. Packets come all the time and memory slowness and getting combined with another busy device on a few narrow lanes can mean not enough PCI Express bus between something like the ESB or a switch cascaded off a switch.

Moving to 10 Gigabit it is, well, ten times worse. You have 1/10^th the time and ripple effect of delaying a packet moves faster. It was so problematic that Data Center Bridging (DCB) and DCBx came out to make flow control end to end. Instead of just link partner to link partner, DCBx allows one overwhelmed end point to tell the overwhelming source to chill out. This moves the delay caused by flow control to the point most able to deal with it. While some backplanes of switches can temporarily store terabits of data, having the starting node just not send it right now is the best result. We’ll do a deeper dive on DCBx another time, but with it you get effectively lossless Ethernet with DCBx and that lets you do FCoE and other storage technologies.

Thanks for using Intel Ethernet and turn on your Flow Control!!

Linux* e1000 driver support model update

webadmin@intel.com — Tue, 03 Jan 2012 20:28:10 GMT

The e1000 driver has been around for many years. But the products that are supported by the driver are starting to end of life. That means keeping the driver in an active state becomes less and less useful. But we remain committed to the driver, and here is how things will be different moving forwards. The e1000 driver on SourceForge.net* will not be changing. Any bug fixes or changes will be made only to the kernel.org version. (It’s so important I’ve colored it red! ;)) We’ll still have the e1000 driver on SourceForge.net, and thanks to it being open source, you can make your own fixes should you run into them. We have no plans around other drivers at this time, and we’ll update you here on the blog should anything change.

NOTE: The e1000 driver will implement the change to a kernel only support model on/around Q2 of 2012. If you have open issues you would like to see resolved, please submit them through http://sourceforge.net/projects/e1000/ as soon as possible.

Guidance for a Home Ethernet Network

webadmin@intel.com — Mon, 26 Dec 2011 18:25:08 GMT

This is the time of year when presents are exchanged all across the world. Sometimes those presents are a home Ethernet network. Sometimes wired, sometimes not. Here is some guidance to help you stay safe while joining the wired world. Nothing hard and fast, and clearly I’m not a trained safety or health professional, so if in doubt consult somebody who is.

Wired vs Wireless. There are times when the choice is made for you. Hand held gaming products are all Wireless. But things like digital video records for television have both, with wireless being an after purchase add-on on some models. Don’t get me wrong, I love Wireless. I just think Wired has some advantages. Like speed. Tapping into a wired network is much harder than a wireless one. I know the wires can be scary, but you’d be surprised how easy they are to run. I wired my own house for 1 Gigabit Ethernet, and it wasn’t that bad. The hard part was going into the crawl space (Spiders!) and figuring out how to go between the first and second floors (I went down in a closet so nobody can see), but it took less than a day to do. Make sure you obey all local building codes; you don’t want to get into any trouble. If you’re not sure what you’re doing, consult a professional.
Security. If you go wireless, don’t name your Wireless Access Point (WAP) with anything that can track back to you. Let’s say somebody wants to do something mean to you, having a WAP named “Dougs House” would make it easier for them to do something via YOUR network. Is ID 21358973 your house or your neighbors? A small thing of course, but sometimes it is the small things that make people into victims. Enforce passwords on all computers in your house, even the kids’ computers. The sooner they know about security the better they will be at it later. Teach them that only a responsible caregiver, like mom and/or dad ever get their password, and that might be even be age dependant. Consider using filter software to keep the more adult portions of the internet at bay. Consult your local school districts to see if they have classes for adults on internet usage for teens, and other internet topics. The price for a safe internet family is eternal vigilance, but with a little bit of help and education you can keep things safe. And fun.
Limits. Computer usage can be addictive. Negotiate time and amount of usage and stick to them. Making sure to limit content to age appropriate material can help avoid other addictions related to the internet. This is a family blog, so I won’t link to them, but that should give you an idea about it. Also, computers are a good way to avoid making real world friends. Real world social networks help prevent bullying and round out personalities. They are good for getting into contact with friends that were left behind after moving, but moderation is a good principle. I got a great piece of advice from my junior level programming class (don’t ask when ;)): Computers are a way to make a living, they are not a way of life. Use them as the tool they are. Don’t make them into your friend.

Happy end of the year to everybody!

Thanks for making this year the Wired blog’s best so far!

Where's the queues?

webadmin@intel.com — Tue, 20 Dec 2011 21:35:08 GMT

A food vendor in the US has started using their old slogan from the 80s again directing people to wonder where the main point of the product is. I’ve started to see some of the same questions around our Intel® Ethernet Controller I350 product and queues. Our Intel® 82576 GbE Controller product had 16 queues per port, and as a virtualization product, it really excelled. Now our I350 part is out, and people are looking at it having only 8 queues per port, and exclaiming “Where’s the queues!”

An easy reaction to have. You have to look at the math and typical usage models to see why we went in the direction we did. The I350 is a quad port product, which means the total number of available queues is the same as the 82576 product. But by spreading that load across four physical ports it allows for more total throughput and efficiency. A modern CPU can do 1 Gigabit of traffic on a single queue with very little CPU overhead, something that is very hard to do on a 10 Gigabit port. So having extra queues at 1 Gigabit allows for oversubscription, which means more applications or processes using the same maxed out port. By spreading the same number of queues across four physical ports it avoids the oversubscription. I’ve heard people talking about some O/S vendors have been recommending one physical port per virtual machine! I’d recommend one queue per VM. One queue per CPU core has been enough at 1G for a while. So with a single I350 quad port, that means 32 CPU cores, which is hard to do, or 32 VMs which is easier to do. But 32 VMs is a lot of them. With the quad port density you can scale quickly without having to add switch chips which add to latency. Four quad port I350 cards would give you 128 VMs or 128 cores, all at a modest amount of slots. With our Virtual Machine Device Queues (VMDq) and SR-IOV technologies you don’t have to do one core one queue, or one VM per queue, you can share queues efficiently and effectively. But with that many why share? All this applies to all the other queue using technologies. With RSS, for example, the cores are the thing, and most machines don’t have 32 cores. Yet.

We do make a dual port I350 product, and there will be a decrease since there are not enough ports to make up the loss of the queues per port. But that product is designed to allow design compatibility with our new 10 Gigabit BASE-T product, so it’s designed to fulfill a different market role. This new product has over a hundred queues, so the I350 and even the 82576 can’t keep up. The dual I350 was designed to allow a 1G LOM today that can be upgraded to a 10 Gigabit LOM without a redesign (if designed right to start with). This dual build strategy is the point of the I350 dual, so in the interest of bringing the new features of the I350 to market, the tradeoffs of queues was made. And leaving the dual port I350 internally the same as the quad port I350 allowed us to be timely in our product offerings.

Thanks for using Intel® Ethernet!

Understanding Intel Teaming: Multiple Vendor Teaming

webadmin@intel.com — Wed, 07 Dec 2011 14:35:08 GMT

The Intel® Advanced Network Services (Intel® ANS) provides teaming and VLAN functionality to Intel® Ethernet network products running under various Microsoft Windows* operating systems. The same role is done by channel bonding in Linux*. Unlike Linux, there is no common teaming architecture provided by the O/S. That might change for the next Windows edition; we will have to wait and see. Until then it’s an each vendor have at it free-for-all. But Intel ANS does provide some wiggle room for our partners that have Intel add-in cards and 3^rd party LAN on Motherboard (LOM) implementations. We call it Multi-Vendor Teaming (MVT) mode of ANS.

While MVT might sound like a great thing, you need to know upfront that MVT is not as good as an all Intel team. This is a natural outcome of the unique way each vendor implements and defines their products. Because of this unevenness in the implementations, we have to enforce lowest common denominator (LCD) rules for the MVT. Jumbo frames are out, as is anything else that isn’t a Microsoft standard with a standard OID. Why are jumbo’s out? Does the size listed as supported include the CRC or not? The difference means the packet will be rejected by the infrastructure for being bigger than the MTU. Things like RSS and checksum offloads are typical things that end up being turned off for LCD reasons.

We do less testing with MVT than with our own native teams mostly because we are unable to address defects in the third party products. One day maybe it will move up into the O/S provided realm, like in Linux, but until then, you should know if you add Intel adapters to your non-Intel LOM you can still make a team using Intel ANS.

Let me know if you have any MVT questions and thanks for using Intel® Ethernet!

Maximum Teaming

webadmin@intel.com — Fri, 16 Sep 2011 20:52:10 GMT

A question I get on occasion is how to make my teaming go faster. This webpage outlines a TON about teaming. It's a great “go to” reference and lists abilities the infrastructure must have.

In terms of the throughput, I'm looking at a test pass report in which we ran some teaming testing. In a two Intel® Ethernet Server Adapter X520 team under Windows* in our testing we saw 18Gb doing just TX, almost 19Gb doing just RX, and almost 32Gb bi-direction(BX). That's 18 out of 20, 19 out of 20 and 32 out 40 (10TX1 + 10TX2 + 10RX1 + 10RX2). Where did the 8Gb go? Overhead! Each time somebody has to act on that packet, it takes time, CPU and trips to memory. Each step and packet touch slows things down enough for it to add up. Also remember that the usable data throughput of 10GbE unidirectional is about 9.49Gb. Headers, checksums, consume the rest.

We used 16 1Gb clients per port for this test. That's 32Gb RX and 32Gb TX so that should provide saturation. With 10Gb clients you can lower the number needed. The ultimate goal with over-saturating the traffic load is to ensure that traffic generation isn’t the bottleneck for your benchmark.

Make sure your switch is up to the task. Most people assume the switch can handle it, but some switches don’t have a lot of backplane throughput to handle the saturation traffic levels. And for some teaming options, like 802.3ad, you will need specific switch configurations. These may need additional license. Consult your switch vendor for more details.

On the system side, in the BIOS we get aggressive with the Power Management related setting. On high performance runs, we don’t care about power. We usually turn off processor C-states and Speed Step off so the processor doesn't try to sleep during the test. The time lost coming out of the sleep states will cost you performance. Performance at its core isn’t about bandwidth; it truly is about time. The processor guys are epic in keeping these transitions as fast as possible, but in the Ethernet performance land, we can’t spare time for anything. At 10Gb, that can be 11 million packets per second. Even with a 4 Gigahertz CPU, nanoseconds lost can mean packets lost.

A dynamic team seems faster than static ones since we use dynamic more for our testing. It also keeps you away from proprietary solutions. Make sure your server has LOTS of RAM. Ours have 12GB. Obviously you’ll need an O/S that can address all that memory. Clients had only 2GB each.

Sorry I can't share the test report (it includes can’t-share info on 3^rd party’s adapters) but I think I captured the high points to help you get your teams tuned. Let me know if you have questions in comment section.

And, as always, thanks for using Intel® Ethernet

Ethernet and Space

webadmin@intel.com — Tue, 26 Jul 2011 23:33:48 GMT

I’ve been lucky enough in my life to see some amazing feats of engineering during my travels.

Things like the Falkirk Wheel (above), The Taipei 101 and an achievement of the ages that is once again headline news, the Space Shuttle. I’ve talked about it before, but as the Space Shuttle era ends, I look back with pride at the time Ethernet and Space came together.

After the Columbia disaster, NASA needed a way to inspect the shuttle and its launch debris field as the orbiter was still low enough in altitude to successfully abort the mission. This effort became known as the Shuttle Safe program, and NASA engineers decided to use high definition video feeds to watch for dangerous ice and foam chunks. The problem was distributing this data to Houston Mission Control and other analysis sites. Gigabit was the answer. Our Intel® 82546GB Gigabit Ethernet Controller was used on the winning solution, but there were some challenges with the design. Until the issues were resolved, the Shuttle couldn’t launch. As you can tell, we solved the challenge, and I watch every shuttle launch knowing that my personal work helped keep the brave astronauts on the mission safe.

Here is a picture taken using our 82546GB.

I hope you’ll drop me a line if I’ve helped you create something awesome so I can add to my list to great engineering feats.

Thanks for using Intel® Ethernet, and I hope one day Intel® Ethernet will be in space again.

Linux* drivers updated on SourceForge.net*

webadmin@intel.com — Mon, 25 Jul 2011 17:57:52 GMT

As outlined in our earlier Linux process blog, we’ve updated our major Linux drivers. Here is a summary of the changes for each at a high level.

ixgbe: 3.4.24

Supports kernel versions up through 2.6.39.3. Changes in this release:

New Intel® 82599 10 Gigabit Ethernet Contoller support for device ID 154d 8086 7b11
Enable rx hash use for the stack
New Intel® 82599 10 Gigabit Ethernet Controller support for device ID 10f8 1028 1f63
ethtool support for disabling RxCSUM - X540 OS2BMC support
Support for Dell* customizations
Support for UPD RSS (no packet frag)
Improved ethtool support for Perfect Filter
Allow Perfect Filter w/ DCB and SR-IOV

e1000e: 1.4.4

Intel® 82579 Gigabit Ethernet Controller: Fix for Tx Hang on FTS ME Platform
Intel® 82571 Gigabit Ethernet Controller: Add device id to fix for alternate MAC address feature
Upstream commits: 25db0338, 70739497 and 14ad2513, Change ethtool interfaces to use ethtool_cmd_speed[_set] API.
Convert use of receive descriptors from legacy to extended format to allow future enablement of advanced features such as timesync and RPS.

igb: 3.1.16

Fix for Malicious Driver Detection Feature on i350 devices in a virtualized environment.
Add module parameter for DMA Coalescing feature to allow user to determine approximate latency setting at init.
Updated DMA Coalescing init parameters per performance test lab results. Feature now saves more power.
Fix for problem found in pre-release RHEL 5.7 which caused a panic at device configuration.
Added function to set rx_hash and a call to it during receive. Not changeable by ethtool at this time, however.

*modified later in the posting day, I used an acronym wrong.

Parameter Talk: TX and RX descriptors

webadmin@intel.com — Fri, 24 Jun 2011 22:18:18 GMT

We use constructs called descriptors to move data into and out of the Intel® Ethernet device. Under Windows* these are controlled via our Intel® PROSet application, or via the property tabs if installed without Intel® PROSet. Intel® PROset allows the user to change the number of Descriptors with a parameter called “Receive Buffers” for RX Descriptors and “Transmit Buffers” For TX Descriptors. Some drivers will call it buffers and others descriptors. For our discussion they are the same thing. Why “buffers” when it influences the descriptor count? Most people understand what a network buffer is, but descriptors are a logical construct for our hardware and we don’t want to confuse people.

Here is what it looks like on my laptop:

You can see the “Transmit buffers” are below it. To modify the number of descriptors you just increase the value. In our Windows offerings there will be a limit of 2048 and it must be in increments of 8. On the Transmit side the starting value is 512, but the same rules of 2048 by 8 still apply. Why more TX than RX? Our architecture favors the nondeterministic RX side for priority so there is more turnover of descriptors than the TX side. Plus the O/S can sometimes not return TX assets back to the driver in a timely fashion.

Now under Linux*, for igb you use the ethtool interface to set ring parameters and the max is 4096 by 8. For the e1000 version 8 and greater, you pass it a parameter to change the size. The TX parameter is a little different in that it is the exponent you want, two multiplied by the value, to get the ring size you want: 2^7 to 2^12 (128 to 4096). This is already set to max, so not much to worry about there. On the RX size, it’s a straight number with a range of 80 to 4096, with the default being 256. The e1000e driver uses the same method as the igb driver, so ethtool will be your friend in that case.

Why does Linux have a bigger top end? Windows makes a bigger deal out of non-pageable memory than Linux does, and so our driver team limits the top end. Since each descriptor comes with a 2048 byte buffer, (and sometimes more) that can be a lot of memory. 8MB doesn’t sound like a lot for a modern system that can have 8GB of RAM, but we like to keep our memory footprint small.

When should you make changes? There is no magic formula for it. This article outlines some performance tuning ideas. But in general, there are two cases where you would want to make changes: 1) Low memory and 2) CPU or bus saturation.

If you running light on memory, you can drop the number of descriptors down but how fast the packets get processed by the O/S will determine if you suffer from missed and dropped packets because of a lack of resources.

If you have a large number of packets incoming and they are being processed slowly, you might want to turn up the number of buffers. This will increase the entry buffers, but the real problem is why the packets are being processed slowly above it. There is a cure for that most of the time, and it’s a called the Intel® XEON® processor E7 Family. Look around for articles on how to tune the memory usage of your favorite application; most serious application vendors have tuning tips on the web. If the O/S can service and return the buffer back to the driver in under a millisecond, you don’t need more buffers since the data is moving fast enough. If the buffers go to an upper layer like the stack or the application and sit, then more buffers is just going to treat a symptom and not the problem. You are better off to keep digging for the root cause than just slapping on more buffers. The stack is part of the equation a great deal, so check out its statistics and errors to see if your network is underperforming because of slowness there.

Other than more descriptors what can I do? Make sure the data is going to the core that is going to be doing the work. Use RSS and MSI-X to make sure that you’re not moving your data several times. If you’re sending your traffic to a core that is saturated, maybe consider moving some of what is running on that core to another core. Process affinity is pretty easy to use, and can make sure you’re keeping all those cores working evenly. You might also consider updating the O/S as an option. Not always a very attractive option, but modern O/Ses are very aware of the loads that a network can bring, and the vendors listen to our suggestions like never before. We saw major improvements moving from one O/S to a newer O/S from the same vendor. I won’t name names since we all have our off days, but since the driver for our stuff didn’t change, it clearly pointed at the cause. The application can also be a good source of tuning, so scour the apps support site for tuning ideas.

As always, thanks for using Intel® Ethernet

All about NDIS

webadmin@intel.com — Sat, 11 Jun 2011 17:13:30 GMT

If you run Windows* and you run networking, you run NDIS. NDIS - Network Driver Interface Specification has been around since well, forever. I still have a copy of the old NDIS2 documentation in a hard copy at my desk. Boy have things changed. It started basically as a miniport, went monolithic and end up miniport again. Miniport “plugs” into a protocol layer, while a monolithic has the protocol (or most of it) built into the driver.

This is a list of NDIS versions and the OS they tended to run on. For a long time you could run the older revisions on the new O/S. For example Win95* could run NDIS2. It just wasn’t the first choice for performance and other reasons. If you can run the previous version of the specification, I would recommend you run the native version since that will offer the best performance. If we offer a driver of that specification, it will be listed as Y. In the NDIS2 case, we only ship the DOS version of the driver. If you need O/S2 NDIS2, put it in the comments and we’ll see about it. 10 Gigabit never got an NDIS2 driver, mostly because NDIS2 today is only used during manufacturing. And 10 Gigabit LOM systems tend to have EFI and use that for “gas pumping” O/S images down. Need 10 Gigabit NDIS2? Again, let me know in the comments!

(Warning HTML tables!)

Version	OS Support	1 G	10 G
NDIS 2	O/S 2, Windows for Workgroups* 3.1, DOS	Y	N
NDIS 3	Windows for Workgroups 3.11	N	N
NDIS 4	Windows* 95, NT4	N	N
NDIS 5.0	Windows 98, 98 SE, ME, Windows 2000	N	N
NDIS 5.1	Windows XP*, Server 2003, CE6	Y	N
NDIS 5.2	Windows Server* 2003 SP2 or SP1 with the Scalable Networking Pack	Y	Y
NDIS 6.0	Windows Vista*	Y	Y
NDIS 6.1	Windows Vista SP1, Server 2008	Y	Y
NDIS 6.20	Windows 7, Server 2008 R2	Y	Y

That’s a lot of drivers! So what did each of those iterations bring to the table? Here is a listing of features from our point of view. Some TCP/IP Offload Engine stuff AKA TOE AKA Chimney was added somewhere in there, but since we didn’t use it; I don’t have in our chart. I start with NDIS5 since it is the start of the modern era. Ancient history isn’t my strong suit

NDIS 5.0: Return to the “full” miniport, start of the modern Windows networking driver.
NDIS 5.1: Windows XP, Server 2003

IPsec Offload v1 - Offload Internet protocol security tasks (either IPsec or LSO)

IP (v4) and TCP/UDP Checksum offloads

NDIS 5.2: Windows Server 2003 SP2

RSS (Receive Side Scaling) was introduced in SP1 w/Scalable Networking Pack and in SP2

Start of Quick Data support

DCA (Direct Cache Access)

LLI (Low Latency Interrupts)

LSO (Large Send Offload)--in 2003 it is known as TCP Segmentation Offload

NDIS 6.0: Windows Vista

LSOv1 Large Send Offload - Offload segmentation of large TCP packets greater than MTU - IPv4

LSOv2 Large Send Offload - Offload segmentation of packets greater than 64K and support for IPv6

Checksum Offload - Offload TCP and IP checksums (required for LSO)

RSS - Receive Side Scaling - enables the efficient distribution of network receive processing across multiple CPUs in multiprocessor systems

MSI-X - Message-signaled interrupts

NETDMA v1.0 - generic interface for memory-to-memory direct memory access (DMA) transfers

SGMDA - Scatter/Gather DMA - method to transfer data between a NIC and system memory

Miniport State Machine - A chart listing the various NDIS Miniport operational states.

NDIS 6.1: Windows Vista SP1, Server 2008

IPSec Offload v2 - Improved performance, security algorithms, IPv6, and can work with LSOv2

Header data split - improves network performance by splitting the header and data in received Ethernet frames into separate buffers

NETDMA v1.1 & v2.0

NDIS 6.2 : Windows 7, Server 2008 R2

VMQ - Virtual Machine Queue - Offload technology in which networking stack offloads receive processing in the context of a virtualized environment

RST - Receive Side Throttle - reduce the possibility of glitches in multimedia application

Here is another useful link for you. Part of the Windows experience is that rarely, just sometimes, it crashes. Known worldwide as The Blue Screen of Death (BSOD), it will often come with a code. Here is the cheat sheet for those codes.

Bug Check Codes - list of all the Windows bug check codes

A lot of these links are to the MSDN resource that Microsoft provides. They are used for education purposes and do not constitute endorse of either party by either party. That should keep the lawyers happy. Is there a link about Windows Networking that makes you happy? Post it in the comments!

Thanks to Christopher P for all his help making this article!

Updated Doc notice.

webadmin@intel.com — Fri, 03 Jun 2011 22:52:42 GMT

We continually update our documentation. It can be a lot to keep track of, so here is a quick blog to let you know what we’ve revised over the last month or so.

Intel® 82599 10 GbE Controller Datasheet, 2.7

Intel® 82580EB/82580DB Gigabit Ethernet Controller Datasheet, 2.48

Intel® 82576 Gigabit Ethernet Controller Datasheet, 2.62

Intel® 82579 Gigabit Ethernet Connection Specification Update, 2.2

Intel® 82576 Gigabit Ethernet Controller Specification Update, 2.76

Intel® 82579 Gigabit Ethernet Connection Product Brief

Intel® 82571_82572 Specification Update, 6.4

We keep improving our documentation so come back to the blog for details or go to the controller page and look for your product.

Thanks for using Intel® Ethernet.