Flow control is a key part of keeping your 1 Gigabit and faster network running smoothly.  Somewhere along the line some websites started telling people to turn off flow control so their network would go faster.  In the short term this might be fine; in the long term you’re going to see bigger problems and probably drop more packets than you’ll make up by being able to send as needed.  The problem people would say is that Flow Control stops the traffic, and this costs performance.  Absolutely it stops traffic.  But it stops the traffic the receiver doesn’t have room for!  The flow control is like a stop light controlling access to the highway.  Instead of letting them all in at once, when there is no room for them and gumming up the works even further, flow control gives protection to the receiver.  This protection allows for long term speed and less dropped packets.

 

Consider the following data set that was gathered using ethtool -S eth0 from a real system.

NIC statistics:

     rx_packets: 329461
      tx_packets: 302120
      rx_bytes: 34897969
      tx_bytes: 32293428
      rx_no_buffer_count: 39147
      rx_missed_errors: 1097931
      rx_flow_control_xon: 0
      rx_flow_control_xoff: 0
      tx_flow_control_xon: 228
      tx_flow_control_xoff: 1098233

 

Let’s look at it in detail.  Tx Flow control XOFF is the NIC telling the link partner, “I’m overwhelmed, stop the packets”, Rx Flow Control is the Link partner telling the NIC “I’m overwhelmed, stop the packets”.  Note the difference.  TX FC is transmitting TO the partner, RX FC is receiving FROM the partner.  In this case, the NIC is basically screaming, I’m overwhelmed (More XOFF than packets), and the rx_no_buffer_count and rx_missed_error confirms it.  What this means is the NIC has no resources and is actively dropping packets.  But FC is on!  Why are we still dropping/missing packets?  The link partner is not honoring the flow control packets! In this case, the link partner has sent 1.4 million frames, but only 300K got through because the link partner didn’t care about flow control.  With flow control the packets might take a little longer to get there, but they will get there.

 

Looking at the data, see the 228 XON?  The NIC only caught up 228 times.  That’s not so good.  So what was causing all these missed packets?  Most likely cause is a slow PCI express and/or slow memory implementation.  Packets come all the time and memory slowness and getting combined with another busy device on a few narrow lanes can mean not enough PCI Express bus between something like the ESB or a switch cascaded off a switch.

Moving to 10 Gigabit it is, well, ten times worse.  You have 1/10th the time and ripple effect of delaying a packet moves faster.  It was so problematic that Data Center Bridging (DCB) and DCBx came out to make flow control end to end.  Instead of just link partner to link partner, DCBx allows one overwhelmed end point to tell the overwhelming source to chill out.  This moves the delay caused by flow control to the point most able to deal with it.  While some backplanes of switches can temporarily store terabits of data, having the starting node just not send it right now is the best result.  We’ll do a deeper dive on DCBx another time, but with it you get effectively lossless Ethernet with DCBx and that lets you do FCoE and other storage technologies.

 

Thanks for using Intel Ethernet and turn on your Flow Control!!