5 Replies Latest reply on Oct 4, 2012 8:51 AM by dbraden

    Packet Loss on 82579LM in Fedora 14 and 17

    dbraden

      0eHello,


      I've been trying to solve a problem I'm having with an onboard 82579LM (on a Supermicro X9SCV-Q board).  Whenever the NIC is connected to a gigabit switch and negotiates to 1000/full, I see packet loss when running UDP traffic.  This packet loss occurs at any bit rate of traffic. I've seen the problem on Fedora 14 (2.6.35.14-106.fc14.i686) and Fedora 17 (3.3.4-5.fc17.x86_64) as well as RHEL5 (2.6.32-279.9.1.e16.x86_64).  When I use ethtool -s em1 speed 100 duplex full, I can run up to 95Mbps with 0 lost packets.  Set the unit to speed 1000 duplex full and I can't run the same test without regular loss.  I've tried several versions of the e1000e drivers available from intel built several ways:

       

      e1000e-1.2.20

      e1000e-2.0.0

      e1000e-2.0.0.1

      e1000e-2.1.4

       

      I've tried building with and without NAPI and I've played with the crc stripping, interrupt throttle rate, and interrupt mode when choosing the module.  Some settings made problems worse but nothing helped.

       

      I've used ethtool to run at speed 100 duplex full and that allows for 0 loss up to 95 Mbps.  I have also tried the following other ethtool commands with no success:

       

      ethtool -G em1 rx 4096

      ethtool -G em1 tx 4096

      ethtool -S em1 speed 1000 duplex full autoneg off

      ethtool -K em1 rx off

      ethtool -K em1 rx on

      ethtool -K em1 tso off

      ethtool -K em1 gso off

      ethtool -K em1 gro off

      ethtool -K em1 rxhash off

       

       

      Any help would be appreciated.  I have 16 computers running this NIC and am hoping I can solve the problem.

       

      My actual application for this computer is streaming UDP multicast video, but to see the loss I've been testing with iperf.  My setup is as follows:

       

      On box A:

      IP - 192.168.188.126

      Subnet 255.255.255.0

      iperf -s -u -i1 -l1358

      sysctl -w net.core.rmem_max=1047856

      sysctl -w net.core.rmem_default=1047856

      sysctl -w net.core.wmem_default=1047856

      sysctl -w net.core.wmem_max=1047856

       

      On box B:

      IP - 192.168.188.127

      Subnet 255.255.255.0

      iperf -c 192.168.188.126 -b 95000000 -i1 -l1358 -t10 -d

      sysctl -w net.core.rmem_max=1047856

      sysctl -w net.core.rmem_default=1047856

      sysctl -w net.core.wmem_default=1047856

      sysctl -w net.core.wmem_max=1047856

       

      Results will look something like this on both sides when running at 1000 full:

       

      ------------------------------------------------------------

      Server listening on UDP port 5001

      Receiving 1358 byte datagrams

      UDP buffer size: 1023 KByte (default)

      ------------------------------------------------------------

      ------------------------------------------------------------

      Client connecting to 192.168.188.126, UDP port 5001

      Sending 1358 byte datagrams

      UDP buffer size: 1023 KByte (default)

      ------------------------------------------------------------

      [  4] local 192.168.188.127 port 46518 connected with 192.168.188.126 port 5001

      [  3] local 192.168.188.127 port 5001 connected with 192.168.188.126 port 48961

      [ ID] Interval       Transfer     Bandwidth

      [  4]  0.0- 1.0 sec  11.4 MBytes  95.3 Mbits/sec

      [  3]  0.0- 1.0 sec  11.4 MBytes  95.3 Mbits/sec   0.004 ms    1/ 8773 (0.011%)

      [  4]  1.0- 2.0 sec  11.4 MBytes  95.3 Mbits/sec

      [  3]  1.0- 2.0 sec  11.4 MBytes  95.3 Mbits/sec   0.004 ms    1/ 8772 (0.011%)

      [  4]  2.0- 3.0 sec  11.4 MBytes  95.3 Mbits/sec

      [  3]  2.0- 3.0 sec  11.4 MBytes  95.3 Mbits/sec   0.004 ms    1/ 8772 (0.011%)

      [  4]  3.0- 4.0 sec  11.4 MBytes  95.3 Mbits/sec

      [  3]  3.0- 4.0 sec  11.4 MBytes  95.2 Mbits/sec   0.005 ms    6/ 8771 (0.068%)

      [  4]  4.0- 5.0 sec  11.4 MBytes  95.3 Mbits/sec

      [  3]  4.0- 5.0 sec  11.4 MBytes  95.3 Mbits/sec   0.001 ms    0/ 8772 (0%)

      [  4]  5.0- 6.0 sec  11.4 MBytes  95.3 Mbits/sec

      [  3]  5.0- 6.0 sec  11.4 MBytes  95.2 Mbits/sec   0.002 ms    5/ 8772 (0.057%)

      [  4]  6.0- 7.0 sec  11.4 MBytes  95.3 Mbits/sec

      [  3]  6.0- 7.0 sec  11.4 MBytes  95.3 Mbits/sec   0.002 ms    2/ 8772 (0.023%)

      [  4]  7.0- 8.0 sec  11.4 MBytes  95.3 Mbits/sec

      [  3]  7.0- 8.0 sec  11.4 MBytes  95.3 Mbits/sec   0.003 ms    2/ 8772 (0.023%)

      [  4]  8.0- 9.0 sec  11.4 MBytes  95.3 Mbits/sec

      [  3]  8.0- 9.0 sec  11.4 MBytes  95.3 Mbits/sec   0.002 ms    4/ 8772 (0.046%)

      [  4]  9.0-10.0 sec  11.4 MBytes  95.3 Mbits/sec

      [  4]  0.0-10.0 sec   114 MBytes  95.3 Mbits/sec

      [  4] Sent 87720 datagrams

      [  3]  0.0-10.0 sec   114 MBytes  95.3 Mbits/sec   0.003 ms   24/87719 (0.027%)

      [  3]  0.0-10.0 sec  1 datagrams received out-of-order

      [  4] Server Report:

      [  4]  0.0-10.0 sec   113 MBytes  95.2 Mbits/sec   0.001 ms  100/87719 (0.11%)

      [  4]  0.0-10.0 sec  1 datagrams received out-of-order

       


      When running at 100 full

       

      WARNING: option -b implies udp testing

      ------------------------------------------------------------

      Server listening on UDP port 5001

      Receiving 1358 byte datagrams

      UDP buffer size: 1023 KByte (default)

      ------------------------------------------------------------

      ------------------------------------------------------------

      Client connecting to 192.168.188.126, UDP port 5001

      Sending 1358 byte datagrams

      UDP buffer size: 1023 KByte (default)

      ------------------------------------------------------------

      [  4] local 192.168.188.127 port 45827 connected with 192.168.188.126 port 5001

      [  3] local 192.168.188.127 port 5001 connected with 192.168.188.126 port 57617

      [ ID] Interval       Transfer     Bandwidth

      [  4]  0.0- 1.0 sec  11.4 MBytes  95.3 Mbits/sec

      [  3]  0.0- 1.0 sec  11.4 MBytes  95.3 Mbits/sec   0.121 ms    0/ 8773 (0%)

      [  4]  1.0- 2.0 sec  11.4 MBytes  95.3 Mbits/sec

      [  3]  1.0- 2.0 sec  11.4 MBytes  95.3 Mbits/sec   0.121 ms    0/ 8772 (0%)

      [  4]  2.0- 3.0 sec  11.4 MBytes  95.3 Mbits/sec

      [  3]  2.0- 3.0 sec  11.4 MBytes  95.3 Mbits/sec   0.121 ms    0/ 8772 (0%)

      [  4]  3.0- 4.0 sec  11.4 MBytes  95.3 Mbits/sec

      [  3]  3.0- 4.0 sec  11.4 MBytes  95.3 Mbits/sec   0.122 ms    0/ 8770 (0%)

      [  4]  4.0- 5.0 sec  11.4 MBytes  95.3 Mbits/sec

      [  3]  4.0- 5.0 sec  11.4 MBytes  95.3 Mbits/sec   0.120 ms    0/ 8773 (0%)

      [  4]  5.0- 6.0 sec  11.4 MBytes  95.3 Mbits/sec

      [  3]  5.0- 6.0 sec  11.4 MBytes  95.3 Mbits/sec   0.122 ms    0/ 8772 (0%)

      [  4]  6.0- 7.0 sec  11.4 MBytes  95.3 Mbits/sec

      [  3]  6.0- 7.0 sec  11.4 MBytes  95.3 Mbits/sec   0.121 ms    0/ 8773 (0%)

      [  4]  7.0- 8.0 sec  11.4 MBytes  95.3 Mbits/sec

      [  3]  7.0- 8.0 sec  11.4 MBytes  95.3 Mbits/sec   0.121 ms    0/ 8771 (0%)

      [  4]  8.0- 9.0 sec  11.4 MBytes  95.3 Mbits/sec

      [  3]  8.0- 9.0 sec  11.4 MBytes  95.3 Mbits/sec   0.121 ms    0/ 8772 (0%)

      [  4]  9.0-10.0 sec  11.4 MBytes  95.3 Mbits/sec

      [  4]  0.0-10.0 sec   114 MBytes  95.3 Mbits/sec

      [  4] Sent 87720 datagrams

      [  3]  0.0-10.0 sec   114 MBytes  95.3 Mbits/sec   0.122 ms    0/87719 (0%)

      [  3]  0.0-10.0 sec  1 datagrams received out-of-order

      [  4] Server Report:

      [  4]  0.0-10.0 sec   114 MBytes  95.3 Mbits/sec   0.120 ms    0/87719 (0%)

      [  4]  0.0-10.0 sec  1 datagrams received out-of-order

       

      Message was edited by: Doug Braden - Added testing on RHEL5

        • 1. Re: Packet Loss on 82579LM in Fedora 14 and 17
          mark_h_@intel

          This issue might be fixed by a BIOS update. I notice that Super Micro has a recent BIOS update for your motherboard. I recommend giving that a try.

           

          Mark H

          • 2. Re: Packet Loss on 82579LM in Fedora 14 and 17
            dbraden

            I updated to the most recent Supermicro BIOS (X9SCVQ2.829) and the behavior is the same.  Other thoughts?  I know my hardware is all capable outside the box as I've run the same tests just moving to the second NIC (82574) and it's clean as a whistle.

             

            As another data point, I ran iperf as TCP and I can get 742 Mbps across the link.  The problem seems to be limited to UDP or masked by the acks in TCP. 

             

            On box A:

            IP - 192.168.188.126

            Subnet 255.255.255.0

            iperf -s -i1 -l1358

            sysctl -w net.core.rmem_max=1047856

            sysctl -w net.core.rmem_default=1047856

            sysctl -w net.core.wmem_default=1047856

            sysctl -w net.core.wmem_max=1047856

             

            On box B:

            IP - 192.168.188.127

            Subnet 255.255.255.0

            iperf -c 192.168.188.126 -i1 -l1358 -t10

            sysctl -w net.core.rmem_max=1047856

            sysctl -w net.core.rmem_default=1047856

            sysctl -w net.core.wmem_default=1047856

            sysctl -w net.core.wmem_max=1047856

            • 3. Re: Packet Loss on 82579LM in Fedora 14 and 17
              dbraden

              As another observation, I loaded Windows 7 Professional on the same hardware and ran the same test (using driver 17.3).  The results are actually worse.  In this case iperf pushed ~50Mbps despite running on a 1000/Full connection.  It also still had packet loss even at 50Mbps.  This is a clean install of Windows 7 Professional no other software loaded just the driver and iperf. 

               

              iperf -c 192.168.188.126 -b 100000000 -t10 -i1 -l1358 -d

              WARNING: option -b implies udp testing

              ------------------------------------------------------------

              Server listening on UDP port 5001

              Receiving 1358 byte datagrams

              UDP buffer size: 1021 KByte (default)

              ------------------------------------------------------------

              ------------------------------------------------------------

              Client connecting to 192.168.188.126, UDP port 5001

              Sending 1358 byte datagrams

              UDP buffer size: 1021 KByte (default)

              ------------------------------------------------------------

              [  5] local 192.168.188.127 port 60908 connected with 192.168.188.126 port 5001

              [  3] local 192.168.188.127 port 5001 connected with 192.168.188.126 port 64112

              [ ID] Interval       Transfer     Bandwidth

              [  5]  0.0- 1.0 sec  12.0 MBytes   101 Mbits/sec

              [  3]  0.0- 1.0 sec  5.95 MBytes  49.9 Mbits/sec   0.110 ms    0/ 4597 (0%)

              [  5]  1.0- 2.0 sec  12.0 MBytes   101 Mbits/sec

              [  3]  1.0- 2.0 sec  5.86 MBytes  49.1 Mbits/sec   0.108 ms    0/ 4524 (0%)

              [  5]  2.0- 3.0 sec  12.0 MBytes   101 Mbits/sec

              [  3]  2.0- 3.0 sec  6.08 MBytes  51.0 Mbits/sec   0.645 ms    0/ 4693 (0%)

              [  5]  3.0- 4.0 sec  12.0 MBytes   101 Mbits/sec

              [  3]  3.0- 4.0 sec  6.58 MBytes  55.2 Mbits/sec   0.108 ms    1/ 5078 (0.02%)

              [  5]  4.0- 5.0 sec  12.0 MBytes   101 Mbits/sec

              [  3]  4.0- 5.0 sec  6.24 MBytes  52.3 Mbits/sec   0.636 ms    2/ 4817 (0.042%)

              [  5]  5.0- 6.0 sec  12.0 MBytes   101 Mbits/sec

              [  3]  5.0- 6.0 sec  6.22 MBytes  52.2 Mbits/sec   0.340 ms    1/ 4806 (0.021%)

              [  5]  6.0- 7.0 sec  12.0 MBytes   101 Mbits/sec

              [  3]  6.0- 7.0 sec  5.87 MBytes  49.2 Mbits/sec   0.183 ms    1/ 4532 (0.022%)

              [  5]  7.0- 8.0 sec  12.0 MBytes   101 Mbits/sec

              [  3]  7.0- 8.0 sec  6.04 MBytes  50.7 Mbits/sec   0.146 ms    0/ 4665 (0%)

              [  5]  8.0- 9.0 sec  12.0 MBytes   101 Mbits/sec

              [  3]  8.0- 9.0 sec  6.23 MBytes  52.2 Mbits/sec   0.120 ms    3/ 4812 (0.062%)

              [  5]  9.0-10.0 sec  12.0 MBytes   101 Mbits/sec

              [  5]  0.0-10.0 sec   120 MBytes   101 Mbits/sec

              [  5] Sent 92593 datagrams

              [  3]  9.0-10.0 sec  6.05 MBytes  50.7 Mbits/sec   0.092 ms    1/ 4671 (0.021%)

              [  5] Server Report:

              [  5]  0.0-10.0 sec   120 MBytes   101 Mbits/sec   0.125 ms    0/92592 (0%)

              [  5]  0.0-10.0 sec  1 datagrams received out-of-order

              [  3]  0.0-10.1 sec  61.2 MBytes  51.1 Mbits/sec   0.899 ms    8/47261 (0.017%)

              [  3]  0.0-10.1 sec  1 datagrams received out-of-order

              • 4. Re: Packet Loss on 82579LM in Fedora 14 and 17
                mark_h_@intel

                Erratum 23 in the Intel® 6 Series Chipset, Intel® C200 Series Chipset: Spec Update lists an issue titled Packet Loss on Intel® 82579 Gigabit Ethernet Controller. The workaround says, "A BIOS code change has been identified and may be implemented as a workaround for this erratum." That was why I suggested the BIOS update. However, this specification update was published in August, so Super Micro's posted BIOS update might not include this workaround. I recommend that you check with Super Micro concerning the availability of a BIOS update that contains the workaround for the erratum.

                 

                Mark H

                • 5. Re: Packet Loss on 82579LM in Fedora 14 and 17
                  dbraden

                  Mark,

                   

                  Thanks for the link to the spec document.  I have a parallel conversation going with Supermicro and have asked them whether this change is reflected in the August BIOS I flashed to.  I will update if there is a change.