10 Replies Latest reply on Feb 15, 2017 10:23 PM by snowman386

    X710 dropping LLDP frames ?

    michelm

      Hello,

       

      We have servers with 2 XL710 quad cards each.

       

      In first, we got this firmware version:

      firmware-version: f4.22 a1.1 n04.26 e8000152d

      LLDP was working fine, no problem, trames are correctly received.

       

      But after upgrading to (seems to be the last version):

      firmware-version: 4.53 0x80001da6 0.0.0

      We don't receive any LLDP frames anymore.

      It seems that theses frames are dropped by the X710 cards.

       

      I tried to downgrade to 4.42 firmware but ... same thing

       

      The problem occurs on both vSphere 6.0 and Linux (RHEL 6.6) on the 5 servers that have theses ethernet adapter and their firmware updated

       

      We have other cards from other vendor on theses servers that are plugged on the same network equipments and they receive LLDP frames without problems.

       

      Some debug informations (vsphere side, we upgraded the driver to the last version according the HCL, but the problem is the same with the "base" driver):

      # lspci|grep Ethernet

      0000:02:00.0 Network controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet [vmnic0]

      0000:02:00.1 Network controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet [vmnic1]

      0000:02:00.2 Network controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet [vmnic2]

      0000:02:00.3 Network controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet [vmnic3]

      0000:04:00.0 Network controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet [vmnic4]

      0000:04:00.1 Network controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet [vmnic5]

      0000:08:00.0 Network controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [vmnic10]

      0000:08:00.1 Network controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [vmnic11]

      0000:08:00.2 Network controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [vmnic12]

      0000:08:00.3 Network controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [vmnic13]

      0000:0b:00.0 Network controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [vmnic6]

      0000:0b:00.1 Network controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [vmnic7]

      0000:0b:00.2 Network controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [vmnic8]

      0000:0b:00.3 Network controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [vmnic9]

      # vmkchdev -l|grep vmnic6

      0000:0b:00.0 8086:1572 8086:0002 vmkernel vmnic6

      # ethtool -i vmnic6

      driver: i40e

      version: 1.3.38

      firmware-version: 4.53 0x80001da6 0.0.0

      bus-info: 0000:0b:00.0

       

       

       

       

      Some debug informations (Linux side, i40e driver upgraded too, but problem is the same with the "base" driver):

      # lspci | grep Ethernet

      02:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)

      02:00.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)

      02:00.2 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)

      02:00.3 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)

      04:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)

      04:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)

      08:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)

      08:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)

      08:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)

      08:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)

      0b:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)

      0b:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)

      0b:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)

      0b:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)

       

      # ethtool -i eth6

      driver: i40e

      version: 1.3.39.1

      firmware-version: 4.53 0x80001da6 0.0.0

      bus-info: 0000:0b:00.0

      supports-statistics: yes

      supports-test: yes

      supports-eeprom-access: yes

      supports-register-dump: yes

      supports-priv-flags: yes

       

      # ethtool -k eth6

      Features for eth6:

      rx-checksumming: on

      tx-checksumming: on

              tx-checksum-ipv4: on

              tx-checksum-unneeded: off

              tx-checksum-ip-generic: off

              tx-checksum-ipv6: on

              tx-checksum-fcoe-crc: off [fixed]

              tx-checksum-sctp: on [fixed]

      scatter-gather: on

              tx-scatter-gather: on

              tx-scatter-gather-fraglist: off [fixed]

      tcp-segmentation-offload: on

              tx-tcp-segmentation: on

              tx-tcp-ecn-segmentation: on

              tx-tcp6-segmentation: on

      udp-fragmentation-offload: off [fixed]

      generic-segmentation-offload: on

      generic-receive-offload: on

      large-receive-offload: off

      rx-vlan-offload: on

      tx-vlan-offload: on

      ntuple-filters: on

      receive-hashing: on

      highdma: on [fixed]

      rx-vlan-filter: on [fixed]

      vlan-challenged: off [fixed]

      tx-lockless: off [fixed]

      netns-local: off [fixed]

      tx-gso-robust: off [fixed]

      tx-fcoe-segmentation: off [fixed]

      tx-gre-segmentation: off [fixed]

      tx-udp_tnl-segmentation: off [fixed]

      fcoe-mtu: off [fixed]

      loopback: off [fixed]


       

       

      vSphere says that "Link Layer Discovery Protocol is not available on this physical network adapter"

      LLDPD under Linux don't output any informations on neighbors switchs for the XL710's cards

       

      From switch perspective we are seeing the LLDP frame well transmitted on the port !

      But when trying to capture LLDP frames both tcpdump under linux (tcpdump -i eth6 -s 1500 -XX 'ether proto 0x88cc') and pktcap-uw under vSphere (pktcap-uw --uplink vmnic1 --ethtype 0x88cc -c 1) never output any frames.

        • 1. Re: X710 dropping LLDP frames ?
          Sandy_Intel

          Hi michelm,

           

               The LLDP may have been disabled when you updated the firmware.  I'll check on this further.

           

          Sincerely,

           

          Sandy

          • 2. Re: X710 dropping LLDP frames ?
            michelm

            Hi Sandy,

             

            First, thanks for your reply.

             

            Yes, LLDP seems disabled.

            Perhaps did you known how to enable it again ?

            I tried to play with some settings thanks to 'ethtool -K ...' but no luck and there's no clear settings about LLDP.

             

            MM.

            • 3. Re: X710 dropping LLDP frames ?
              Sandy_Intel

              Hi michelm,

               

              Please refer to "Configuring Link Layer Discovery Protocol agent daemon (lldpad)"

               

              Start lldpad service and configure to start at boot time.

              root# service lldpad start

              Starting lldpad: [done]                [ OK ]

              root# chkconfig lldpad on

               

              NOTE: There is no output from this command but it will enable lldpad to automatically start when the system

              is booted.

               

              User Guide:

              http://www.intel.com/content/dam/www/public/us/en/documents/guides/fcoe-user-guide.pdf

               

              Sincerely,

               

              Sandy

              • 4. Re: X710 dropping LLDP frames ?
                michelm

                Hello Sandy,

                 

                Same result when installing and enabling some settings thanks to LLDPAD.

                LLDP RX frames are never seen from OS side ..

                 

                Here what I do (after installing and starting lldpad):

                 

                # dmesg | grep i40e|grep eth|grep Link

                i40e 0000:08:00.0: eth2: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None

                i40e 0000:08:00.1: eth3: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None

                i40e 0000:08:00.2: eth4: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None

                i40e 0000:08:00.3: eth5: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None

                i40e 0000:0b:00.0: eth6: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None

                i40e 0000:0b:00.1: eth7: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None

                i40e 0000:0b:00.2: eth8: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None

                i40e 0000:0b:00.3: eth9: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None

                # for i in `seq 2 9`; do lldptool set-lldp -i eth$i adminStatus=rxtx  ; lldptool -T -i eth$i -V  sysName enableTx=yes; lldptool -T -i eth$i -V  portDesc enableTx=yes ; lldptool -T -i eth$i -V  sysDesc enableTx=yes; lldptool -T -i eth$i -V sysCap enableTx=yes; lldptool -T -i eth$i -V mngAddr enableTx=yes; done

                Doing eth2

                adminStatus = rxtx

                enableTx = yes

                enableTx = yes

                enableTx = yes

                enableTx = yes

                enableTx = yes

                Doing eth3

                adminStatus = rxtx

                enableTx = yes

                enableTx = yes

                enableTx = yes

                enableTx = yes

                enableTx = yes

                Doing eth4

                adminStatus = rxtx

                enableTx = yes

                enableTx = yes

                enableTx = yes

                enableTx = yes

                enableTx = yes

                Doing eth5

                adminStatus = rxtx

                enableTx = yes

                enableTx = yes

                enableTx = yes

                enableTx = yes

                enableTx = yes

                Doing eth6

                adminStatus = rxtx

                enableTx = yes

                enableTx = yes

                enableTx = yes

                enableTx = yes

                enableTx = yes

                Doing eth7

                adminStatus = rxtx

                enableTx = yes

                enableTx = yes

                enableTx = yes

                enableTx = yes

                enableTx = yes

                Doing eth8

                adminStatus = rxtx

                enableTx = yes

                enableTx = yes

                enableTx = yes

                enableTx = yes

                enableTx = yes

                Doing eth9

                adminStatus = rxtx

                enableTx = yes

                enableTx = yes

                enableTx = yes

                enableTx = yes

                enableTx = yes

                 

                # /etc/init.d/lldpad restart

                Stopping lldpad:                                           [  OK  ]

                Starting lldpad:                                           [  OK  ]

                 

                 

                <WAIT FOR 2Minutes>

                 

                # lldptool get-lldp -i eth6 adminStatus

                adminStatus=rxtx

                # lldptool -t -n -i eth6

                <NOTHING>

                # tcpdump -i eth6 -s 1500 -XX 'ether proto 0x88c
                <ONLY SHOW TRANSMITTED FRAMES>

                 

                Some informations regarding DCB:

                # dcbtool gc eth6 dcb

                Command:        Get Config

                Feature:        DCB State

                Port:           eth6

                Status:         Device not capable

                # dcbtool sc eth6 dcb on

                Command:        Set Config

                Feature:        DCB State

                Port:           eth6

                Status:         Device not capable

                 

                 

                Seems that's impossible to enable it ?

                 

                Driver/Frimware version (same as below):

                # ethtool -i eth6

                driver: i40e

                version: 1.3.39.1

                firmware-version: 4.53 0x80001da6 0.0.0

                bus-info: 0000:0b:00.0

                supports-statistics: yes

                supports-test: yes

                supports-eeprom-access: yes

                supports-register-dump: yes

                supports-priv-flags: yes

                 

                Tests outputed with eth6 but it's the same result with any other ethernet interface from i40e.

                 

                It really seems that LLDP RX Frames are dropped directly inside the NIC (firmware setting to tune ?) whatever the OS (vSphere side, same result, LLDP frames are never received on the ESXi).


                MM.

                • 5. Re: X710 dropping LLDP frames ?
                  jbrandeb

                  LLDP is meant to be handled directly on the NIC in the X710, and the firmware defaults to this feature being on.  You can work around the issue (at least under linux) using

                  # echo lldp stop > /sys/kernel/debug/i40e/<bus.dev.fn>/command


                  • 6. Re: X710 dropping LLDP frames ?
                    michelm

                    Great, it's working now.

                    Perhaps Intel should put this in a documentation somewhere ... ?

                     

                    Thank you jbrandeb ! :-)

                    • 7. Re: X710 dropping LLDP frames ?
                      cloaked1

                      This seems to be the root cause of the problem we're having as well. However, it seems that the kernel version we're using (2.6.x) requires a different method to disable the internal LLDP agent. When I attempt to echo 'lldp stop' to the path specified above, I get an error and no matter what I try, I'm unable to issue the command in the manner stated. Does Intel provide *any* documentation on how to disable this internal LLDP agent for Linux 2.6.x and 3.x?

                       

                      Also asked:

                      centos6 - Disable internal Intel X710 LLDP agent - Server Fault

                       

                      TIA, Jim

                      • 8. Re: X710 dropping LLDP frames ?
                        cloaked1

                        Found the steps I needed to make the above (correct) step work. Please follow the link to ServerFault for the steps required if you're having the same issues. For the record, I'm quite disappointed in Intel for not making this a way more transparent process/issue.

                        • 9. Re: X710 dropping LLDP frames ?
                          olivier.romain

                          Hello,

                           

                          I am having the same problem. I use a XL710 card in HP DL380 servers, running a Debian 7 OS.

                           

                          We found out the card do not bring LLDP packets to the Linux stack, and discard it. This behavior is different from X520 cards which pass the LLDP packets to the linux stack.

                           

                          The work around like '# echo lldp stop > /sys/kernel/debug/i40e/<bus.dev.fn>/command' works, but we cannot do that on production systems.

                           

                          Could intel provide some way to set this value in a proper manner (like with ethtool) in the next driver version? Or simply change the default value so that the default behaviour is the same as X520 cards.

                           

                          Thanks in advance.

                           

                          Olivier

                          • 10. Re: X710 dropping LLDP frames ?
                            snowman386

                            Is there a way to do this from esxi? I need esxi to exchange lldp info with our switches for our SDN product to do its magic. ESXi is currently showing that the physical adapter is not lldp capable. Updating firmware and driver to the latest did not help.

                            Thanks