6 Replies Latest reply on Jan 6, 2017 12:11 AM by U-map

    Unable to establish link with Intel 82599ES 10-Gigabit SFI/SFP+ Network Connection

    jmarshall80

      Hi,

       

      I'm looking for help establishing (and troubleshooting) a link to a SPAN in promiscuous mode on Ubuntu Server 14.04.

       

      This link was up and running and seeing packets for a few weeks. To help with an application issue, we installed the PF_RING kernel module, and after a server reboot, we have not been able to establish a link. We've rolled back our changes and removed PF_RING, but we are still not able to establish a link, and I'm looking for some help.

       

      We've looked at our Gigamon SPAN and it appears to be sending data to our device but we have not been able to see any traffic ('tcpdump -i em1 -vv' does not see any traffic).

       

      Any help would be greatly appreciated.

       

      Jeff

       

      $ifconfig  em1

      em1      Link encap:Ethernet  HWaddr ec:f4:bb:**:**:**

                UP BROADCAST NOARP PROMISC MULTICAST  MTU:1500  Metric:1

                RX packets:0 errors:0 dropped:7180 overruns:0 frame:0

                TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

                collisions:0 txqueuelen:1000

                RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

       

       

      $ sudo ethtool em1

      Settings for em1:

        Supported ports: [ FIBRE ]

        Supported link modes:   1000baseT/Full

                                10000baseT/Full

        Supported pause frame use: No

        Supports auto-negotiation: Yes

        Advertised link modes:  1000baseT/Full

                                10000baseT/Full

        Advertised pause frame use: No

        Advertised auto-negotiation: Yes

        Speed: Unknown!

        Duplex: Unknown! (255)

        Port: FIBRE

        PHYAD: 0

        Transceiver: external

        Auto-negotiation: on

        Supports Wake-on: umbg

        Wake-on: g

        Current message level: 0x00000007 (7)

              drv probe link

        Link detected: no

       

      $ cat /var/log/syslog | grep ixgbe

      Feb 17 15:16:46 servername kernel: [    7.371724] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 3.15.1-k

      Feb 17 15:16:46 servername kernel: [    7.371727] ixgbe: Copyright (c) 1999-2013 Intel Corporation.

       

      $ cat /var/log/syslog | grep em1

      Feb 17 15:16:46 servername kernel: [  11.789614] systemd-udevd[572]: renamed network interface eth0 to em1

      Feb 17 15:16:46 servername kernel: [  14.071614] IPv6: ADDRCONF(NETDEV_UP): em1: link is not ready

      Feb 17 15:16:46 servername kernel: [  15.404429] ixgbe 0000:01:00.0: registered PHC device on em1

      Feb 17 15:16:46 servername kernel: [  15.509280] IPv6: ADDRCONF(NETDEV_UP): em1: link is not ready

      Feb 17 15:16:47 servername kernel: [  15.580413] ixgbe 0000:01:00.0 em1: detected SFP+: 6

      Feb 17 15:16:48 servername kernel: [  16.713425] device em1 entered promiscuous mode

      Feb 17 15:16:48 servername kernel: [  17.498677] ixgbe 0000:01:00.0 em1: detected SFP+: 6

      $sudo ethtool -m em1

        Identifier                                : 0x03 (SFP)

        Extended identifier                      : 0x04 (GBIC/SFP defined by 2-wire interface ID)

        Connector                                : 0x07 (LC)

        Transceiver codes                        : 0x10 0x00 0x00 0x01 0x00 0x00 0x00 0x00

        Transceiver type                          : 10G Ethernet: 10G Base-SR

        Transceiver type                          : Ethernet: 1000BASE-SX

        Encoding                                  : 0x06 (64B/66B)

        BR, Nominal                              : 10300MBd

        Rate identifier                          : 0x02 (8/4/2G Rx Rate_Select only)

        Length (SMF,km)                          : 0km

        Length (SMF)                              : 0m

        Length (50um)                            : 80m

        Length (62.5um)                          : 30m

        Length (Copper)                          : 0m

        Length (OM3)                              : 300m

        Laser wavelength                          : 850nm

        Vendor name                              : Intel Corp     

        Vendor OUI                                : 00:1b:21

        Vendor PN                                : FTLX8571D3BCV-IT

        Vendor rev                                : A 

        Optical diagnostics support              : Yes

        Laser bias current                        : 26.446 mA

        Laser output power                        : 0.6639 mW / -1.78 dBm

        Receiver signal average optical power    : 0.0014 mW / -28.54 dBm

        Module temperature                        : 51.65 degrees C / 124.97 degrees F

        Module voltage                            : 3.3214 V

        Alarm/warning flags implemented          : Yes

        Laser bias current high alarm            : Off

        Laser bias current low alarm              : Off

        Laser bias current high warning          : Off

        Laser bias current low warning            : Off

        Laser output power high alarm            : Off

        Laser output power low alarm              : Off

        Laser output power high warning          : Off

        Laser output power low warning            : Off

        Module temperature high alarm            : Off

        Module temperature low alarm              : Off

        Module temperature high warning          : Off

        Module temperature low warning            : Off

        Module voltage high alarm                : Off

        Module voltage low alarm                  : Off

        Module voltage high warning              : Off

        Module voltage low warning                : Off

        Laser rx power high alarm                : Off

        Laser rx power low alarm                  : On

        Laser rx power high warning              : Off

        Laser rx power low warning                : On

        Laser bias current high alarm threshold  : 11.800 mA

        Laser bias current low alarm threshold    : 2.000 mA

        Laser bias current high warning threshold : 10.800 mA

        Laser bias current low warning threshold  : 3.000 mA

        Laser output power high alarm threshold  : 0.8318 mW / -0.80 dBm

        Laser output power low alarm threshold    : 0.1585 mW / -8.00 dBm

        Laser output power high warning threshold : 0.6607 mW / -1.80 dBm

        Laser output power low warning threshold  : 0.1995 mW / -7.00 dBm

        Module temperature high alarm threshold  : 78.00 degrees C / 172.40 degrees F

        Module temperature low alarm threshold    : -13.00 degrees C / 8.60 degrees F

        Module temperature high warning threshold : 73.00 degrees C / 163.40 degrees F

        Module temperature low warning threshold  : -8.00 degrees C / 17.60 degrees F

        Module voltage high alarm threshold      : 3.7000 V

        Module voltage low alarm threshold        : 2.9000 V

        Module voltage high warning threshold    : 3.6000 V

        Module voltage low warning threshold      : 3.0000 V

        Laser rx power high alarm threshold      : 1.0000 mW / 0.00 dBm

        Laser rx power low alarm threshold        : 0.0100 mW / -20.00 dBm

        Laser rx power high warning threshold    : 0.7943 mW / -1.00 dBm

        Laser rx power low warning threshold      : 0.0158 mW / -18.01 dBm

        • 1. Re: Unable to establish link with Intel 82599ES 10-Gigabit SFI/SFP+ Network Connection
          wb_Intel

          HI Jmarshall80,

           

          What is the exact 10 Gigabit NIC used here? Is it a X520series ? and what is the exact model?

          This will enable us to further assist on this.

           

          thanks,

          wb

          • 2. Re: Unable to establish link with Intel 82599ES 10-Gigabit SFI/SFP+ Network Connection
            jmarshall80

            Hi,

             

            This is a "Intel X520 DP 10Gb DA/SFP+, + I350 DP 1Gb Ethernet, Network Daughter Card"

             

            Thanks,

             

            Jeff

            • 3. Re: Unable to establish link with Intel 82599ES 10-Gigabit SFI/SFP+ Network Connection
              wb_Intel

              Hi jmarshall80,

               

              thanks for the info, will check on this.

               

              rgds,

              wb

              • 4. Re: Unable to establish link with Intel 82599ES 10-Gigabit SFI/SFP+ Network Connection
                wb_Intel

                Hi jmarshall80,

                Please help share the ff info:

                1) output of ethtool -t

                2. The setup script for further investigation on why the link fails after installing PF_ring

                3)  Use 10Gbps connection instead of 1Gbps

                4) Based on your log info, you are using Finisar ftlx8571d3bcv which is not a compatible SFP+ module

                (http://www.intel.com/support/network/adapter/pro100/sb/CS-030612.htm)

                 

                Thanks,

                wb

                • 5. Re: Unable to establish link with Intel 82599ES 10-Gigabit SFI/SFP+ Network Connection
                  U-map

                  Hi,

                   

                  We are experiencing similar issue when deploying a production system. We have installed PF_RING with ZC prior to cabling and ethtool says "No link detected" after connecting fiber. Uninstalling PF_RING+ZC and updating ixgbe driver didn't help to resolve the issue.

                   

                  Linux  4.4.0-57-generic x86_64

                   

                  DISTRIB_ID=Ubuntu

                  DISTRIB_RELEASE=16.04

                   

                  05:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

                          Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 560SFP+ Adapter

                          Kernel driver in use: ixgbe

                          Kernel modules: ixgbe, ixgbe_zc

                  05:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

                          Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 560SFP+ Adapter

                          Kernel driver in use: ixgbe

                          Kernel modules: ixgbe, ixgbe_zc

                  08:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

                          Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 560SFP+ Adapter

                          Kernel driver in use: ixgbe

                          Kernel modules: ixgbe, ixgbe_zc

                  08:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

                          Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 560SFP+ Adapter

                          Kernel driver in use: ixgbe

                          Kernel modules: ixgbe, ixgbe_zc

                   

                  root@host1:~# ethtool -i ens1f1

                  driver: ixgbe

                  version: 4.2.1-k

                  firmware-version: 0x80000897

                  expansion-rom-version:

                  bus-info: 0000:05:00.1

                  supports-statistics: yes

                  supports-test: yes

                  supports-eeprom-access: yes

                  supports-register-dump: yes

                  supports-priv-flags: no

                  root@host1:~# ethtool ens1f1

                  Settings for ens1f1:

                          Supported ports: [ FIBRE ]

                          Supported link modes:   10000baseT/Full

                          Supported pause frame use: No

                          Supports auto-negotiation: No

                          Advertised link modes:  10000baseT/Full

                          Advertised pause frame use: No

                          Advertised auto-negotiation: No

                          Speed: Unknown!

                          Duplex: Unknown! (255)

                          Port: FIBRE

                          PHYAD: 0

                          Transceiver: external

                          Auto-negotiation: off

                          Supports Wake-on: d

                          Wake-on: d

                          Current message level: 0x00000007 (7)

                                                 drv probe link

                          Link detected: no

                   

                   

                  root@host1:~# ethtool -t ens1f1

                  The test result is FAIL

                  The test extra info:

                  Register test  (offline)         0

                  Eeprom test    (offline)         0

                  Interrupt test (offline)         0

                  Loopback test  (offline)         0

                  Link test   (on/offline)         1

                   

                  Dec 22 17:14:03  kernel: [   13.404410] IPv6: ADDRCONF(NETDEV_UP): eno1: link is not ready

                  Dec 22 17:14:03  kernel: [   13.741596] cgroup: new mount options do not match the existing superblock, will be ignored

                  Dec 22 17:14:06  kernel: [   16.917021] tg3 0000:02:00.0 eno1: Link is up at 1000 Mbps, full duplex

                  Dec 22 17:14:06  kernel: [   16.917035] tg3 0000:02:00.0 eno1: Flow control is off for TX and off for RX

                  Dec 22 17:14:06 kernel: [   16.917037] tg3 0000:02:00.0 eno1: EEE is enabled

                  Dec 22 17:14:06  kernel: [   16.917073] IPv6: ADDRCONF(NETDEV_CHANGE): eno1: link becomes ready

                  Dec 22 17:21:09  kernel: [  439.915859] ixgbe 0000:05:00.0: registered PHC device on ens1f0

                  Dec 22 17:21:09  kernel: [  440.019386] IPv6: ADDRCONF(NETDEV_UP): ens1f0: link is not ready

                  Dec 22 17:21:09  kernel: [  440.083870] ixgbe 0000:05:00.0 ens1f0: detected SFP+: 6

                  Dec 22 17:22:04  kernel: [  494.852207] ixgbe 0000:05:00.1: registered PHC device on ens1f1

                  Dec 22 17:22:04  kernel: [  494.953737] IPv6: ADDRCONF(NETDEV_UP): ens1f1: link is not ready

                  Dec 22 17:22:04  kernel: [  495.018225] ixgbe 0000:05:00.1 ens1f1: detected SFP+: 5

                  Dec 22 17:24:32  kernel: [  642.245720] ixgbe 0000:08:00.0: registered PHC device on ens3f0

                  Dec 22 17:24:32  kernel: [  642.348922] IPv6: ADDRCONF(NETDEV_UP): ens3f0: link is not ready

                  Dec 22 17:24:32  kernel: [  642.413442] ixgbe 0000:08:00.0 ens3f0: detected SFP+: 6

                  Dec 22 17:26:18  kernel: [  748.661562] ixgbe 0000:08:00.1: registered PHC device on ens3f1

                  Dec 22 17:26:18  kernel: [  748.766107] IPv6: ADDRCONF(NETDEV_UP): ens3f1: link is not ready

                  Dec 22 17:26:18  kernel: [  748.830408] ixgbe 0000:08:00.1 ens3f1: detected SFP+: 5

                  Dec 22 22:19:08  kernel: [18319.317635] ixgbe 0000:05:00.1: removed PHC on ens1f1

                  Dec 22 22:19:12  kernel: [18323.695150] ixgbe 0000:05:00.1: registered PHC device on ens1f1

                  Dec 22 22:19:12  kernel: [18323.798353] IPv6: ADDRCONF(NETDEV_UP): ens1f1: link is not ready

                  Dec 22 22:19:12  kernel: [18323.864169] ixgbe 0000:05:00.1 ens1f1: detected SFP+: 5

                  Dec 22 22:20:24  kernel: [18395.349270] ixgbe 0000:05:00.1: removed PHC on ens1f1

                  Dec 22 22:20:28  kernel: [18399.726247] ixgbe 0000:05:00.1: registered PHC device on ens1f1

                  Dec 22 22:20:28  kernel: [18399.830017] IPv6: ADDRCONF(NETDEV_UP): ens1f1: link is not ready

                  Dec 22 22:20:28  kernel: [18399.894342] ixgbe 0000:05:00.1 ens1f1: detected SFP+: 5

                   

                  Can you please assist to resolve this issue?

                   

                  Thanks.

                  • 6. Re: Unable to establish link with Intel 82599ES 10-Gigabit SFI/SFP+ Network Connection
                    U-map

                    ethtool -m ens1f1

                            Identifier                                : 0x03 (SFP)

                            Extended identifier                       : 0x04 (GBIC/SFP defined by 2-wire interface ID)

                            Connector                                 : 0x07 (LC)

                            Transceiver codes                         : 0x10 0x00 0x00 0x00 0x00 0x00 0x00 0x00

                            Transceiver type                          : 10G Ethernet: 10G Base-SR

                            Encoding                                  : 0x06 (64B/66B)

                            BR, Nominal                               : 10300MBd

                            Rate identifier                           : 0x00 (unspecified)

                            Length (SMF,km)                           : 0km

                            Length (SMF)                              : 0m

                            Length (50um)                             : 80m

                            Length (62.5um)                           : 30m

                            Length (Copper)                           : 0m

                            Length (OM3)                              : 300m

                            Laser wavelength                          : 850nm

                            Vendor name                               : FINISAR CORP.

                            Vendor OUI                                : 00:90:65

                            Vendor PN                                 : FTLX8571D3BCL-HP

                            Vendor rev                                : A

                            Option values                             : 0x00 0x1a

                            Option                                    : RX_LOS implemented

                            Option                                    : TX_FAULT implemented

                            Option                                    : TX_DISABLE implemented

                            BR margin, max                            : 0%

                            BR margin, min                            : 0%

                            Vendor SN                                 : MW10G56

                            Date code                                 : 16070928

                            Optical diagnostics support               : Yes

                            Laser bias current                        : 8.268 mA

                            Laser output power                        : 0.6088 mW / -2.16 dBm

                            Receiver signal average optical power     : 0.0008 mW / -30.97 dBm

                            Module temperature                        : 43.75 degrees C / 110.76 degrees F

                            Module voltage                            : 3.3046 V

                            Alarm/warning flags implemented           : Yes

                            Laser bias current high alarm             : Off

                            Laser bias current low alarm              : Off

                            Laser bias current high warning           : Off

                            Laser bias current low warning            : Off

                            Laser output power high alarm             : Off

                            Laser output power low alarm              : Off

                            Laser output power high warning           : Off

                            Laser output power low warning            : Off

                            Module temperature high alarm             : Off

                            Module temperature low alarm              : Off

                            Module temperature high warning           : Off

                            Module temperature low warning            : Off

                            Module voltage high alarm                 : Off

                            Module voltage low alarm                  : Off

                            Module voltage high warning               : Off

                            Module voltage low warning                : Off

                            Laser rx power high alarm                 : Off

                            Laser rx power low alarm                  : On

                            Laser rx power high warning               : Off

                            Laser rx power low warning                : On

                            Laser bias current high alarm threshold   : 13.200 mA

                            Laser bias current low alarm threshold    : 4.000 mA

                            Laser bias current high warning threshold : 12.600 mA

                            Laser bias current low warning threshold  : 5.000 mA

                            Laser output power high alarm threshold   : 1.0000 mW / 0.00 dBm

                            Laser output power low alarm threshold    : 0.2512 mW / -6.00 dBm

                            Laser output power high warning threshold : 0.7943 mW / -1.00 dBm

                            Laser output power low warning threshold  : 0.3162 mW / -5.00 dBm

                            Module temperature high alarm threshold   : 78.00 degrees C / 172.40 degrees F

                            Module temperature low alarm threshold    : -13.00 degrees C / 8.60 degrees F

                            Module temperature high warning threshold : 73.00 degrees C / 163.40 degrees F

                            Module temperature low warning threshold  : -8.00 degrees C / 17.60 degrees F

                            Module voltage high alarm threshold       : 3.7000 V

                            Module voltage low alarm threshold        : 2.9000 V

                            Module voltage high warning threshold     : 3.6000 V

                            Module voltage low warning threshold      : 3.0000 V

                            Laser rx power high alarm threshold       : 1.0000 mW / 0.00 dBm

                            Laser rx power low alarm threshold        : 0.0100 mW / -20.00 dBm

                            Laser rx power high warning threshold     : 0.7943 mW / -1.00 dBm

                            Laser rx power low warning threshold      : 0.0158 mW / -18.01 dBm