1 2 Previous Next 28 Replies Latest reply on Apr 26, 2018 12:35 AM by Intel Corporation Go to original post Branched to a new discussion.
      • 15. Re: i40e x710 4x10G card not working anymore with DAC
        ninwardspiral

        Hi Sharon,

         

        I did validate that an SFP+ to SFP+ of the same brand works fine back to back, and to other devices. The interesting thing here is the SFP+ module is basically the same as the ones used on the 10G SFP+ side of the 40G-4x10G breakout cables. Apparently cable length does not matter because the results are always the same regardless of length.

         

        Thanks,
        Jason

        • 16. Re: i40e x710 4x10G card not working anymore with DAC
          Intel Corporation
          This message was posted on behalf of Intel Corporation

          Hi Ninwardspiral,

          Thank you for the update. I will check if there is any information to share. 

          Regards,
          Sharon T

          • 17. Re: i40e x710 4x10G card not working anymore with DAC
            Intel Corporation
            This message was posted on behalf of Intel Corporation

            Hi Ninwardspiral,
             
             I sent a PM to you, please reply me from there.

            Thanks,
            Sharon T 

            • 18. Re: i40e x710 4x10G card not working anymore with DAC
              ninwardspiral

              Hi Sharon,

               

              Interestingly, we have had a bunch of customer-reported issues similar to this one. Most of them are more like Edgar's issue, where the Intel 10G-SR optics are not working properly with the Cumulus Linux switches.

              At this point, it may be best to open this as a support case with Intel.

               

              I have been trying to downgrade one of my cards to see if the problem is in the new firmware. Sadly, the NVM downgrade reports an error, but appears to have changed it. Perhaps a bug?

               

              root@lab4:~/Intel_NIC/nvm/XL710_downgrade_6.0-to-5.05/Linux_x64# ./nvmupdate64e

               

              Intel(R) Ethernet NVM Update Tool

              NVMUpdate version 1.30.22.1

              Copyright (C) 2013 - 2017 Intel Corporation.

               

              WARNING: To avoid damage to your device, do not stop the update or reboot or power off the system during this update.

              Inventory in progress. Please wait [.-********]

               

              Num Description                               Ver. DevId S:B    Status

              === ======================================== ===== ===== ====== ===============

              01) Intel(R) Ethernet Converged Network       6.01  1572 00:008 Update

                  Adapter X710-4                                              available

              02) Intel(R) Ethernet Converged Network       6.01  1572 00:009 Update <<<<<<<<<<<< Started at 6.01

                  Adapter X710-4                                              available

              03) Intel(R) Ethernet Converged Network       6.01  1584 00:065 Update

                  Adapter XL710-Q1                                            available

              04) Intel(R) 82576 Gigabit Dual Port Network        10C9 00:068 Access error

                  Connection

              05) Intel(R) 82576 Gigabit Dual Port Network        10C9 00:069 Access error

                  Connection

               

              Options: Adapter Index List (comma-separated), [A]ll, e[X]it

              Enter selection:2

              Would you like to back up the NVM images? [Y]es/[N]o: y

              Update in progress. This operation may take several minutes.

              [**-.......]

              Reboot is required to complete the update process.

               

              Tool execution completed with the following status: An error occurred accessing the device

              Press any key to exit.

               

              root@lab4:~/Intel_NIC/nvm/XL710_downgrade_6.0-to-5.05/Linux_x64# ./nvmupdate64e

               

              Intel(R) Ethernet NVM Update Tool

              NVMUpdate version 1.30.22.1

              Copyright (C) 2013 - 2017 Intel Corporation.

               

              WARNING: To avoid damage to your device, do not stop the update or reboot or power off the system during this update.

              Inventory in progress. Please wait [******+...]

               

              Num Description                               Ver. DevId S:B    Status

              === ======================================== ===== ===== ====== ===============

              01) Intel(R) Ethernet Converged Network       6.01  1572 00:008 Update

                  Adapter X710-4                                              available

              02) Intel(R) Ethernet Converged Network       5.05  1572 00:009 Up to date <<<<<<<<<<<<< Appears to have downgraded this, I will reboot to verify!

                  Adapter X710-4

              03) Intel(R) Ethernet Converged Network       6.01  1584 00:065 Update

                  Adapter XL710-Q1                                            available

              04) Intel(R) 82576 Gigabit Dual Port Network        10C9 00:068 Access error

                  Connection

              05) Intel(R) 82576 Gigabit Dual Port Network        10C9 00:069 Access error

                  Connection

               

              Options: Adapter Index List (comma-separated), [A]ll, e[X]it

              Enter selection:x

               

              Tool execution completed with the following status: An error occurred accessing the device

              Press any key to exit.

              root@lab4:~/Intel_NIC/nvm/XL710_downgrade_6.0-to-5.05/Linux_x64# reboot

               

              root@lab4:~# ethtool -i eth10

              driver: i40e

              version: 1.6.42

              firmware-version: 5.05 0x8000289d 1.1568.0 <<<<<<<<<<<<<<<<<< Looks like it worked.

              expansion-rom-version:

              bus-info: 0000:09:00.0

              supports-statistics: yes

              supports-test: yes

              supports-eeprom-access: yes

              supports-register-dump: yes

              supports-priv-flags: yes

               

               

              Now BOTH the generic Amphenol AND the Dell breakout cable are working. Looks like there is a bug in the latest firmware and/or driver.

              Unfortunately the ethtool -m output does not work with this driver...  I suppose I can prove it by showing the LLDP output from these ports and the switch.

              But you can take my word for it that the Dell is in eth10, and the Generic is in eth11.

               

              Here are the ethtool outputs:

               

              Dell Breakout DAC:

              ----------------------

              Settings for eth10:

              Supported ports: [ FIBRE ]      <<<<<<<<<<<<<<< This was not correct in the new FW and version!

              Supported link modes:   10000baseT/Full <<<<<<<<<<<<<<< This was not correct in the new FW and version!

              Supported pause frame use: Symmetric

              Supports auto-negotiation: No

              Advertised link modes:  Not reported

              Advertised pause frame use: No

              Advertised auto-negotiation: No

              Speed: 10000Mb/s

              Duplex: Full

              Port: Direct Attach Copper

              PHYAD: 0

              Transceiver: external

              Auto-negotiation: off

              Supports Wake-on: d

              Wake-on: d

              Current message level: 0x0000000f (15)

                     drv probe link timer

              Link detected: yes

               

              root@lab4:~# ethtool -i eth10

              driver: i40e

              version: 1.6.42

              firmware-version: 5.05 0x8000289d 1.1568.0

              expansion-rom-version:

              bus-info: 0000:09:00.0

              supports-statistics: yes

              supports-test: yes

              supports-eeprom-access: yes

              supports-register-dump: yes

              supports-priv-flags: yes

              root@lab4:~# ethtool -m eth10

              Cannot get module EEPROM information: Operation not supported

               

              Generic Breakout DAC:

              ---------------------------

              root@lab4:~# ethtool eth11

              Settings for eth11:

              Supported ports: [ FIBRE ]        <<<<<<<<<<<<<<< This was not correct in the new FW and version!

              Supported link modes:   10000baseT/Full  <<<<<<<<<<<<<<< This was not correct in the new FW and version!

              Supported pause frame use: Symmetric

              Supports auto-negotiation: No

              Advertised link modes:  Not reported

              Advertised pause frame use: No

              Advertised auto-negotiation: No

              Speed: 10000Mb/s

              Duplex: Full

              Port: Direct Attach Copper

              PHYAD: 0

              Transceiver: external

              Auto-negotiation: off

              Supports Wake-on: d

              Wake-on: d

              Current message level: 0x0000000f (15)

                     drv probe link timer

              Link detected: yes

               

              root@lab4:~# ethtool -m eth11

              Cannot get module EEPROM information: Operation not supported

               

              root@lab4:~# ethtool -i eth11

              driver: i40e

              version: 1.6.42

              firmware-version: 5.05 0x8000289d 1.1568.0

              expansion-rom-version:

              bus-info: 0000:09:00.1

              supports-statistics: yes

              supports-test: yes

              supports-eeprom-access: yes

              supports-register-dump: yes

              supports-priv-flags: yes

              • 19. Re: i40e x710 4x10G card not working anymore with DAC
                Intel Corporation
                This message was posted on behalf of Intel Corporation

                Hi Ninwardspiral,

                 Thank you for the additional information provided. Interesting to know the test you did and Amphenol cable works after downgraded to firmware 5.05. Just to double check one thing here. Are you referring this issue happened with Cumulus Linux switches only ? How about other switch model? 

                     We will further investigate on this and update this thread accordingly. 

                Thanks,
                Sharon T 

                • 20. Re: i40e x710 4x10G card not working anymore with DAC
                  ninwardspiral

                  Hi Sharon,

                   

                  Yes, I have tested with Cumulus Linux on both Broadcom and Mellanox switches, as well as Cisco switches.

                  I have not tested to other hosts because I find connecting back-to-back with Intel NICs always seems to work (I assume that is how you folks test).

                   

                  More information:

                  I decided to try this with the latest driver, while keeping the firmware on version 5.05.

                   

                  root@lab4:~# ethtool -i eth10

                  driver: i40e

                  version: 2.4.6

                  firmware-version: 5.05 0x8000289d 1.1568.0

                  expansion-rom-version:

                  bus-info: 0000:09:00.0

                  supports-statistics: yes

                  supports-test: yes

                  supports-eeprom-access: yes

                  supports-register-dump: yes

                  supports-priv-flags: yes

                   

                  root@lab4:~# ethtool eth10

                  Settings for eth10:

                  Supported ports: [ FIBRE ] 

                  Supported link modes:   10000baseT/Full  <<<< Still not right.... it SHOULD actually list many, including "10000baseCR/Full"

                  Supported pause frame use: Symmetric

                  Supports auto-negotiation: No

                  Advertised link modes:  10000baseT/Full

                  Advertised pause frame use: No

                  Advertised auto-negotiation: No

                  Speed: 10000Mb/s

                  Duplex: Full

                  Port: Direct Attach Copper

                  PHYAD: 0

                  Transceiver: internal

                  Auto-negotiation: off

                  Supports Wake-on: d

                  Wake-on: d

                  Current message level: 0x0000000f (15)

                         drv probe link timer

                  Link detected: yes

                   

                  root@lab4:~# ip link show eth10

                  12: eth10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000

                      link/ether 68:05:ca:2f:7d:c8 brd ff:ff:ff:ff:ff:ff

                   

                  root@lab4:~# ethtool -m eth10

                  Cannot get module EEPROM information: Invalid argument

                  ^^^^^  I suppose this is a firmware thing, rather than a driver thing.

                   

                  root@lab4:~# ethtool -i eth11

                  driver: i40e

                  version: 2.4.6

                  firmware-version: 5.05 0x8000289d 1.1568.0

                  expansion-rom-version:

                  bus-info: 0000:09:00.1

                  supports-statistics: yes

                  supports-test: yes

                  supports-eeprom-access: yes

                  supports-register-dump: yes

                  supports-priv-flags: yes

                   

                  root@lab4:~# ip link show eth11

                  13: eth11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000

                      link/ether 68:05:ca:2f:7d:c9 brd ff:ff:ff:ff:ff:ff

                   

                   

                  Here is the output from the Cumulus Linux on a Mellanox 2740 switch, for the DAC initially reported in this post (Generic Amphenol 4x10G_to_40G breakout):

                   

                  root@cumulus:~# ethtool swp13

                  Settings for swp13:

                  Supported ports: [ TP FIBRE ]

                  Supported link modes:   1000baseT/Full

                                          10000baseT/Full

                                          40000baseCR4/Full

                                          40000baseSR4/Full

                                          40000baseLR4/Full

                                          25000baseCR/Full

                                          50000baseCR2/Full

                                          100000baseSR4/Full

                                          100000baseCR4/Full

                                          100000baseLR4_ER4/Full

                                          10000baseCR/Full

                                          10000baseSR/Full

                                          10000baseLR/Full

                                          10000baseLRM/Full

                                          10000baseER/Full

                  Supported pause frame use: Symmetric

                  Supports auto-negotiation: Yes

                  Supported FEC modes: None BaseR RS

                  Advertised link modes:  1000baseT/Full

                                          10000baseT/Full

                                          40000baseCR4/Full

                                          40000baseSR4/Full

                                          40000baseLR4/Full

                                          25000baseCR/Full

                                          50000baseCR2/Full

                                          100000baseSR4/Full

                                          100000baseCR4/Full

                                          100000baseLR4_ER4/Full

                                          10000baseCR/Full

                                          10000baseSR/Full

                                          10000baseLR/Full

                                          10000baseLRM/Full

                                          10000baseER/Full

                  Advertised pause frame use: Symmetric

                  Advertised auto-negotiation: Yes

                  Advertised FEC modes: Not reported

                  Speed: 10000Mb/s

                  Duplex: Full

                  Port: Other

                  PHYAD: 0

                  Transceiver: internal

                  Auto-negotiation: on

                  Current message level: 0xffffffa1 (-95)

                         drv ifup tx_err tx_queued intr tx_done rx_status pktdata hw wol 0xffff8000

                  Link detected: yes

                   

                  root@cumulus:~# ethtool -m swp13

                  Identifier                                : 0x0d (QSFP+)

                  Extended identifier                       : 0x00

                  Extended identifier description           : 1.5W max. Power consumption

                  Extended identifier description           : No CDR in TX, No CDR in RX

                  Extended identifier description           : High Power Class (> 3.5 W) not enabled

                  Connector                                 : 0x23 (No separable connector)

                  Transceiver codes                         : 0x08 0x00 0x00 0x00 0x00 0x00 0x00 0x00

                  Transceiver type                          : 40G Ethernet: 40G Base-CR4

                  Encoding                                  : 0x00 (unspecified)

                  BR, Nominal                               : 10300Mbps

                  Rate identifier                           : 0x00

                  Length (SMF,km)                           : 0km

                  Length (OM3 50um)                         : 0m

                  Length (OM2 50um)                         : 0m

                  Length (OM1 62.5um)                       : 0m

                  Length (Copper or Active cable)           : 3m

                  Transmitter technology                    : 0xa0 (Copper cable unequalized)

                  Attenuation at 2.5GHz                     : 8db

                  Attenuation at 5.0GHz                     : 12db

                  Attenuation at 7.0GHz                     : 0db

                  Attenuation at 12.9GHz                    : 0db

                  Vendor name                               : Amphenol

                  Vendor OUI                                : 78:a7:14

                  Vendor PN                                 : 624400003

                  Vendor rev                                : A

                  Vendor SN                                 : APF134600356WM

                  Revision Compliance                       : Revision not specified

                  Module temperature                        : 0.00 degrees C / 32.00 degrees F

                  Module voltage                            : 0.0000 V

                   

                  root@cumulus:~# lldpctl swp13

                  -------------------------------------------------------------------------------

                  LLDP neighbors:

                  -------------------------------------------------------------------------------

                  Interface:    swp13, via: LLDP, RID: 28, Time: 0 day, 00:41:02

                    Chassis:

                      ChassisID:    mac b0:83:fe:c0:4e:f9

                      SysName:      lab4.rdu.cumulusnetworks.com

                      SysDescr:     Debian GNU/Linux 9 (stretch) Linux 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u5 (2017-09-19) x86_64

                      MgmtIP:       192.168.121.1

                      MgmtIP:       fe80::5054:ff:fed8:baab

                      Capability:   Bridge, on

                      Capability:   Router, on

                      Capability:   Wlan, off

                      Capability:   Station, off

                    Port:

                      PortID:       mac 68:05:ca:2f:7d:c9

                      PortDescr:    eth11

                      TTL:          120

                      PMD autoneg:  supported: no, enabled: no

                        MAU oper type: 10GigBaseCX4 - X copper over 8 pair 100-Ohm balanced cable

                  -------------------------------------------------------------------------------

                  • 21. Re: i40e x710 4x10G card not working anymore with DAC
                    Intel Corporation
                    This message was posted on behalf of Intel Corporation

                    HI Ninwardspiral,

                     Thank you for taking time to perform additional tests and provide the information.  We are still checking and will update you.

                    Thanks,
                    Sharon T 


                     

                    • 22. Re: i40e x710 4x10G card not working anymore with DAC
                      Intel Corporation
                      This message was posted on behalf of Intel Corporation

                      Hi Ninwardspiral,
                        
                          I sent a PM to you about some request, please check and reply from there. As an additional information, you may refer to page 10 and 15 of below XL710 controller matrix for the changes from firmware 5.05 to  6.01, there are some enhancement such as 
                      SR-IOV (Intel® Ethernet Adaptive Virtual Function) and support of 1GbE SX\LX Optical modules.  

                      https://www.intel.com/content/dam/www/public/us/en/documents/release-notes/xl710-ethernet-controller-feature-matrix.pdf

                         Looking forward to your reply.

                      Thanks,
                      Sharon T 

                        

                      • 23. Re: i40e x710 4x10G card not working anymore with DAC
                        ninwardspiral

                        Hi Sharon,

                         

                        I have looked through the source code for the driver, and this is the function in the file i40e_main.c that is creating the error message found in the logs. As I understand this, the firmware is setting the values for the struct, and the driver is reading them. What I cannot figure out is why the firmware is flagging a 10G SFP+ DAC as 'I40E_AQ_QUALIFIED_MODULE'. Clearly this function is not the one suppressing the module, but it is reporting the problem based on the firmware. Clearly the firmware is determining this module is unqualified for some reason. However, everything I have compared, between the Dell Amphenol breakout DAC, the Generic Amphenol breakout DAC, and the straight Amphenol SFP+ DAC, there is absolutely no functional difference in the eeprom information. Are you sure there is no "whitelist" or "blacklist" in the Firmware?

                         

                        Thanks,
                        Jason

                         

                         

                         

                        /**

                        * i40e_handle_link_event - Handle link event

                        * @pf: board private structure

                        * @e: event info posted on ARQ

                        **/

                        static void i40e_handle_link_event(struct i40e_pf *pf,

                                                           struct i40e_arq_event_info *e)

                        {

                                struct i40e_aqc_get_link_status *status =

                                        (struct i40e_aqc_get_link_status *)&e->desc.params.raw;

                         

                         

                                /* Do a new status request to re-enable LSE reporting

                                 * and load new status information into the hw struct

                                 * This completely ignores any state information

                                 * in the ARQ event info, instead choosing to always

                                 * issue the AQ update link status command.

                                 */

                                i40e_link_event(pf);

                         

                         

                                /* Check if module meets thermal requirements */

                                if (status->phy_type == I40E_PHY_TYPE_NOT_SUPPORTED_HIGH_TEMP) {

                                        dev_err(&pf->pdev->dev,

                                                "Rx/Tx is disabled on this device because the module does not meet thermal requirements.\n");

                                        dev_err(&pf->pdev->dev,

                                                "Refer to the Intel(R) Ethernet Adapters and Devices User Guide for a list of supported modules.\n");

                                } else {

                                        /* check for unqualified module, if link is down, suppress

                                         * the message if link was forced to be down.

                                         */

                                        if ((status->link_info & I40E_AQ_MEDIA_AVAILABLE) &&

                                            (!(status->an_info & I40E_AQ_QUALIFIED_MODULE)) &&

                                            (!(status->link_info & I40E_AQ_LINK_UP)) &&

                                            (!(pf->flags & I40E_FLAG_LINK_DOWN_ON_CLOSE_ENABLED))) {

                                                dev_err(&pf->pdev->dev,

                                                        "Rx/Tx is disabled on this device because an unsupported SFP+ module type was detected.\n");

                                                dev_err(&pf->pdev->dev,

                                                        "Refer to the Intel(R) Ethernet Adapters and Devices User Guide for a list of supported modules.\n");

                                        }

                                }

                        }

                        • 24. Re: i40e x710 4x10G card not working anymore with DAC
                          Intel Corporation
                          This message was posted on behalf of Intel Corporation

                          Hi Ninwardspiral,

                            Thank you for sharing the information, I will further check with our engineering team.

                          Regards,
                          Sharon T 

                          • 25. Re: i40e x710 4x10G card not working anymore with DAC
                            Intel Corporation
                            This message was posted on behalf of Intel Corporation

                            Hi Swedebay,

                              Thank you for posting in Wired Communities. Can you share what is the exact network adapter model? (e.g. XL710-QDA1 or XL710-QDA2?) and what is the fiber module model attached to the network adapter? 

                              Looking forward to your reply. Thanks.

                            Regards,
                            Sharon T 
                             

                            • 26. Re: i40e x710 4x10G card not working anymore with DAC
                              Intel Corporation
                              This message was posted on behalf of Intel Corporation

                              Hi Ninwardspiral,

                               To provide an update, there is no "whitelist" or "blacklist" in the firmware. Thank you. 

                              Regards,
                              Sharon T

                              • 27. Re: i40e x710 4x10G card not working anymore with DAC
                                Intel Corporation
                                This message was posted on behalf of Intel Corporation

                                Hi Ninwardspiral,

                                 I sent a PM to you, please feel free to reply me from there.

                                Thanks,
                                Sharon T 

                                • 28. Re: i40e x710 4x10G card not working anymore with DAC
                                  Intel Corporation
                                  This message was posted on behalf of Intel Corporation

                                  Hi Ninwardspiral,

                                  I sent a PM to you about some request, please feel free to update me. 

                                  Thanks,
                                  Sharon T

                                  1 2 Previous Next