1 2 Previous Next 16 Replies Latest reply on Feb 22, 2018 6:42 PM by Intel Corporation Branched to a new discussion.

    i40e x710 4x10G card not working anymore with DAC

    ninwardspiral

      Using an updated driver, I cannot seem to get my interfaces up. Looks like the driver is not allowing the NIC to work due to the SFP+ module type, which worked previously on my Debian Stretch host. Clearly the NIC is detecting and reading the DAC, but I don't understand why it is not working. This is a 40G-breakout cable with 40-to-4x10G SFP+ DAC (TwinAx). I have tried adding the following to work around it, but it still does not work:

       

      root@lab4:~# cat /etc/modprobe.d/i40e.conf

      options i40e allow_unsupported_sfp=1

       

      This was also discussed here, but there was no followup:

      Intel Ethernet Drivers and Utilities / Mailing Lists

       

      Output from dmesg:

      [    4.439190] i40e 0000:08:00.3: PCI-Express: Speed 8.0GT/s Width x8

      [    4.447918] i40e 0000:08:00.3: Features: PF-id[3] VFs: 32 VSIs: 34 QP: 8 RSS FD_ATR FD_SB NTUPLE CloudF DCB VxLAN Geneve NVGRE PTP VEPA

      [    4.453414] i40e 0000:08:00.2: Rx/Tx is disabled on this device because an unsupported SFP+ module type was detected.

      [    4.465288] i40e 0000:08:00.2: Refer to the Intel(R) Ethernet Adapters and Devices User Guide for a list of supported modules.

       

      Other pertinent output:

      root@lab4:~# ethtool -m eth6

      Identifier                                : 0x03 (SFP)

      Extended identifier                       : 0x04 (GBIC/SFP defined by 2-wire interface ID)

      Connector                                 : 0x21 (Copper pigtail)

      Transceiver codes                         : 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00

      Transceiver type                          : FC: Copper Passive

      Encoding                                  : 0x00 (unspecified)

      BR, Nominal                               : 10300MBd

      Rate identifier                           : 0x00 (unspecified)

      Length (SMF,km)                           : 0km

      Length (SMF)                              : 0m

      Length (50um)                             : 0m

      Length (62.5um)                           : 0m

      Length (Copper)                           : 3m

      Length (OM3)                              : 0m

      Passive Cu cmplnce.                       : 0x01 (SFF-8431 appendix E) [SFF-8472 rev10.4 only]

      Vendor name                               : Amphenol

      Vendor OUI                                : 78:a7:14

      Vendor PN                                 : 624400003

      Vendor rev                                : A

      Option values                             : 0x00 0x00

      BR margin, max                            : 0%

      BR margin, min                            : 0%

      Vendor SN                                 : APF13470035GDT

      Date code                                 : 131122

       

      root@lab4:~# ethtool -i eth6

      driver: i40e

      version: 2.4.3

      firmware-version: 6.01 0x800034af 1.1747.0

      expansion-rom-version:

      bus-info: 0000:08:00.0

      supports-statistics: yes

      supports-test: yes

      supports-eeprom-access: yes

      supports-register-dump: yes

      supports-priv-flags: yes

       

      root@lab4:~# ethtool eth6

      Settings for eth6:

      Supported ports: [ ]

      Supported link modes:   Not reported

      Supported pause frame use: Symmetric

      Supports auto-negotiation: No

      Advertised link modes:  Not reported

      Advertised pause frame use: No

      Advertised auto-negotiation: No

      Speed: Unknown!

      Duplex: Unknown! (255)

      Port: Other

      PHYAD: 0

      Transceiver: internal

      Auto-negotiation: off

      Supports Wake-on: d

      Wake-on: d

      Current message level: 0x0000000f (15)

             drv probe link timer

      Link detected: no

       

      root@lab4:~# ip link show eth6

      5: eth6: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000

          link/ether 68:05:ca:2f:7c:b0 brd ff:ff:ff:ff:ff:ff

        • 1. Re: i40e x710 4x10G card not working anymore with DAC
          Intel Corporation
          This message was posted on behalf of Intel Corporation

          Hi Ninwardspiral,

           Thank you for posting in Wired Communities. Can you share what is the exact brand and model of the 40G break out cable used? 

          Thanks,
          Sharon T

          • 2. Re: i40e x710 4x10G card not working anymore with DAC
            ninwardspiral

            Hi Sharon,

             

            This is a generic Amphenol 3 meter breakout 40G QSFP+ to 4x10G SFP+ DAC.

            All of the generic Amphenol straight 10G SFP+ DACs appear to work fine.

            I know these used to work prior to updating the firmware and drivers...

            • 3. Re: i40e x710 4x10G card not working anymore with DAC
              Intel Corporation
              This message was posted on behalf of Intel Corporation

              Hi Ninwardspiral,

                Thank you for the information. I would like to clarify further information in order to better assist you.
              1) Based on your NIC descreption, are you using an X710-DA4? If this is the model, it is recommended to use the cable listed in the website below:
              https://www.intel.com/content/www/us/en/support/articles/000007045/network-and-i-o/ethernet-products.html

              2)The breakout cable are applicable for use on XL710 series (not X710 series), and we recommended to use cable model listed in the link below:
              https://www.intel.com/content/www/us/en/support/articles/000006051/network-and-i-o/ethernet-products.html

                Having said that, can you help clarify :
              1) Are you saying the Amphenol 3 meter break out 40Gb QSFP+to 4x10GSFP+DAC used to work on this X710-DA4 prior you upgraded to firmware 6.01?
              2) Are the generic Amphenol straight 10 SFP+ (which is another cable) you refer here works on the X710-DA4 ?


                Looking forward to your reply.

              Thanks,
              Sharon T
               

              • 4. Re: i40e x710 4x10G card not working anymore with DAC
                ninwardspiral

                Hi Sharon,

                 

                Yes, this is an X710-DA4. Let me clarify the connections made using the breakout cable.

                On the network switch is a 40G QSFP+ port, which has been configured as 4x10G. So the QSFP+ is inserted in the switch. Then the SFP+ connections are in the Intel NIC.

                I am certain the EEPROM of the SFP+ module is correct, and works in other NICs and devices.

                 

                1) Yes, these generic DACs worked fine in FW 5.01, with the 1.4, and 2.0 driver versions. I did try downgrading the driver to 2.0.33 (leaving the FW at 6.01), but the breakout SFP+ still does not work.

                2) Yes, the Amphenol straight SFP+ to SFP+ 10G cable works fine. This module is almost identical to the one used in the breakout, though it has a few more transceiver codes set in the EEPROM as expected.

                3) Just to try some more tests, I tried an identical Dell branded 40G-to-4x10G breakout cable,  because it is just an amphenol cable with a different sticker and part number. The Dell breakout actually works!

                I have provided the EEPROM output from both, so you can see there is very little difference (see below).

                 

                Is there a documented list of all Passive DACs that are whitelisted for use in the X710 cards?

                It would be good to understand why the NIC is blocking the Amphenol, and allowing the Dell.

                 

                Thanks for your help looking at this.

                 

                Cheers,

                Jason

                 

                 

                 

                ## Amphenol (Generic from Cables on demand) SFP+ side of a 40-to-4x10 breakout

                root@lab4:~# ethtool -m eth7

                Identifier                                : 0x03 (SFP)

                Extended identifier                       : 0x04 (GBIC/SFP defined by 2-wire interface ID)

                Connector                                 : 0x21 (Copper pigtail)

                Transceiver codes                         : 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00

                Transceiver type                          : FC: Copper Passive

                Encoding                                  : 0x00 (unspecified)

                BR, Nominal                               : 10300MBd

                Rate identifier                           : 0x00 (unspecified)

                Length (SMF,km)                           : 0km

                Length (SMF)                              : 0m

                Length (50um)                             : 0m

                Length (62.5um)                           : 0m

                Length (Copper)                           : 5m

                Length (OM3)                              : 0m

                Passive Cu cmplnce.                       : 0x01 (SFF-8431 appendix E) [SFF-8472 rev10.4 only]

                Vendor name                               : Amphenol

                Vendor OUI                                : 78:a7:14

                Vendor PN                                 : 624400005

                Vendor rev                                : A

                Option values                             : 0x00 0x00

                BR margin, max                            : 0%

                BR margin, min                            : 0%

                Vendor SN                                 : APF13490056MGJ

                Date code                                 : 131205

                 

                ## Dell (Amphenol) SFP+ side of a 40-to-4x10 breakout

                root@lab4:~# ethtool -m eth8

                Identifier                                : 0x03 (SFP)

                Extended identifier                       : 0x04 (GBIC/SFP defined by 2-wire interface ID)

                Connector                                 : 0x21 (Copper pigtail)

                Transceiver codes                         : 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00

                Transceiver type                          : FC: Copper Passive

                Encoding                                  : 0x00 (unspecified)

                BR, Nominal                               : 10300MBd

                Rate identifier                           : 0x00 (unspecified)

                Length (SMF,km)                           : 0km

                Length (SMF)                              : 0m

                Length (50um)                             : 0m

                Length (62.5um)                           : 0m

                Length (Copper)                           : 5m

                Length (OM3)                              : 0m

                Passive Cu cmplnce.                       : 0x01 (SFF-8431 appendix E) [SFF-8472 rev10.4 only]

                Vendor name                               : Amphenol

                Vendor OUI                                : 78:a7:14

                Vendor PN                                 : 601110005

                Vendor rev                                : F

                Option values                             : 0x00 0x00

                BR margin, max                            : 0%

                BR margin, min                            : 0%

                Vendor SN                                 : APF12190054F1A

                Date code                                 : 120511

                 

                 

                ## Diff showing that the electrical settings of the modules are identical.

                ## It is simply the customized vendor strings are the only difference:

                 

                root@lab4:~# ethtool -m eth7 > amp

                root@lab4:~# ethtool -m eth8 > dell

                root@lab4:~# diff amp dell

                18,19c18,19

                < Vendor PN                                 : 624400005

                < Vendor rev                                : A

                ---

                > Vendor PN                                 : 601110005

                > Vendor rev                                : F

                23,24c23,24

                < Vendor SN                                 : APF13490056MGJ

                < Date code                                 : 131205

                ---

                > Vendor SN                                 : APF12190054F1A

                > Date code                                 : 120511

                • 5. Re: i40e x710 4x10G card not working anymore with DAC
                  Intel Corporation
                  This message was posted on behalf of Intel Corporation

                  Hi Ninwardspiral,

                   

                  Thank you for the clarification and additional information. Let me further investigate on this. By the way, just to double check have you  sent the output of "ethtool -i <ethX or attaching that to e1000.sourceforge.net as mentioned by the Sourceforge support. Thanks.

                   

                  Regards,
                  Sharon T

                  • 6. Re: i40e x710 4x10G card not working anymore with DAC
                    Intel Corporation
                    This message was posted on behalf of Intel Corporation

                    Hi Ninwardspiral,

                       Please refer to below information:

                    1) If the cable stopped working after the firmware update, you might want to update the firmware again in case there is issue during the firmware update process.
                    You can try downgrading back to 5.x FW and use applicable drivers.  Test the cable again to confirm it works using that firmware. Then try update to firmware 6.01 again.  


                    2) Please check if your X710-DA4 is a retail unit. You can check the Vendor and Device ID based on the information here.
                    https://www.intel.com/content/www/us/en/support/articles/000005612/network-and-i-o/ethernet-products.html

                       You should see the  VendorID:8086 and DvID:1572 for the retail version.

                    3) Can you try a different cable length to see if this will make any difference? 

                    4) With regards to the Dell* cable, it is possible there is difference other than labeling between the generic and the Dell* version of the Amphenol which we do not have the information.

                    Please feel free to update me.

                    Thank you.

                    Regards,
                    Sharon T


                     

                    • 7. Re: i40e x710 4x10G card not working anymore with DAC
                      Intel Corporation
                      This message was posted on behalf of Intel Corporation

                      Hi Ninwardspiral,

                       Please feel free to update me.


                      Thanks,
                      Sharon T

                      • 8. Re: i40e x710 4x10G card not working anymore with DAC
                        ninwardspiral

                        Hi Sharon,

                         

                        1) Unfortunately at this point I cannot downgrade the firmware since there were many fixes we needed in the latest FW and driver. We recently had to upgrade to do some 802.1ad validation. So I hope there is a way to debug this and identify specifically what is not allowing the module to be used.

                         

                        2) I have several of these cards, and they are the retail version:

                        root@lab2:~# lspci -nn | grep -i 'X710'

                        08:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)

                        08:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)

                        08:00.2 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)

                        08:00.3 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)

                        09:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)

                        09:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)

                        09:00.2 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)

                        09:00.3 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)

                         

                        3) I have tried 2m, 3M and 5M, with the last one the one I realized matches the Dell DAC.

                         

                        Can you tell me if there is a knob in the driver that can be used to ignore some embedded check of the DAC part number, vendor, etc?

                        Perhaps something that can be passed to the driver via modprobe (something like ignore_whitelist=1). I think the ixgbe driver has an allow_unsupported option...is there anything similar for i40e?

                         

                        4) From a layer 1 point of view, all of the electrically pertinent details of the DAC modules are identical between the vendors (dell and generic). If there is simply a whitelist built into the firmware, or driver, I would be interested in understanding how to add to that list.

                         

                        Thanks,
                        Jason

                        • 9. Re: i40e x710 4x10G card not working anymore with DAC
                          Intel Corporation
                          This message was posted on behalf of Intel Corporation

                          Hi Ninwardspiral,

                           Thank you for the information. I will have to check on your inquiries.

                          Regards,
                          Sharon T

                          • 10. Re: i40e x710 4x10G card not working anymore with DAC
                            Intel Corporation
                            This message was posted on behalf of Intel Corporation

                            Hi Ninwardspiral,

                               There is no whitelist.  Can you the part number of the X710-DA4 you have. This can be found on the white label of the physical network card which is near 2D barcode. The format goes like 15 alphanumeric + 6 alphanumeric + 5 alphanumeric - 3 digits.

                            Thanks.

                            Regards,
                            Sharon T

                             

                            • 11. Re: i40e x710 4x10G card not working anymore with DAC
                              ninwardspiral

                              Hi Sharon,

                               

                              This is the numbers from the card/sticker I can see:

                               

                              6805CA302920008  4414AD  H58361 002

                               

                              Thanks,

                              Jason

                              • 12. Re: i40e x710 4x10G card not working anymore with DAC
                                Intel Corporation
                                This message was posted on behalf of Intel Corporation

                                Hi Ninwardspiral,

                                  Thank you for the information. I will check on this.

                                Regards,
                                Sharon T 

                                • 13. Re: i40e x710 4x10G card not working anymore with DAC
                                  Intel Corporation
                                  This message was posted on behalf of Intel Corporation

                                  Hi Ninwardspiral,

                                       Just to double check, can you try connect using a regular SFP-SFP cable?  Even try back to back connection. Please feel free to update me.

                                  Thanks,
                                  Sharon T 

                                  • 14. Re: i40e x710 4x10G card not working anymore with DAC
                                    Intel Corporation
                                    This message was posted on behalf of Intel Corporation

                                    Hi Ninwardspiral,

                                     Please feel free to update me the result of using regular SFP-SFP cable and back to back connection.

                                    Thanks,
                                    Sharon T
                                     

                                    1 2 Previous Next