1 2 Previous Next 15 Replies Latest reply on Jun 7, 2018 7:36 AM by ninwardspiral Branched from an earlier discussion.

    x710 4x10G card not working Amphenol cable

    cedgar

      Hi Sharon and Ninwardspiral,

       

      We are having similar problems with the exact same NIC, and by coincidence with the same breakout cable (although we tested many).

       

      Driver and firmware are updated to the latest versions available (2.4.3 and 6.01 respectively):

      version: 2.4.3

      firmware-version: 6.01 0x800034af 1.1853.0

       

      First cable that does not work: ( QSFP to 4x10G SFP+, amphenol, 2m)


      From the manufacturer website: Each QSFP-SFP+ splitter cable features a single QSFP+ connector (SFF-8436) rated for 40-Gbps on one end and (4) SFP+ connectors (SFF-8431), each rated for 10-Gbps, on the other.

       

      SF-QSFP4SFPPS-002 Amphenol Commercial Products | Cable Assemblies | DigiKey (exact cable we bought)

       

      ethtool -m output:

      Identifier : 0x03 (SFP)

      Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)

      Connector : 0x21 (Copper pigtail)

      Transceiver codes : 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00

      Transceiver type : FC: Copper Passive

      Encoding : 0x00 (unspecified)

      BR, Nominal : 10300MBd

      Rate identifier : 0x00 (unspecified)

      Length (SMF,km) : 0km

      Length (SMF) : 0m

      Length (50um) : 0m

      Length (62.5um) : 0m

      Length (Copper) : 2m

      Length (OM3) : 0m

      Passive Cu cmplnce. : 0x01 (SFF-8431 appendix E) [SFF-8472 rev10.4 only]

      Vendor name : Amphenol

      Vendor OUI : 78:a7:14

      Vendor PN : 624400002

      Vendor rev : A

      Option values : 0x00 0x00

      BR margin, max : 0%

      BR margin, min : 0%

      Vendor SN : APF1718002044N

      Date code : 170513

       

      ethtool <intf> output (after bringing interfaces up):

      Supported ports: [ ]

      Supported link modes: Not reported

      Supported pause frame use: Symmetric

      Supports auto-negotiation: No

      Advertised link modes: Not reported

      Advertised pause frame use: No

      Advertised auto-negotiation: No

      Speed: Unknown!

      Duplex: Unknown! (255)

      Port: Other

      PHYAD: 0

      Transceiver: internal

      Auto-negotiation: off

      Supports Wake-on: d

      Wake-on: d

      Current message level: 0x0000000f (15)

      drv probe link timer

      Link detected: no

       

      Second cable that does not work: ( QSFP to 4x10G SFP+, generic brand):


      2m(6.6ft) 40G QSFP+ to 4SFP+ Passive Breakout DAC Cable | FS.COM

       

      ethtool -m <intf> output:

      Identifier : 0x03 (SFP)

      Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)

      Connector : 0x21 (Copper pigtail)

      Transceiver codes : 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00

      Transceiver type : FC: Copper Passive

      Encoding : 0x00 (unspecified)

      BR, Nominal : 10300MBd

      Rate identifier : 0x00 (unspecified)

      Length (SMF,km) : 0km

      Length (SMF) : 0m

      Length (50um) : 0m

      Length (62.5um) : 0m

      Length (Copper) : 2m

      Length (OM3) : 0m

      Passive Cu cmplnce. : 0x03 (unknown) [SFF-8472 rev10.4 only]

      Vendor name : FIBERSTORE

      Vendor OUI : 00:00:00

      Vendor PN : QSFP-4SFP10G-DAC

      Vendor rev :

       

      Third Cable, it does work but its not what we need:

       

      https://www.brack.ch/delock-direct-attach-kabel-330520

       

      As a way to see if the NICs work, we also bought a SFP+ to SFP+ cable that is supposed to support SFP MSA and SFF-8431, but not SFF-8472. What is impressive is that this cable is detected and allows us to bring the link up

      and even exchange some data at 10gbps. However, that is not what we need.

       

      Encoding : 0x00 (unspecified)

      BR, Nominal : 10300MBd

      Rate identifier : 0x00 (unspecified)

      Length (SMF,km) : 0km

      Length (SMF) : 0m

      Length (50um) : 0m

      Length (62.5um) : 0m

      Length (Copper) : 1m

      Length (OM3) : 0m

      Active Cu cmplnce. : 0x0c (unknown) [SFF-8472 rev10.4 only]

      Vendor name : CISCO-TYCO

      Vendor OUI : 00:40:20

      Vendor PN : 2127493-1

      Vendor rev : A

      Option values : 0x00 0x12

      Option : RX_LOS implemented

      Option : TX_DISABLE implemented

      BR margin, max : 0%

      BR margin, min : 0%

      Vendor SN : 164800277_______

      Date code

amphenol

       

      Settings for ens787f3:

      Supported ports: [ FIBRE ]

      Supported link modes: Not reported

      Supported pause frame use: Symmetric

      Supports auto-negotiation: No

      Advertised link modes: Not reported

      Advertised pause frame use: No

      Advertised auto-negotiation: No

      Speed: 10000Mb/s

      Duplex: Full

      Port: Direct Attach Copper

      PHYAD: 0

      Transceiver: internal

      Auto-negotiation: off

      Supports Wake-on: d

      Wake-on: d

      Current message level: 0x0000000f (15)

      drv probe link timer

      Link detected: yes

       

      Fourth Cable (to be delivered):

      Intel X4DACBL3 40G QSFP+ Breakout DAC Cable | FS.COM

       

      We are waiting for a fourth cable (this time is supposed to be intel compatible, but we dont know if it will be compatible with our switch...).

       

      Why is the amphenol cable not working if the output of ethernet says: (SFF-8431 appendix E) [SFF-8472 rev10.4 only] which as stated in : Compatible SFP+ Modules, SFP Modules, and Cables for Intel® Ethernet...  should be enough to be compatible. Why

      only SFP+ to SFP+ cables work? What should we do? We spent to many resources and time buying different cables.

       

      And to finish, would this NIC Intel® Ethernet Network Adapter XXV710-DA2 Product Specifications  give us better results?


      Thanks.

      Edgar

        • 1. Re: x710 4x10G card not working Amphenol cable
          Intel Corporation
          This message was posted on behalf of Intel Corporation

          Hi Cedgar,

           Thank you for posting in Wired Communities. I am sorry to hear what happened. 
          Please refer to information below hopefully can clarify the concern:

          1) Just want to confirm if the NIC model is a X710-DA4?

          2) The first two cables you tried are breakout cables, which means one end is a single connection and the other end are with 4 connections. This kind of cable is applicable for XL710 series (XL710-QDA1 or XL710-QDA2 NIC) which has a single 40GB port or dual port 40GB per port. e.g by connecting the breakout cable to a XL710-QDA1, the single 40GB port is subdivided to 10Gbps per port on each port, which means 40Gb is divided to 4 x 10 Gbps. 

             Please refer to our QSFP+ breakout cable product brief, as you can see breakout cable is for the XL710 series NIC
          https://www.intel.com/content/www/us/en/ethernet-products/optics-cables/ethernet-qsfp-optics-brief.html?_ga=2.34173224.84908822.1519284483-1145062571.1519008255

              The third cable you tested works on the X710-DA4 since it is not a breakout cable.

          3) The cable supported by X710-DA4 is SFP+ DA twin Axial cable up to 10m
          https://ark.intel.com/products/83965/Intel-Ethernet-Converged-Network-Adapter-X710-DA4

          4) One of the difference of X710 series and XXV710-DA2 lies in the speed supported. XXV710-DA2 can support up to 25Gbps. This does not determine this will work or work better with the cables you mentioned.

          Thanks,
          Sharon T 


           

          • 2. Re: x710 4x10G card not working Amphenol cable
            cedgar

            Hi Sharon T,

             

            1) Yes we have 2 x710-DA4 that we are trying to connect to switches with QSFP+ ports.

             

            2) Why is it only suitable for XL710 series? I connect the QSFP side of the cable to the switch and the 4x10G connections to the NIC, what is the problem on doing that?  In this thread: i40e x710 4x10G card not working anymore with DAC

            which i was referring to in my first message (I actually posted that there), Ninwardspiral is asking you the same exact thing, we have the same problem. And at some point he says that he found a cable that is exactly the same than the amphenol, but its labeled as a Dell cable and it works.

            3)  So you are telling me that QSFP+ to 4xSFP+ does not work? I am connecting the SFP+ side of the cable to the NIC....

             

            4)  I know that that NIC is a 2x25G port NIC. So if it does not work there, how can we even use BREAKOUT cables?

             

            Edgar

            • 3. Re: x710 4x10G card not working Amphenol cable
              ninwardspiral

              Hi Edgar,

               

              1) Yes we have 2 x710-DA4 that we are trying to connect to switches with QSFP+ ports.

              ...

              4)  I know that that NIC is a 2x25G port NIC. So if it does not work there, how can we even use BREAKOUT cables?

               

              I thought the x710-4DA was only a 4x10G NIC... Can you provide the 'ethtool -m' output from the host? What type of switch is it?

               

              Regardless, one thing to consider with a 25G port is the pluggable has to be an SFP28. 

              If you are connecting the SFP+ to it, make sure auto-negotiation is on for both ends of the link to establish the link in 10G. I have seen native 25G ports not come up with common SFP+ modules. It all depends on vendor.

               

              Cheers,
              Jason

              • 4. Re: x710 4x10G card not working Amphenol cable
                cedgar

                Hi Jason,

                 

                First I would like to to clarify something. When I said 2 x710-DA4 I meant that we have two NICs in our set up, and of course they are 4x10G as you thought. -

                 

                I provided the 'ethtool -m' output in my first post. But here it goes again:

                 

                Identifier                                : 0x03 (SFP)

                Extended identifier                       : 0x04 (GBIC/SFP defined by 2-wire interface ID)

                Connector                                 : 0x21 (Copper pigtail)

                Transceiver codes                         : 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00

                Transceiver type                          : FC: Copper Passive

                Encoding                                  : 0x00 (unspecified)

                BR, Nominal                               : 10300MBd

                Rate identifier                           : 0x00 (unspecified)

                Length (SMF,km)                           : 0km

                Length (SMF)                              : 0m

                Length (50um)                             : 0m

                Length (62.5um)                           : 0m

                Length (Copper)                           : 2m

                Length (OM3)                              : 0m

                Passive Cu cmplnce.                       : 0x01 (SFF-8431 appendix E) [SFF-8472 rev10.4 only]

                Vendor name                               : Amphenol

                Vendor OUI                                : 78:a7:14

                Vendor PN                                 : 624400002

                Vendor rev                                : A

                 

                 

                We have a switch that is a bit special (barefoot tofino). It supports QSFP+ and QSFP28, in this case my module is QSFP+. If we don't find a solution with the current NICs, we will return them and go for the 25G and QSFP28 to 4xSFP28, so I guess that should work right?

                 

                 

                PS: Did you ever managed to make a QSFP+ to 4xSFP+ breakout cable work with this NIC? As far as I understood you say that before upgrading drivers/firmware your amphenol cable was working, right?

                 

                Kind regards,
                Edgar

                • 5. Re: x710 4x10G card not working Amphenol cable
                  ninwardspiral

                  Hi Edgar,

                   

                  I have tested the Mellanox 100G switch ports, using the Dell and Generic Amphenol cables, to the Intel X710 SFP+ port. The results are the same as the native 40G switch reported originally in the other thread. There is no reason these should not BOTH work, as they are essentially the same cable.

                   

                  Before we got the Dell cables, we only had the generic amphenol DACs, and these certainly worked with the older firmware and the 1.4 driver. I can try downgrading the firmware on one card...not entirely sure how to do that.

                   

                  I would also recommend staying away from FS.com cables, as we have noticed plenty of issues with those in customer networks.

                   

                  Dell Breakout DAC (WORKING):

                  root@lab4:~# ethtool eth10

                  Settings for eth10:

                  Supported ports: [ FIBRE ]

                  Supported link modes:   10000baseT/Full

                  Supported pause frame use: Symmetric

                  Supports auto-negotiation: No

                  Advertised link modes:  10000baseT/Full

                  Advertised pause frame use: No

                  Advertised auto-negotiation: No

                  Speed: 10000Mb/s

                  Duplex: Full

                  Port: Direct Attach Copper

                  PHYAD: 0

                  Transceiver: internal

                  Auto-negotiation: off

                  Supports Wake-on: d

                  Wake-on: d

                  Current message level: 0x0000000f (15)

                         drv probe link timer

                  Link detected: yes

                   

                  root@lab4:~# ethtool -m eth10

                  Identifier                                : 0x03 (SFP)

                  Extended identifier                       : 0x04 (GBIC/SFP defined by 2-wire interface ID)

                  Connector                                 : 0x21 (Copper pigtail)

                  Transceiver codes                         : 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00

                  Transceiver type                          : FC: Copper Passive

                  Encoding                                  : 0x00 (unspecified)

                  BR, Nominal                               : 10300MBd

                  Rate identifier                           : 0x00 (unspecified)

                  Length (SMF,km)                           : 0km

                  Length (SMF)                              : 0m

                  Length (50um)                             : 0m

                  Length (62.5um)                           : 0m

                  Length (Copper)                           : 5m

                  Length (OM3)                              : 0m

                  Passive Cu cmplnce.                       : 0x01 (SFF-8431 appendix E) [SFF-8472 rev10.4 only]

                  Vendor name                               : Amphenol

                  Vendor OUI                                : 78:a7:14

                  Vendor PN                                 : 601110005

                  Vendor rev                                : F

                  Option values                             : 0x00 0x00

                  BR margin, max                            : 0%

                  BR margin, min                            : 0%

                  Vendor SN                                 : APF12190054F1A

                  Date code                                 : 120511

                   

                  Amphenol DAC (NOT WORKING):

                  root@lab4:~# ethtool eth10

                  Settings for eth10:

                  Supported ports: [ ]

                  Supported link modes:   Not reported

                  Supported pause frame use: Symmetric

                  Supports auto-negotiation: No

                  Advertised link modes:  Not reported

                  Advertised pause frame use: No

                  Advertised auto-negotiation: No

                  Speed: Unknown!

                  Duplex: Unknown! (255)

                  Port: Other

                  PHYAD: 0

                  Transceiver: internal

                  Auto-negotiation: off

                  Supports Wake-on: d

                  Wake-on: d

                  Current message level: 0x0000000f (15)

                         drv probe link timer

                  Link detected: no

                   

                  root@lab4:~# ethtool -m eth10

                  Identifier                                : 0x03 (SFP)

                  Extended identifier                       : 0x04 (GBIC/SFP defined by 2-wire interface ID)

                  Connector                                 : 0x21 (Copper pigtail)

                  Transceiver codes                         : 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00

                  Transceiver type                          : FC: Copper Passive

                  Encoding                                  : 0x00 (unspecified)

                  BR, Nominal                               : 10300MBd

                  Rate identifier                           : 0x00 (unspecified)

                  Length (SMF,km)                           : 0km

                  Length (SMF)                              : 0m

                  Length (50um)                             : 0m

                  Length (62.5um)                           : 0m

                  Length (Copper)                           : 3m

                  Length (OM3)                              : 0m

                  Passive Cu cmplnce.                       : 0x01 (SFF-8431 appendix E) [SFF-8472 rev10.4 only]

                  Vendor name                               : Amphenol

                  Vendor OUI                                : 78:a7:14

                  Vendor PN                                 : 624400003

                  Vendor rev                                : A

                  Option values                             : 0x00 0x00

                  BR margin, max                            : 0%

                  BR margin, min                            : 0%

                  Vendor SN                                 : APF134600356WM

                  Date code                                 : 131122

                  • 6. Re: x710 4x10G card not working Amphenol cable
                    cedgar

                    Thanks for the answer Jason,

                     

                     

                    Next week we will receive a new breakout cable, if that one does not work either, we will change the NICs. I feel like we are playing the lottery...

                     

                    Do you recommend any good multi-port NIC? It can be SFP+ or SFP28

                     

                    Kind regards,
                    Edgar

                    • 7. Re: x710 4x10G card not working Amphenol cable
                      ninwardspiral

                      Hi Edgar,

                       

                      Just to answer your question about what other NIC I would recommend, I thought I would try this Generic Amphenol breakout DAC on a Mellanox Connect-X4 NIC, and it works perfectly! You can see in the output below, that the serial number in the EEPROM output is the same module I previously had in the Intel x710 port, which is depicted under "Amphenol DAC (NOT WORKING)".

                      So I guess if you want to swap NICs, this is an alternative, and it is a 10G/25G capable NIC.

                       

                      Cheers,

                      Jason

                       

                      root@lab1:~# lspci -s 01:00

                      01:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]

                      01:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]

                       

                      root@lab1:~# ethtool eth7

                      Settings for eth7:

                      Supported ports: [ Backplane ]

                      Supported link modes:   1000baseKX/Full

                                              10000baseKR/Full

                                              25000baseCR/Full

                                              25000baseKR/Full

                                              25000baseSR/Full

                      Supported pause frame use: Symmetric

                      Supports auto-negotiation: Yes

                      Advertised link modes:  1000baseKX/Full

                                              10000baseKR/Full

                                              25000baseCR/Full

                                              25000baseKR/Full

                                              25000baseSR/Full

                      Advertised pause frame use: Symmetric

                      Advertised auto-negotiation: Yes

                      Link partner advertised link modes:  Not reported

                      Link partner advertised pause frame use: No

                      Link partner advertised auto-negotiation: Yes

                      Speed: 10000Mb/s

                      Duplex: Full

                      Port: Direct Attach Copper

                      PHYAD: 0

                      Transceiver: internal

                      Auto-negotiation: on

                      Supports Wake-on: d

                      Wake-on: d

                      Current message level: 0x00000004 (4)

                             link

                      Link detected: yes

                       

                      root@lab1:~# ethtool -m eth7

                      Identifier                                : 0x03 (SFP)

                      Extended identifier                       : 0x04 (GBIC/SFP defined by 2-wire interface ID)

                      Connector                                 : 0x21 (Copper pigtail)

                      Transceiver codes                         : 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00

                      Transceiver type                          : FC: Copper Passive

                      Encoding                                  : 0x00 (unspecified)

                      BR, Nominal                               : 10300MBd

                      Rate identifier                           : 0x00 (unspecified)

                      Length (SMF,km)                           : 0km

                      Length (SMF)                              : 0m

                      Length (50um)                             : 0m

                      Length (62.5um)                           : 0m

                      Length (Copper)                           : 3m

                      Length (OM3)                              : 0m

                      Passive Cu cmplnce.                       : 0x01 (SFF-8431 appendix E) [SFF-8472 rev10.4 only]

                      Vendor name                               : Amphenol

                      Vendor OUI                                : 78:a7:14

                      Vendor PN                                 : 624400003

                      Vendor rev                                : A

                      Option values                             : 0x00 0x00

                      BR margin, max                            : 0%

                      BR margin, min                            : 0%

                      Vendor SN                                 : APF134600356WM

                      Date code                                 : 131122

                      Optical diagnostics support               : No

                      • 8. Re: x710 4x10G card not working Amphenol cable
                        cedgar

                        Thanks Jason.

                         

                        Today I got an "intel compatible" cable and it does not work... I will return the NICs and get some 25G NIC.

                         

                        Since the intel cable did not work either, I was thinking that it might be the switches fault, but I do not know how to check which side of the link is the problematic one. Do you know if there is a way other than using ethtool <intf> which only
                        reports something different when both sides of the connection establish a link (even if its down). Is there a way I could debug that? Note that we dont have any device with QSFP ports apart from this one switch i told you before.

                        • 9. Re: x710 4x10G card not working Amphenol cable
                          ninwardspiral

                          Hi Cedgar,

                           

                          If you still have the X710 cards, try downgrading the firmware to 5.05 using the NVM downgrade utility. I bet things will start working.

                          • 10. Re: x710 4x10G card not working Amphenol cable
                            cedgar

                            That would have been good to know, I returned the cables 1 day before you posted that... a bit unlucky..

                             

                            I made the last cable we ordered work, by disabling auto-negociation in the switch, but only with a generic intel compatible breakout, the amphenol never worked .. ( https://www.fs.com/products/36276.html )


                            By the way, dont you lose features by downgrading the firmware?

                             

                            Thanks for eveything!

                            • 11. Re: x710 4x10G card not working Amphenol cable
                              ninwardspiral

                              Hi Cedgar,

                               

                              Yes, there are a lot of features lost if you downgrade, making it not optimal. It is a good way to prove that there is something broken between 5.05 and 6.01.

                               

                              We noticed a similar problem while testing with Intel optics, which are supposed to work. We found there MUST be a problem in the auto-negotiation code, because as soon as we executed 'ethtool -s eth7 advertise 0x80000000000', the links came up. So clearly constraining the advertised link modes to the highest, removes any decision from the auto-negotiation code. I also disabled auto-neg on the switch (Broadcom Trident2+) side of the link, however I am not sure this is necessary once I pinned the host (X710) to only try 10G-SR/Full.

                               

                              This specific solution is not working with the DACs, but perhaps there is an ethtool option that can force the interface to recognise the 10G side of a breakout DAC. The funny thing is, a 10G to 10G Amphenol DAC cable (essentially using the same electrical module) works fine without changing anything on the switch for Layer1. Furthermore, I do not see any difference in the information in the eeprom. In fact, the EEPROM output is identical.... See supporting output below...

                               

                              Regardless, I am quite certain the root cause of the problems in the 6.x firmware, is most likely related to:

                              1) auto-negotiation

                              2) module detection is not recognising the modules and setting the correct parameters

                              3) whitelist in firmware (we know this is true for optical modules) and explains this output in dmesg:

                              [427334.492215] i40e 0000:08:00.2: Rx/Tx is disabled on this device because an unsupported SFP+ module type was detected.

                              [427334.504168] i40e 0000:08:00.2: Refer to the Intel(R) Ethernet Adapters and Devices User Guide for a list of supported modules.

                              [427675.493990] i40e 0000:08:00.2 eth8: Autoneg not supported on this phy

                               

                              Clearly Intel has not done a lot of interop testing here. These problems are really easy to detect, so I hope Sharon opens a bug to get this firmware fixed.

                               

                              Cheers,
                              Jason

                               

                              Amphenol Breakout DAC 10G SFP+ side

                              ----------------------------------------------------------

                              root@lab4:~# ethtool eth8

                              Settings for eth8:

                              Supported ports: [ ]

                              Supported link modes:   Not reported

                              Supported pause frame use: Symmetric

                              Supports auto-negotiation: No

                              Advertised link modes:  Not reported

                              Advertised pause frame use: No

                              Advertised auto-negotiation: No

                              Speed: Unknown!

                              Duplex: Unknown! (255)

                              Port: Other

                              PHYAD: 0

                              Transceiver: internal

                              Auto-negotiation: off

                              Supports Wake-on: d

                              Wake-on: d

                              Current message level: 0x0000000f (15)

                                     drv probe link timer

                              Link detected: no

                               

                              root@lab4:~# ethtool -m eth8

                              Identifier                                : 0x03 (SFP)

                              Extended identifier                       : 0x04 (GBIC/SFP defined by 2-wire interface ID)

                              Connector                                 : 0x21 (Copper pigtail)

                              Transceiver codes                         : 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00

                              Transceiver type                          : FC: Copper Passive

                              Encoding                                  : 0x00 (unspecified)

                              BR, Nominal                               : 10300MBd

                              Rate identifier                           : 0x00 (unspecified)

                              Length (SMF,km)                           : 0km

                              Length (SMF)                              : 0m

                              Length (50um)                             : 0m

                              Length (62.5um)                           : 0m

                              Length (Copper)                           : 3m

                              Length (OM3)                              : 0m

                              Passive Cu cmplnce.                       : 0x01 (SFF-8431 appendix E) [SFF-8472 rev10.4 only]

                              Vendor name                               : Amphenol

                              Vendor OUI                                : 78:a7:14

                              Vendor PN                                 : 624400003

                              Vendor rev                                : A

                              Option values                             : 0x00 0x00

                              BR margin, max                            : 0%

                              BR margin, min                            : 0%

                              Vendor SN                                 : APF134600356WM

                              Date code                                 : 131122

                               

                               

                              Amphenol 10G SFP+ DAC

                              ----------------------------------------------------------

                              root@lab4:~# ethtool eth9

                              Settings for eth9:

                              Supported ports: [ FIBRE ]

                              Supported link modes:   10000baseT/Full

                              Supported pause frame use: Symmetric

                              Supports auto-negotiation: No

                              Advertised link modes:  10000baseT/Full

                              Advertised pause frame use: No

                              Advertised auto-negotiation: No

                              Speed: 10000Mb/s

                              Duplex: Full

                              Port: Direct Attach Copper

                              PHYAD: 0

                              Transceiver: internal

                              Auto-negotiation: off

                              Supports Wake-on: d

                              Wake-on: d

                              Current message level: 0x0000000f (15)

                                     drv probe link timer

                              Link detected: yes

                               

                              root@lab4:~# ethtool -m eth9

                              Identifier                                : 0x03 (SFP)

                              Extended identifier                       : 0x04 (GBIC/SFP defined by 2-wire interface ID)

                              Connector                                 : 0x21 (Copper pigtail)

                              Transceiver codes                         : 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00

                              Transceiver type                          : FC: Copper Passive

                              Encoding                                  : 0x00 (unspecified)

                              BR, Nominal                               : 10300MBd

                              Rate identifier                           : 0x00 (unspecified)

                              Length (SMF,km)                           : 0km

                              Length (SMF)                              : 0m

                              Length (50um)                             : 0m

                              Length (62.5um)                           : 0m

                              Length (Copper)                           : 2m

                              Length (OM3)                              : 0m

                              Passive Cu cmplnce.                       : 0x01 (SFF-8431 appendix E) [SFF-8472 rev10.4 only]

                              Vendor name                               : Amphenol

                              Vendor OUI                                : 41:50:48

                              Vendor PN                                 : 599700003

                              Vendor rev                                : C

                              Option values                             : 0x00 0x00

                              BR margin, max                            : 0%

                              BR margin, min                            : 0%

                              Vendor SN                                 : APF12280036UDG

                              Date code                                 : 120801

                               

                               

                              ethtool -e eth8 > eth8_eeprom

                              ethtool -e eth9 > eth9_eeprom

                              root@lab4:~# diff eth8_eeprom eth9_eeprom

                              root@lab4:~#

                               

                              The first page of the EEPROM is the critical one. These may have different part numbers, but the electrical settings are the same. This is why I suspect there may be a whitelist in the firmware... Not cool.

                               

                              root@lab4:~# more eth8_eeprom

                              Offset Values

                              ------ ------

                              0x0000: 49 02 04 00 05 00 28 80 24 80 2e 80 ff 7f 80 7d

                              0x0010: b6 0b 2a 80 04 01 50 81 ff ff 50 01 2c 80 20 80

                              0x0020: 00 04 ff ff ff ff ff ff ff ff fa fa 3b 06 63 06

                              0x0030: 01 60 ff ff ff ff ff ff ff ff ff ff ff ff ff ff

                              0x0040: ff ff ff ff ff ff ff ff ff ff ff ff ff 7f ff ff

                              0x0050: 0a 06 36 60 39 00 00 40 ff ff af 34 00 80 0a 04

                              0x0060: 41 06 52 06 42 10 0b 40 2d 15 00 80 ff ff af 0b

                              0x0070: b9 0c ff 7f fc 01 d3 62 c3 0e 22 80 00 7e 25 97

                              0x0080: 72 82 22 01 ff ff 10 00 94 83 20 00 26 80 02 00

                              0x0090: a8 67 ff 7f 05 04 ff ff 00 01 b4 83 ff 7f ff ff

                              0x00a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

                              0x00b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

                              0x00c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

                              0x00d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

                              0x00e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

                              0x00f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

                               

                               

                              root@lab4:~# more eth9_eeprom

                              Offset Values

                              ------ ------

                              0x0000: 49 02 04 00 05 00 28 80 24 80 2e 80 ff 7f 80 7d

                              0x0010: b6 0b 2a 80 04 01 50 81 ff ff 50 01 2c 80 20 80

                              0x0020: 00 04 ff ff ff ff ff ff ff ff fa fa 3b 06 63 06

                              0x0030: 01 60 ff ff ff ff ff ff ff ff ff ff ff ff ff ff

                              0x0040: ff ff ff ff ff ff ff ff ff ff ff ff ff 7f ff ff

                              0x0050: 0a 06 36 60 39 00 00 40 ff ff af 34 00 80 0a 04

                              0x0060: 41 06 52 06 42 10 0b 40 2d 15 00 80 ff ff af 0b

                              0x0070: b9 0c ff 7f fc 01 d3 62 c3 0e 22 80 00 7e 25 97

                              0x0080: 72 82 22 01 ff ff 10 00 94 83 20 00 26 80 02 00

                              0x0090: a8 67 ff 7f 05 04 ff ff 00 01 b4 83 ff 7f ff ff

                              0x00a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

                              0x00b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

                              0x00c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

                              0x00d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

                              0x00e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

                              0x00f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

                              • 12. Re: x710 4x10G card not working Amphenol cable
                                VladimirPerth

                                Hi Everyone,

                                 

                                After reading the following post, I thought I need to share my experience with the technology.

                                 

                                We have 3 x Lenovo HX nodes with Intel X722 network card.

                                Using a Mellanox SX1012x switch to interconnect.

                                We have tried using 2 types of cables now and waiting for a 3rd type.

                                We have tried using QSFP to SFP+ DAC and SFP+ to SFP+ DAC with QSFP converter. None of which can make the ports on the NIC come up.

                                The cables are all from Mellanox.

                                 

                                We have spoken to all 3 suppliers. Lenovo, Mellanox and Nutanix and none of them are providing any solutions. The only thing we are told is that the Switch and Nic might be incompatible, but no one knows to why ????

                                 

                                The only way I can get the ports to come up on the NIC or Mellanox switch is to use the SFP+ to SFP+ cable and loop the ports together. Magically they come up and identify the transceivers and the cable as they should.

                                We have tried turning off or on the Autoneg on both ends. We have forced the QSFP side to 10GB on the switch.

                                 

                                We are now waiting for a breakout cable to arrive as our third option. QSFP to 4 x SFP+ from Mellanox.

                                 

                                Will keep you posted.

                                1 of 1 people found this helpful
                                • 13. Re: x710 4x10G card not working Amphenol cable
                                  Intel Corporation
                                  This message was posted on behalf of Intel Corporation

                                  Hi Cedgar,

                                        With regards to the firmware changes, you may refer to page 10 and 15 of below XL710 controller matrix for the changes from firmware 5.05 to  6.01, there are some enhancement such as SR-IOV (Intel® Ethernet Adaptive Virtual Function) and support of 1GbE SX\LX Optical modules.  

                                  https://www.intel.com/content/dam/www/public/us/en/documents/release-notes/xl710-ethernet-controller-feature-matrix.pdf

                                  Thanks,
                                  Sharon T 

                                  • 14. Re: x710 4x10G card not working Amphenol cable
                                    lionelf67

                                    Hi Vladimir, can I know if you finally solved your problem and how? We have same environnement with Lenovo HX Nutanix, X722 and Mellanox switch. We got breakout cables from Mellanox and are not able to get all ports up. Only few are up and it's not always the same... Thank you.

                                    1 2 Previous Next