1 2 Previous Next 16 Replies Latest reply on Apr 1, 2015 11:11 PM by wb_Intel

    Using XL710 with 10Gb link

    M_V

      Hi!

       

      I am trying to use one of the dual port XL710 adapters to connect with a 10Gb port. For that, I am trying to use an adapter from Mellanox: MAM1Q00A-QSA Cable Module ETH 10GbE 40Gb/s to 10Gb/s QSFP to SFP+. Unfortunately, just plugging it into XL710 and connecting the resulting SFP+ does not lead to an established link. So, I wonder, whether XL710 supports such configuration? My setup:

       

      OS: Ubuntu 14.04

       

      lspci -v:

       

      83:00.0 Ethernet controller: Intel Corporation Ethernet Controller LX710 for 40GbE QSFP+ (rev 01)

              Subsystem: Super Micro Computer Inc Device 0000

              Physical Slot: 6

       

      ethtool -i eth4

      driver: i40e

      version: 1.2.37

      firmware-version: f4.22.26225 a1.2 n4.24 e12f3

       

      ethtool eth4

      Settings for eth4:

              Supported ports: [ ]

              Supported link modes:   40000baseCR4/Full

                                      40000baseSR4/Full

                                      40000baseLR4/Full

              Supported pause frame use: Symmetric

              Supports auto-negotiation: No

              Advertised link modes:  40000baseCR4/Full

                                      40000baseSR4/Full

                                      40000baseLR4/Full

              Advertised pause frame use: No

              Advertised auto-negotiation: No

              Speed: Unknown!

              Duplex: Unknown! (255)

              Port: Other

              PHYAD: 0

              Transceiver: external

              Auto-negotiation: off

              Supports Wake-on: g

              Wake-on: d

              Current message level: 0x0000000f (15)

                                     drv probe link timer

              Link detected: no

       

      Thank you

        • 1. Re: Using XL710 with 10Gb link
          wb_Intel

          Hi M_V,

            We will check on this.

           

          thanks,

          wb

          • 2. Re: Using XL710 with 10Gb link
            wb_Intel

            Hi M_V,

                The Mellanox MAM1Q00A-QSA Cable is not the tested cable. You may refer to the list of supported cable at http://www.intel.ph/content/dam/www/public/us/en/documents/product-briefs/ethernet-xl710-brief.pdf

                 Further just to double check based on your information, are you connecting one port of XL710 to another "network adapter" instead of a switch?

             

            Thanks,

            wb

            • 3. Re: Using XL710 with 10Gb link
              M_V

              Hi WB:

               

              True, this adapter is not listed (yet?). In the end, I would need to connect it to Intel 10GbE card (can't tell you exact model this time). However, the link was not established neither with the card nor 10GbE switch.

               

              It seems to me that I miss something very simple, such as autonegotiation of the speed of the link. How can I enable 10000baseXXXX link mode on a port of XL710 card in Linux? As you could see above in ethtool printout, this mode is not supported.

               

              Best,

               

              M_V

              • 4. Re: Using XL710 with 10Gb link
                wb_Intel

                Hi M_V,

                 

                Thanks for the clarification. Let me check on this.

                 

                rgds,

                wb

                • 5. Re: Using XL710 with 10Gb link
                  M_V

                  Hi WB:

                   

                  since it's probably not easy to get your hands on the adapter, let me ask a simpler question:

                   

                  * does XL710 support 10Gb data rates? It seems that it does support it according to the "Data Brief" of the adapter (

                  Data rate supported per port: • Optical: 10 GbE/40 GbE, • Direct Attach: 40 GbE).

                   

                  If it does support 10Gb, how do I enable it in Linux?

                   

                  Best,

                   

                  M_V

                  • 6. Re: Using XL710 with 10Gb link
                    wb_Intel

                    Hi M_V,

                     

                        You may follow the instruction in the guide below to enable the adapter to configure Port Speed  for XL710.

                     

                    http://www.intel.com/content/dam/www/public/us/en/documents/guides/qsfp-configuration-utility-quick-usage-guide.pdf

                     

                       Should see 4 x 10Gb devices  with driver loaded

                       

                        Hope this helps.

                     

                     

                     

                    rgds,

                    wb

                    • 7. Re: Using XL710 with 10Gb link
                      M_V

                      Hi WB:

                       

                      thank you for instructions. However, I do have a problem. Namely, it seems that QCU is not working for us as it should (see below). In addition, in our server, the official name of the card is Supermicro AOC-S40G-i2Q which uses XL710 controller. I wounder if that the cause for the problems.

                       

                      Here is mu session with QCU:

                       

                      # ./qcu64e

                      Intel(R) QSFP+ Configuration Utility

                       

                      QCU version: v1.24.35.01

                      Copyright(C) 2015 by Intel Corporation.

                      Software released under Intel Proprietary License.

                       

                      NIC B:D  Ven-Dev  Mode  Adapter Name

                      === ===== ========= ====== ====================================================

                      1) 83:00 8086-1583 N/A    Intel(R) Ethernet Converged Network Adapter XL710-Q2

                       

                      Warning: No adapter selected.

                       

                      # ./qcu64e /nic=1 /info

                       

                      Intel(R) QSFP+ Configuration Utility

                       

                      QCU version: v1.24.35.01

                      Copyright(C) 2015 by Intel Corporation.

                      Software released under Intel Proprietary License.

                       

                      QSFP+ Configuration modification is not supported by this adapter.

                      # echo $?

                      5

                       

                      # ./qcu64e /exitcodes

                      Intel(R) QSFP+ Configuration Utility

                       

                      QCU version: v1.24.35.01

                      Copyright(C) 2015 by Intel Corporation.

                      Software released under Intel Proprietary License.

                       

                      Exit Codes:

                           0x0  := Requested action executed successfully.

                           0x1  := Insufficient privileges to run the utility.

                           0x2  := Invalid adapter selected.

                           0x3  := Bad command line parameter.

                           0x4  := Unsupported link mode selected.

                           0x5  := Adapter does not support link mode selection.

                           0x6  := No adapter selected.

                           0x7  := Memory allocation error.

                           0x8  := No base driver available.

                           0x9  := Unsupported base driver version detected.

                           0xA  := QSFP+ configuration for selected adapter not supported by this utility version.

                       

                       

                      Here are the driver/firmware info:

                       

                      ethtool -i eth4

                      driver: i40e

                      version: 1.2.37

                      firmware-version: f4.22.26225 a1.2 n4.24 e12f3

                       

                       

                      Best,

                       

                      M_V

                      • 8. Re: Using XL710 with 10Gb link
                        DHekimian

                        @M_V

                         

                        For the QCU utility to work correctly, you need to update the firmware and have the latest Linux driver (v1.2.37) loaded. It looks like you have the latest Linux Driver.

                         

                        NVM-Update-Utility-for-Intel-Ethernet-Converged-Network-Adapter-XL710-X710-Series

                         

                        Be sure to reboot after running NVM Update before running QCU for it to work properly.

                         

                        My XL710 Card:

                        driver: i40e

                        firmware-version: f4.33.31377 a1.2 n4.85 e1930

                         

                        - David

                        • 9. Re: Using XL710 with 10Gb link
                          M_V

                          Hi David:

                           

                          strange, I get from NVM Update utility (run in linux) that firmware is up-to-date:

                           

                          # ./nvmupdate64e

                           

                          Intel(R) Ethernet NVM Update Tool

                          NVMUpdate version 1.24.33.08

                          Copyright (C) 2013 - 2015 Intel Corporation.

                           

                           

                          WARNING: TO AVOID DAMAGE TO YOUR DEVICE, DO NOT EXIT OR REBOOT OR POWER OFF THE SYSTEM DURING THIS UPDATE

                          Inventory in progress. Please wait [.......-**]

                           

                           

                          Num Description                            Device-Id B:D   Adapter Status

                          === ====================================== ========= ===== ====================

                          01) Intel(R) 82599 10 Gigabit Network Conn 8086-1557 03:00 Update not available

                          02) Intel(R) I350 Gigabit Network Connecti 8086-1521 04:00 Update not available

                          03) Intel(R) Ethernet Converged Network Ad 8086-1583 131:00 Up to date         

                           

                           

                          Tool execution completed with the following status: All operations completed successfully

                           

                          ethtool -i eth3

                          driver: i40e

                          version: 1.2.37

                          firmware-version: f4.33.31377 a1.2 n4.42 e18a3

                           

                          Do I have to unplug the server from mains? Or is the reboot sufficient?

                           

                          Best,

                           

                          M_V

                          • 10. Re: Using XL710 with 10Gb link
                            DHekimian

                            @M_V,

                             

                            Looking at your post above:

                            ethtool -i eth4

                            driver: i40e

                            version: 1.2.37

                            firmware-version: f4.22.26225 a1.2 n4.24 e12f3

                             

                            And your current post:

                            ethtool -i eth3

                            driver: i40e

                            version: 1.2.37

                            firmware-version: f4.33.31377 a1.2 n4.42 e18a3

                             

                            Shows the Firmware to have been updated between each post.

                             

                            I upgraded the firmware on (8) XL710 cards without powering off, a reboot was sufficient. Same with running the QCU utility.

                             

                            - David

                            • 11. Re: Using XL710 with 10Gb link
                              M_V

                              I guess I haven't rebooted after the update. However, even now, I still get 'no support' message from QCU. I do wonder if its due to the fact that its Supermicro-branded version?

                               

                              # ./qcu64e /nic=1 /info

                              Intel(R) QSFP+ Configuration Utility

                               

                              QCU version: v1.24.35.01

                              Copyright(C) 2015 by Intel Corporation.

                              Software released under Intel Proprietary License.

                               

                              QSFP+ Configuration modification is not supported by this adapter.

                               

                              # echo $?

                              5

                               

                              # ethtool -i eth4

                              driver: i40e

                              version: 1.2.37

                              firmware-version: f4.33.31377 a1.2 n4.42 e18a3

                              bus-info: 0000:83:00.1

                              supports-statistics: yes

                              supports-test: yes

                              supports-eeprom-access: yes

                              supports-register-dump: yes

                              supports-priv-flags: yes

                              • 12. Re: Using XL710 with 10Gb link
                                DHekimian

                                What does './qcu64e' show after the reboot? You're looking for something in the Mode to look like (1x40 or 4x10, etc).

                                 

                                Your Card:

                                NIC B:D   Ven-Dev   Mode   Adapter Name

                                === ===== ========= ====== ====================================================

                                1)  83:00 8086-1583 N/A    Intel(R) Ethernet Converged Network Adapter XL710-Q2


                                My Card:

                                NIC B:D   Ven-Dev   Mode   Adapter Name

                                === ===== ========= ====== ====================================================

                                1) 03:00 8086-1584 4x10   Intel(R) Ethernet Converged Network Adapter XL710-Q1

                                 

                                Not sure about the Supermicro OEM vs Intel Branded.


                                - David


                                • 13. Re: Using XL710 with 10Gb link
                                  M_V

                                  It still shows N/A:

                                   

                                  # ./qcu64e

                                  Intel(R) QSFP+ Configuration Utility

                                   

                                  QCU version: v1.24.35.01

                                  Copyright(C) 2015 by Intel Corporation.

                                  Software released under Intel Proprietary License.

                                   

                                  NIC B:D   Ven-Dev   Mode   Adapter Name

                                  === ===== ========= ====== ====================================================

                                  1) 83:00 8086-1583 N/A    Intel(R) Ethernet Converged Network Adapter XL710-Q2

                                   

                                  I'll try to power-circle it on Monday. Thank you very much for  help!

                                   

                                  M_V

                                  • 14. Re: Using XL710 with 10Gb link
                                    M_V

                                    Unfortunately, no change after power-circle. QCU and ethtool show the same info:

                                     

                                    ethtool -i eth3

                                    driver: i40e

                                    version: 1.2.37

                                    firmware-version: f4.33.31377 a1.2 n4.42 e18a3

                                     

                                     

                                    ./qcu64e /nic=1 /info

                                    Intel(R) QSFP+ Configuration Utility

                                     

                                    QCU version: v1.24.35.01

                                    Copyright(C) 2015 by Intel Corporation.

                                    Software released under Intel Proprietary License.

                                     

                                    QSFP+ Configuration modification is not supported by this adapter.

                                     

                                     

                                    ./qcu64e

                                    Intel(R) QSFP+ Configuration Utility

                                     

                                    QCU version: v1.24.35.01

                                    Copyright(C) 2015 by Intel Corporation.

                                    Software released under Intel Proprietary License.

                                     

                                    NIC B:D   Ven-Dev   Mode   Adapter Name

                                    === ===== ========= ====== ====================================================

                                    1) 83:00 8086-1583 N/A    Intel(R) Ethernet Converged Network Adapter XL710-Q2

                                     

                                    Warning: No adapter selected.

                                     

                                    Maybe we managed to get an adapter that does not support 4 x 10Gb?

                                     

                                    Best

                                     

                                    M_V

                                    1 2 Previous Next