1 Reply Latest reply on Nov 11, 2015 12:04 PM by Carl_Wilson

    X710 Ethernet Adapter not Being Brought Up by IXGBE Driver

    KarlW

      Greetings Wired Ethernet,

       

      So I have two X710-DA4 adapters installed into two different servers (different motherboards on each) experiencing the same problem on both bringing up the interfaces.

       

      I'm running SLES 11 SP 3 on both (I believe this issue would exist on RHEL or SLES 11 sp4 as well though, still waiting to verify)

       

      lspci | grep Eth output:

       

      01:00.0 Ethernet controller: Intel Corporation Device 1572 (rev 01)

      01:00.1 Ethernet controller: Intel Corporation Device 1572 (rev 01)

      01:00.2 Ethernet controller: Intel Corporation Device 1572 (rev 01)

      01:00.3 Ethernet controller: Intel Corporation Device 1572 (rev 01)

      42:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

      42:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

      42:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

      42:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

       

      The Device 1572 is the X710 DA4 interfaces, but ifconfig -a only show's the on-board 1G I350 interfaces.

       

      I've updated to ixgbe driver 4.1.5 , when I rmmod and insmod ixgbe , dmesg only reports:

       

      [  101.916366] Disabling lock debugging due to kernel taint

      [  101.918805] Intel(R) 10 Gigabit PCI Express Network Driver - version 4.1.5

      [  101.918809] Copyright (c) 1999-2015 Intel Corporation.

      [ 1370.304497] Intel(R) 10 Gigabit PCI Express Network Driver - version 4.1.5

      [ 1370.304503] Copyright (c) 1999-2015 Intel Corporation.

       

      I also found two versions of NVM Update Tool (1.24.33.08) and (1.25.20.12)

       

      It shows the following output on both:

       

      Num Description                            Device-Id B:D   Adapter Status

      === ====================================== ========= ===== ====================

      01) Intel(R) I350 Gigabit Network Connecti 8086-1521 66:00 Update not available

      02) Intel(R) Ethernet Converged Network Ad 8086-1572 01:00 Access error

       

      I looked at some of the BIOS PCI settings, but I wouldn't know which ones to tune that might hint at what is happening. A verbose look at lspci for those interfaces show that the PCI device is training up at gen3 x8 successfully:

       

      01:00.0 Ethernet controller: Intel Corporation Device 1572 (rev 01)

              Subsystem: Intel Corporation Device 0001

              Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-

              Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

              Latency: 0, Cache Line Size: 64 bytes

              Interrupt: pin A routed to IRQ 11

              Region 0: Memory at 383ffe800000 (64-bit, prefetchable) [size=8M]

              Region 3: Memory at 383fff818000 (64-bit, prefetchable) [size=32K]

              Expansion ROM at ab280000 [disabled] [size=512K]

              Capabilities: [40] Power Management version 3

                      Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)

                      Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-

              Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+

                      Address: 0000000000000000  Data: 0000

                      Masking: 00000000  Pending: 00000000

              Capabilities: [70] MSI-X: Enable- Count=129 Masked-

                      Vector table: BAR=3 offset=00000000

                      PBA: BAR=3 offset=00001000

              Capabilities: [a0] Express (v2) Endpoint, MSI 00

                      DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us

                              ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+

                      DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported-

                              RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-

                              MaxPayload 256 bytes, MaxReadReq 512 bytes

                      DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-

                      LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Latency L0 <2us, L1 <16us

                              ClockPM- Surprise- LLActRep- BwNot-

                      LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+

                              ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

                      LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

                      DevCap2: Completion Timeout: Range ABCD, TimeoutDis+

                      DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-

                      LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB

                               Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-

                               Compliance De-emphasis: -6dB

                      LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+

                               EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-

              Capabilities: [100 v2] Advanced Error Reporting

                      UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

                      UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

                      UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-

                      CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+

                      CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+

                      AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-

              Capabilities: [140 v1] Device Serial Number 8c-7f-43-ff-ff-ed-e0-00

              Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)

                      ARICap: MFVC- ACS-, Next Function: 1

                      ARICtl: MFVC- ACS-, Function Group: 0

              Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)

                      IOVCap: Migration-, Interrupt Message Number: 000

                      IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+

                      IOVSta: Migration-

                      Initial VFs: 32, Total VFs: 32, Number of VFs: 32, Function Dependency Link: 00

                      VF offset: 16, stride: 1, Device ID: 154c

                      Supported Page Size: 00000553, System Page Size: 00000001

                      Region 0: Memory at 0000383fff600000 (64-bit, prefetchable)

                      Region 3: Memory at 0000383fff9a0000 (64-bit, prefetchable)

                      VF Migration: offset: 00000000, BIR: 0

              Capabilities: [1a0 v1] Transaction Processing Hints

                      Device specific mode supported

                      No steering table available

              Capabilities: [1b0 v1] Access Control Services

                      ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-

                      ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-

              Capabilities: [1d0 v1] #19

       

       

       

      Any help or ideas would be greatly appreciated on next steps or data collection that might be useful.