2 Replies Latest reply on Jun 17, 2017 4:06 AM by Intel Corporation

    X710 - VFs - TX driver issue detected, PF reset issued, when running iperf3

    vtran

      Hi,

       

      * In summary:

      When I run iperf3 between 2 machines throgh SRIO VFs interface and I got the following error in dmesg:

       

      TX driver issue detected, PF reset issued

       

      * How to setup and detail results

      + I got 2 machines each with 1 X710 4ports 10G cards.  These 2 X710 cards has direct cable connect between them.

      + On machine 1, I run the following commands to create VF0 and setup its IP. VF0 has pci address of 0000:04:0a.0

          $ echo 1 > /sys/bus/pci/devices/0000\:04\:00.2/sriov_numvfs

          $ ifconfig enp4s0f2 mtu 9700

          $ ifconfig enp4s10 1.1.1.1 netmask 255.255.255.0 mtu 9216

        $ echo 'msg_enable 0xffff' > /sys/kernel/debug/i40e/0000\:04\:00.2/command

       

      + On machine 2, I run the following commands to create VF0 and setup its IP. VF0 has pci address of 0000:04:0a.0

          $ echo 1 > /sys/bus/pci/devices/0000\:04\:00.2/sriov_numvfs

          $ ifconfig enp4s0f2 mtu 9700

          $ ifconfig enp4s10 1.1.1.2 netmask 255.255.255.0 mtu 9216

       

      + I was able to ping 1.1.1.2 from machine 1 and ping 1.1.1.1 from machine 2

      + On machine 2, start iperf3 server:

         $ iperf3 -s -p 8000

       

      + On machine 1, start iperf3 client:

         $ iperf3 -c 1.1.1.12 -p 8000 -l 64 -p 4

         $ dmesg

       

      [ 5895.270158] i40e 0000:04:00.2: Malicious Driver Detection event 0x00 on TX queue 68 PF number 0x02 VF number 0x40

      [ 5895.270166] i40e 0000:04:00.2: TX driver issue detected, PF reset issued

      [ 5895.270170] i40e 0000:04:00.2: TX driver issue detected on VF 0

      [ 5895.270173] i40e 0000:04:00.2: Too many MDD events on VF 0, disabled

      [ 5895.270176] i40e 0000:04:00.2: Use PF Control I/F to re-enable the VF

      [ 5895.284494] i40evf 0000:04:0a.0: PF reset warning received

      [ 5895.284500] i40evf 0000:04:0a.0: Scheduling reset task

      [ 5895.334703] i40e 0000:04:00.2: Query for DCB configuration failed, err I40E_ERR_ADMIN_QUEUE_ERROR aq_err I40E_AQ_RC_EPERM

      [ 5895.334711] i40e 0000:04:00.2: DCB init failed -53, disabled

      [ 5895.612086] i40e 0000:04:00.2: Malicious Driver Detection event 0x00 on TX queue 67 PF number 0x02 VF number 0x40

      [ 5895.612096] i40e 0000:04:00.2: Too many MDD events on VF 0, disabled

      [ 5895.612098] i40e 0000:04:00.2: Use PF Control I/F to re-enable the VF

      [ 5895.640978] i40e 0000:04:00.2: Invalid message from VF 0, opcode 3, len 4

       

      * X710 card ports info

      04:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)

      04:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)

      04:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)

      04:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)

       

      * This is driver info of PF X710:

      driver: i40e

      version: 1.4.25-k

      firmware-version: 5.04 0x80002549 0.0.0

      expansion-rom-version:

      bus-info: 0000:04:00.2

      supports-statistics: yes

      supports-test: yes

      supports-eeprom-access: yes

      supports-register-dump: yes

      supports-priv-flags: yes

       

      * This is the PF X710 card info:

      04:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)

        Subsystem: Intel Corporation Ethernet Converged Network Adapter X710

        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+

        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

        Latency: 0, Cache Line Size: 256 bytes

        Interrupt: pin A routed to IRQ 35

        Region 0: Memory at dc800000 (64-bit, prefetchable) [size=8M]

        Region 3: Memory at dc7f8000 (64-bit, prefetchable) [size=32K]

        Expansion ROM at df380000 [disabled] [size=512K]

        Capabilities: [40] Power Management version 3

        Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)

        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-

        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+

        Address: 0000000000000000  Data: 0000

        Masking: 00000000  Pending: 00000000

        Capabilities: [70] MSI-X: Enable+ Count=129 Masked-

        Vector table: BAR=3 offset=00000000

        PBA: BAR=3 offset=00001000

        Capabilities: [a0] Express (v2) Endpoint, MSI 00

        DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us

        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+

        DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+

        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop- FLReset-

        MaxPayload 128 bytes, MaxReadReq 512 bytes

        DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-

        LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L0s <2us, L1 <16us

        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+

        LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+

        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

        LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

        DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported

        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-

        EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-

        Capabilities: [e0] Vital Product Data

        Product Name: XL710 40GbE Controller

        Read-only fields:

        [PN] Part number:

        [EC] Engineering changes:

        [FG] Unknown:

        [LC] Unknown:

        [MN] Manufacture ID:

        [PG] Unknown:

        [SN] Serial number:

        [V0] Vendor specific:

        [RV] Reserved: checksum good, 0 byte(s) reserved

        Read/write fields:

        [V1] Vendor specific:

        End

        Capabilities: [100 v2] Advanced Error Reporting

        UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

        UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-

        UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-

        CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+

        CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+

        AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-

        Capabilities: [140 v1] Device Serial Number 90-9c-a4-ff-ff-fe-fd-3c

        Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)

        ARICap: MFVC- ACS-, Next Function: 3

        ARICtl: MFVC- ACS-, Function Group: 0

        Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)

        IOVCap: Migration-, Interrupt Message Number: 000

        IOVCtl: Enable+ Migration- Interrupt- MSE+ ARIHierarchy-

        IOVSta: Migration-

        Initial VFs: 32, Total VFs: 32, Number of VFs: 1, Function Dependency Link: 02

        VF offset: 78, stride: 1, Device ID: 154c

        Supported Page Size: 00000553, System Page Size: 00000001

        Region 0: Memory at 00000000dc400000 (64-bit, prefetchable)

        Region 3: Memory at 00000000dc380000 (64-bit, prefetchable)

        VF Migration: offset: 00000000, BIR: 0

        Capabilities: [1a0 v1] Transaction Processing Hints

        Device specific mode supported

        No steering table available

        Capabilities: [1b0 v1] Access Control Services

        ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-

        ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-

        Kernel driver in use: i40e

        Kernel modules: i40e

      04:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)

        Subsystem: Intel Corporation Ethernet Converged Network Adapter X710

        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+

        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

        Latency: 0, Cache Line Size: 256 bytes

        Interrupt: pin A routed to IRQ 35

        Region 0: Memory at dc800000 (64-bit, prefetchable) [size=8M]

        Region 3: Memory at dc7f8000 (64-bit, prefetchable) [size=32K]

        Expansion ROM at df380000 [disabled] [size=512K]

        Capabilities: [40] Power Management version 3

        Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)

        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-

        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+

        Address: 0000000000000000  Data: 0000

        Masking: 00000000  Pending: 00000000

        Capabilities: [70] MSI-X: Enable+ Count=129 Masked-

        Vector table: BAR=3 offset=00000000

        PBA: BAR=3 offset=00001000

        Capabilities: [a0] Express (v2) Endpoint, MSI 00

        DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us

        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+

        DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+

        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop- FLReset-

        MaxPayload 128 bytes, MaxReadReq 512 bytes

        DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-

        LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L0s <2us, L1 <16us

        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+

        LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+

        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

        LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

        DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported

        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-

        EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-

        Capabilities: [e0] Vital Product Data

        Product Name: XL710 40GbE Controller

        Read-only fields:

        [PN] Part number:

        [EC] Engineering changes:

        [FG] Unknown:

        [LC] Unknown:

        [MN] Manufacture ID:

        [PG] Unknown:

        [SN] Serial number:

        [V0] Vendor specific:

        [RV] Reserved: checksum good, 0 byte(s) reserved

        Read/write fields:

        [V1] Vendor specific:

        End

        Capabilities: [100 v2] Advanced Error Reporting

        UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

        UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-

        UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-

        CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+

        CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+

        AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-

        Capabilities: [140 v1] Device Serial Number 90-9c-a4-ff-ff-fe-fd-3c

        Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)

        ARICap: MFVC- ACS-, Next Function: 3

        ARICtl: MFVC- ACS-, Function Group: 0

        Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)

        IOVCap: Migration-, Interrupt Message Number: 000

        IOVCtl: Enable+ Migration- Interrupt- MSE+ ARIHierarchy-

        IOVSta: Migration-

        Initial VFs: 32, Total VFs: 32, Number of VFs: 1, Function Dependency Link: 02

        VF offset: 78, stride: 1, Device ID: 154c

        Supported Page Size: 00000553, System Page Size: 00000001

        Region 0: Memory at 00000000dc400000 (64-bit, prefetchable)

        Region 3: Memory at 00000000dc380000 (64-bit, prefetchable)

        VF Migration: offset: 00000000, BIR: 0

        Capabilities: [1a0 v1] Transaction Processing Hints

        Device specific mode supported

        No steering table available

        Capabilities: [1b0 v1] Access Control Services

        ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-

        ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-

        Kernel driver in use: i40e

        Kernel modules: i40e

       

      * This is the VF X710 card info:

      04:0a.0 Ethernet controller: Intel Corporation XL710/X710 Virtual Function (rev 01)

        Subsystem: Intel Corporation XL710/X710 Virtual Function

        Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-

        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

        Latency: 0

        Region 0: [virtual] Memory at dc400000 (64-bit, prefetchable) [size=64K]

        Region 3: [virtual] Memory at dc380000 (64-bit, prefetchable) [size=16K]

        Capabilities: [70] MSI-X: Enable+ Count=5 Masked-

        Vector table: BAR=3 offset=00000000

        PBA: BAR=3 offset=00002000

        Capabilities: [a0] Express (v2) Endpoint, MSI 00

        DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us

        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+

        DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-

        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-

        MaxPayload 128 bytes, MaxReadReq 128 bytes

        DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

        LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L0s <2us, L1 <16us

        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+

        LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-

        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

        LnkSta: Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-

        DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported

        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-

        EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-

        Capabilities: [100 v2] Advanced Error Reporting

        UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

        UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

        UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

        CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-

        CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-

        AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-

        Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)

        ARICap: MFVC- ACS-, Next Function: 0

        ARICtl: MFVC- ACS-, Function Group: 0

        Capabilities: [1a0 v1] Transaction Processing Hints

        Device specific mode supported

        No steering table available

        Capabilities: [1d0 v1] Access Control Services

        ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-

        ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-

        Kernel driver in use: i40evf

        Kernel modules: i40evf

       

      * This is driver info of VF X710:

      driver: i40evf

      version: 1.4.15-k

      firmware-version: N/A

      expansion-rom-version:

      bus-info: 0000:04:0a.0

      supports-statistics: yes

      supports-test: no

      supports-eeprom-access: no

      supports-register-dump: no

      supports-priv-flags: yes

       

      * OS info:

      DISTRIB_ID=Ubuntu

      DISTRIB_RELEASE=16.04

      DISTRIB_CODENAME=xenial

      DISTRIB_DESCRIPTION="Ubuntu 16.04.1 LTS"

       

      Please let me know if you need more info.

       

      Thanks!