10 Replies Latest reply: Jun 15, 2012 8:36 AM by Patrick_Kutch RSS

Guest OS enumeration for VF PCIe device for 82599

vaishali Community Member
Currently Being Moderated

I am struk with enabling the MSIx in the Guest OS for 82599.
In hardware MSIx capability is set it has "70" PF driver is loaded to enable the Virtual function in the host then "lspci" command is used to check whether VF is enabled or not .

VF configuration space for msix "capability [70] MSIx " it is showing correct in the host.

But if i map the VF to Guest OS then it is showing "capability [40] MSIx".SO MSIx is enabled is not detect by the hardware and the VF interrupt  is not functioning properly

 

LSPCI output for VFs in Guest

04:10.0 Ethernet controller: Intel Corporation 82559 Ethernet Controller Virtual Function (rev 01)

Subsystem: Intel Corporation Device 0003

Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

Latency: 0

Region 0: [virtual] Memory at df100000 (64-bit, non-prefetchable) [size=16K]

Region 3: [virtual] Memory at df200000 (64-bit, non-prefetchable) [size=16K]

Capabilities: [70] MSI-X: Enable+ Count=3 Masked-

Vector table: BAR=3 offset=00000000

PBA: BAR=3 offset=00002000

Capabilities: [a0] Express (v0) Endpoint, MSI 00

DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us

ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-

DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-

RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-

MaxPayload 128 bytes, MaxReadReq 128 bytes

DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

LnkCap: Port #0, Speed unknown, Width x0, ASPM unknown, Latency L0 <64ns, L1 <1us

ClockPM- Surprise- LLActRep- BwNot-

LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-

ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

LnkSta: Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-

Capabilities: [100 v1] Advanced Error Reporting

UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-

CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-

AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-

Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)

ARICap: MFVC- ACS-, Next Function: 0

ARICtl: MFVC- ACS-, Function Group: 0

Kernel driver in use: ixgbevf

Kernel modules: ixgbevf

 

MAPPED VFS  to a GUEST OS -lspci output

00:05.0 Ethernet controller: Intel Corporation 82559 Ethernet Controller Virtual Function (rev 01)

00: 86 80 ed 10 07 04 10 00 01 00 00 02 00 00 00 00

10: 00 00 04 f2 00 00 00 00 00 00 00 00 00 40 04 f2

20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 03 00

30: 00 00 00 00 40 00 00 00 00 00 00 00 00 00 00 00

40: 11 00 02 80 03 00 00 00 03 20 00 00 00 00 00 00

50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

70: 11 a0 02 00 03 00 00 00 03 20 00 00 00 00 00 00

80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

a0: 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

  

00:05.0 Ethernet controller: Intel Corporation 82559 Ethernet Controller Virtual Function (rev 01)

Subsystem: Intel Corporation Device 0003

Physical Slot: 5

Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

Latency: 0

Region 0: Memory at f2040000 (32-bit, non-prefetchable) [size=16K]

Region 3: Memory at f2044000 (32-bit, non-prefetchable) [size=16K]

Capabilities: [40] MSI-X: Enable+ Count=3 Masked-

Vector table: BAR=3 offset=00000000

PBA: BAR=3 offset=00002000

Kernel driver in use: ixgbevf

Kernel modules: ixgbevf

 

Fedora 14 is used in both host and Guest.
CPU : Intel xeon
Hypervisor : QEMU/KVM
VMM : Fedora 14 default
Kernel Version : 2.6.35

  • 1. Re: Guest OS enumeration for VF PCIe device for 82599
    MARK HANNA Community Member
    Currently Being Moderated

    I asked around about your post and heard back from a few developers.  In our labs MSI-X capability is displayed in the correct offset of [70]. We do not know why you are seeing something different. Of course there could be something differentwith QEMU or the guest on your setup.

     

    You might consider using RHEL 6.x. The tech brief at http://www.intel.com/content/www/us/en/network-adapters/10-gigabit-network-adapters/ethernet-x520-sr-iov-red-hat-tech-brief.html covers the configuration and this is a configuration that is known to work. Maybe going through the paper you can find something that will help you in your setup. On the other hand, if the issue is related to differences with any part of the distribution, then this paper might not help you.

     

    When you say, "the VF interrupt  is not functioning properly," what is happening? Is the VF able to pass traffic? Are you expecting a different number of interrupts to be used? Please explain further.

     

    What is in the output of the 'cat /proc/interrupts` for both the PF and VF?

     

    Mark H

  • 2. Re: Guest OS enumeration for VF PCIe device for 82599
    vaishali Community Member
    Currently Being Moderated

    Thanks for the support i will try to load the RHEL 6.x  an d thanks for the document

    Mean time the information on interrupts

     

    Interrupts for PF

    [root@veena /]# lspci | grep 82599

    83:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit Network Connection (rev 01)

    83:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit Network Connection (rev 01)

    [root@veena /]# modprobe ixgbe max_vfs=1

    [root@veena /]# lspci | grep 82599

    83:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit Network Connection (rev 01)

    83:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit Network Connection (rev 01)

    83:10.0 Ethernet controller: Intel Corporation 82559 Ethernet Controller Virtual Function (rev 01)

    83:10.1 Ethernet controller: Intel Corporation 82559 Ethernet Controller Virtual Function (rev 01)

    [root@veena /]# cat /proc/interrupts

    115: 2 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth4-rx-0

    116: 2 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth4-tx-0

    117: 9 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth4:lsc

    118: 2 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth5-rx-0

    119: 2 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth5-tx-0

    120: 8 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth5:lsc

    121: 2 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth6-rx-0

    122: 2 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth6-tx-0

    123: 3 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth6:mbx

    124: 1 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth7-rx-0

    125: 1 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth7-tx-0

    126: 3 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth7:mbx

     

     

    I have assigned 83:10.0 device to the guest OS

     

    Interrupts on VF

    [root@vm1 VM1]# lspci | grep 82599

    00:06.0 Ethernet controller: Intel Corporation 82559 Ethernet Controller Virtual Function (rev 01)

    [root@vm1 VM1]# cat /proc/interrupts

    CPU0

    0: 278 IO-APIC-edge timer

    1: 199 IO-APIC-edge i8042

    2: 0 XT-PIC-XT cascade

    4: 2 IO-APIC-edge

    8: 0 IO-APIC-edge rtc0

    12: 130 IO-APIC-edge i8042

    14: 8818 IO-APIC-edge ata_piix

    15: 878 IO-APIC-edge ata_piix

    34: 459 IO-APIC-fasteoi virtio1

    35: 1104 IO-APIC-fasteoi uhci_hcd:usb1, Intel 82801AA-ICH

    40: 0 PCI-MSI-edge virtio0-config

    41: 8 PCI-MSI-edge virtio0-input

    42: 1 PCI-MSI-edge virtio0-output

    43: 0 PCI-MSI-edge eth1-rx-0

    44: 0 PCI-MSI-edge eth1-tx-0

    45: 0 PCI-MSI-edge eth1:mbx

    NMI: 0 Non-maskable interrupts

    LOC: 36375 Local timer interrupts

    SPU: 0 Spurious interrupts

    PMI: 0 Performance monitoring interrupts

    PND: 0 Performance pending work

    RES: 0 Rescheduling interrupts

    CAL: 0 Function call interrupts

    TLB: 0 TLB shootdowns

    TRM: 0 Thermal event interrupts

    THR: 0 Threshold APIC interrupts

    MCE: 0 Machine check exceptions

    MCP: 1 Machine check polls

    ERR: 0

    MIS: 0

    [root@vm1 VM1]# lspci

    00:06.0 Ethernet controller: Intel Corporation 82559 Ethernet Controller Virtual Function (rev 01)

    Subsystem: Intel Corporation Device 0003

    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+

    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

    Latency: 0

    Region 0: Memory at f2040000 (32-bit, non-prefetchable) [size=16K]

    Region 3: Memory at f2044000 (32-bit, non-prefetchable) [size=16K]

    Capabilities: [40] MSI-X: Enable+ Count=3 Masked-

    Vector table: BAR=3 offset=00000000

    PBA: BAR=3 offset=00002000

    Kernel driver in use: ixgbevf

    Kernel modules: ixgbevf

     

     

     

  • 3. Re: Guest OS enumeration for VF PCIe device for 82599
    Patrick_Kutch Community Member
    Currently Being Moderated

    My SR-IOV Guru's believe this is likely a Fedora 14 problem.  We would like to suggest using the latest Fedora release, or try to reproduce the problem with RHEL 6.2.

     

    It takes a several components to make SR-IOV work, BIOS, platform,end-point device (the Intel 82599) and of course the OS.  There are lots and lots of patches to the linux kernel for SR-IOV support.  Sometimes not all of those patches make it into a distro release due to any number of reasons.

     

    Please update us on your efforts.

     

    thanks,

     

    Patrick

  • 4. Re: Guest OS enumeration for VF PCIe device for 82599
    vaishali Community Member
    Currently Being Moderated

    Hi Patrick,

    I tried on RHEL 6.2 the previous issue was not there,new issue has been arrived.

    Configuration:

    1. CONFIG_DMAR_TABLE=y in kernel Config menu,
    2. intel_iommu=on in grub.config
    3. VT-d enabled in bios.

    Scenarios 1:

    • Loaded the ixgbe max_vfs=0, and started the testing.
      Below are dmesg from the kernel.
      DRHD: handling fault status reg 2
      DMAR:[DMA Read] Request device [03:00.0] fault addr 619c00000 DMAR:[fault reason 01] Present bit in root entry is clear

    Scenarios 2:

    • loaded the ixgbe max_vfs=2, and started the testing.
    • Assigned the Vfs to the GuestOs.VFs are working fine.

    Didn't receive any DMAR error.

     


    If intel_iommu=off or CONFIG_DMAR_TABLE=n,the above issue is not seen and ethernet driver is working fine(only PF driver ).But VFs cannot be assigned to the guest OS.

     

    Can u please tell where it went worng ......................

  • 5. Re: Guest OS enumeration for VF PCIe device for 82599
    Patrick_Kutch Community Member
    Currently Being Moderated

    Thanks for trying that and reporting your findings.  It would appear to be some kind of DMAR issue - we are preparing to investigate.  It makes sense if there is a DMAR problem that VF's assigned to a VM would no work.

     

    To aid in the investigation, our engineers have asked if you are using the inbox (kernel) drivers, or the latest from sourceforge.

     

    thanks,

    Patrick


  • 6. Re: Guest OS enumeration for VF PCIe device for 82599
    vaishali Community Member
    Currently Being Moderated

    I am using the kernel version 3.2.13 rt23.

  • 7. Re: Guest OS enumeration for VF PCIe device for 82599
    Patrick_Kutch Community Member
    Currently Being Moderated

    That is a very old driver.  I would recommend getting the latest versions (PF and VF) - which is available here:

    http://sourceforge.net/projects/e1000/files/ixgbe%20stable/

     

    Please give this a try and let us know how it works.

     

    thanks,

     

    Patrick

  • 8. Re: Guest OS enumeration for VF PCIe device for 82599
    vaishali Community Member
    Currently Being Moderated

    Hi Patrick,

    Thanks for the support................

    I will download and test the lastest driver and let u know  the result.

     

    Veena

  • 9. Re: Guest OS enumeration for VF PCIe device for 82599
    vaishali Community Member
    Currently Being Moderated

    It is working with the latest source code Thnaks for the suppot

     

     

    I am curious to know what is reason for the DMAR issue????

     

    DRHD: handling fault status reg 2

    DMAR:[DMA Read] Request device [03:00.0] fault addr 619c00000 DMAR:[fault reason 01] Present bit in root entry is clear

  • 10. Re: Guest OS enumeration for VF PCIe device for 82599
    Patrick_Kutch Community Member
    Currently Being Moderated

    Great, I am happy it working and to be able to help.

     

    I do not know what is happening with this DMAR issue you have described.  We suspect it may be a BIOS issue, however I will pass along the information you provided to our SR-IOV experts and relay back anything they may offer by way of explaination.

More Like This

  • Retrieving data ...

Legend

  • Correct Answers - 4 points
  • Helpful Answers - 2 points