10 Replies Latest reply: Jun 15, 2012 8:36 AM by Patrick_Kutch RSS

    Guest OS enumeration for VF PCIe device for 82599

    vaishali
    vaishali May 8, 2012 3:37 AM

      I am struk with enabling the MSIx in the Guest OS for 82599.
      In hardware MSIx capability is set it has "70" PF driver is loaded to enable the Virtual function in the host then "lspci" command is used to check whether VF is enabled or not .

      VF configuration space for msix "capability [70] MSIx " it is showing correct in the host.

      But if i map the VF to Guest OS then it is showing "capability [40] MSIx".SO MSIx is enabled is not detect by the hardware and the VF interrupt  is not functioning properly

       

      LSPCI output for VFs in Guest

      04:10.0 Ethernet controller: Intel Corporation 82559 Ethernet Controller Virtual Function (rev 01)

      Subsystem: Intel Corporation Device 0003

      Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-

      Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

      Latency: 0

      Region 0: [virtual] Memory at df100000 (64-bit, non-prefetchable) [size=16K]

      Region 3: [virtual] Memory at df200000 (64-bit, non-prefetchable) [size=16K]

      Capabilities: [70] MSI-X: Enable+ Count=3 Masked-

      Vector table: BAR=3 offset=00000000

      PBA: BAR=3 offset=00002000

      Capabilities: [a0] Express (v0) Endpoint, MSI 00

      DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us

      ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-

      DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-

      RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-

      MaxPayload 128 bytes, MaxReadReq 128 bytes

      DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

      LnkCap: Port #0, Speed unknown, Width x0, ASPM unknown, Latency L0 <64ns, L1 <1us

      ClockPM- Surprise- LLActRep- BwNot-

      LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-

      ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

      LnkSta: Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-

      Capabilities: [100 v1] Advanced Error Reporting

      UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

      UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

      UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

      CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-

      CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-

      AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-

      Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)

      ARICap: MFVC- ACS-, Next Function: 0

      ARICtl: MFVC- ACS-, Function Group: 0

      Kernel driver in use: ixgbevf

      Kernel modules: ixgbevf

       

      MAPPED VFS  to a GUEST OS -lspci output

      00:05.0 Ethernet controller: Intel Corporation 82559 Ethernet Controller Virtual Function (rev 01)

      00: 86 80 ed 10 07 04 10 00 01 00 00 02 00 00 00 00

      10: 00 00 04 f2 00 00 00 00 00 00 00 00 00 40 04 f2

      20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 03 00

      30: 00 00 00 00 40 00 00 00 00 00 00 00 00 00 00 00

      40: 11 00 02 80 03 00 00 00 03 20 00 00 00 00 00 00

      50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

      60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

      70: 11 a0 02 00 03 00 00 00 03 20 00 00 00 00 00 00

      80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

      90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

      a0: 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

      b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

      c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

      d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

      e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

      f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

      　 

      00:05.0 Ethernet controller: Intel Corporation 82559 Ethernet Controller Virtual Function (rev 01)

      Subsystem: Intel Corporation Device 0003

      Physical Slot: 5

      Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+

      Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

      Latency: 0

      Region 0: Memory at f2040000 (32-bit, non-prefetchable) [size=16K]

      Region 3: Memory at f2044000 (32-bit, non-prefetchable) [size=16K]

      Capabilities: [40] MSI-X: Enable+ Count=3 Masked-

      Vector table: BAR=3 offset=00000000

      PBA: BAR=3 offset=00002000

      Kernel driver in use: ixgbevf

      Kernel modules: ixgbevf

       

      Fedora 14 is used in both host and Guest.
      CPU : Intel xeon
      Hypervisor : QEMU/KVM
      VMM : Fedora 14 default
      Kernel Version : 2.6.35

        • 1. Re: Guest OS enumeration for VF PCIe device for 82599
          MARK HANNA
          MARK HANNA May 8, 2012 3:31 PM (in response to vaishali)

          I asked around about your post and heard back from a few developers.  In our labs MSI-X capability is displayed in the correct offset of [70]. We do not know why you are seeing something different. Of course there could be something differentwith QEMU or the guest on your setup.

           

          You might consider using RHEL 6.x. The tech brief at http://www.intel.com/content/www/us/en/network-adapters/10-gigabit-network-adapters/ethernet-x520-sr-iov-red-hat-tech-brief.html covers the configuration and this is a configuration that is known to work. Maybe going through the paper you can find something that will help you in your setup. On the other hand, if the issue is related to differences with any part of the distribution, then this paper might not help you.

           

          When you say, "the VF interrupt  is not functioning properly," what is happening? Is the VF able to pass traffic? Are you expecting a different number of interrupts to be used? Please explain further.

           

          What is in the output of the 'cat /proc/interrupts` for both the PF and VF?

           

          Mark H

          • 2. Re: Guest OS enumeration for VF PCIe device for 82599
            vaishali
            vaishali May 9, 2012 2:36 AM (in response to MARK HANNA)

            Thanks for the support i will try to load the RHEL 6.x  an d thanks for the document

            Mean time the information on interrupts

             

            Interrupts for PF

            [root@veena /]# lspci | grep 82599

            83:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit Network Connection (rev 01)

            83:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit Network Connection (rev 01)

            [root@veena /]# modprobe ixgbe max_vfs=1

            [root@veena /]# lspci | grep 82599

            83:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit Network Connection (rev 01)

            83:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit Network Connection (rev 01)

            83:10.0 Ethernet controller: Intel Corporation 82559 Ethernet Controller Virtual Function (rev 01)

            83:10.1 Ethernet controller: Intel Corporation 82559 Ethernet Controller Virtual Function (rev 01)

            [root@veena /]# cat /proc/interrupts

            115: 2 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth4-rx-0

            116: 2 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth4-tx-0

            117: 9 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth4:lsc

            118: 2 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth5-rx-0

            119: 2 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth5-tx-0

            120: 8 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth5:lsc

            121: 2 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth6-rx-0

            122: 2 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth6-tx-0

            123: 3 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth6:mbx

            124: 1 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth7-rx-0

            125: 1 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth7-tx-0

            126: 3 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth7:mbx

             

             

            I have assigned 83:10.0 device to the guest OS

             

            Interrupts on VF

            [root@vm1 VM1]# lspci | grep 82599

            00:06.0 Ethernet controller: Intel Corporation 82559 Ethernet Controller Virtual Function (rev 01)

            [root@vm1 VM1]# cat /proc/interrupts

            CPU0

            0: 278 IO-APIC-edge timer

            1: 199 IO-APIC-edge i8042

            2: 0 XT-PIC-XT cascade

            4: 2 IO-APIC-edge

            8: 0 IO-APIC-edge rtc0

            12: 130 IO-APIC-edge i8042

            14: 8818 IO-APIC-edge ata_piix

            15: 878 IO-APIC-edge ata_piix

            34: 459 IO-APIC-fasteoi virtio1

            35: 1104 IO-APIC-fasteoi uhci_hcd:usb1, Intel 82801AA-ICH

            40: 0 PCI-MSI-edge virtio0-config

            41: 8 PCI-MSI-edge virtio0-input

            42: 1 PCI-MSI-edge virtio0-output

            43: 0 PCI-MSI-edge eth1-rx-0

            44: 0 PCI-MSI-edge eth1-tx-0

            45: 0 PCI-MSI-edge eth1:mbx

            NMI: 0 Non-maskable interrupts

            LOC: 36375 Local timer interrupts

            SPU: 0 Spurious interrupts

            PMI: 0 Performance monitoring interrupts

            PND: 0 Performance pending work

            RES: 0 Rescheduling interrupts

            CAL: 0 Function call interrupts

            TLB: 0 TLB shootdowns

            TRM: 0 Thermal event interrupts

            THR: 0 Threshold APIC interrupts

            MCE: 0 Machine check exceptions

            MCP: 1 Machine check polls

            ERR: 0

            MIS: 0

            [root@vm1 VM1]# lspci

            00:06.0 Ethernet controller: Intel Corporation 82559 Ethernet Controller Virtual Function (rev 01)

            Subsystem: Intel Corporation Device 0003

            Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+

            Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

            Latency: 0

            Region 0: Memory at f2040000 (32-bit, non-prefetchable) [size=16K]

            Region 3: Memory at f2044000 (32-bit, non-prefetchable) [size=16K]

            Capabilities: [40] MSI-X: Enable+ Count=3 Masked-

            Vector table: BAR=3 offset=00000000

            PBA: BAR=3 offset=00002000

            Kernel driver in use: ixgbevf

            Kernel modules: ixgbevf

             

             

             

            • 3. Re: Guest OS enumeration for VF PCIe device for 82599
              Patrick_Kutch
              Patrick_Kutch May 18, 2012 4:02 PM (in response to vaishali)

              My SR-IOV Guru's believe this is likely a Fedora 14 problem.  We would like to suggest using the latest Fedora release, or try to reproduce the problem with RHEL 6.2.

               

              It takes a several components to make SR-IOV work, BIOS, platform,end-point device (the Intel 82599) and of course the OS.  There are lots and lots of patches to the linux kernel for SR-IOV support.  Sometimes not all of those patches make it into a distro release due to any number of reasons.

               

              Please update us on your efforts.

               

              thanks,

               

              Patrick

              • 4. Re: Guest OS enumeration for VF PCIe device for 82599
                vaishali
                vaishali May 21, 2012 11:33 PM (in response to Patrick_Kutch)

                Hi Patrick,

                I tried on RHEL 6.2 the previous issue was not there,new issue has been arrived.

                Configuration:

                1. CONFIG_DMAR_TABLE=y in kernel Config menu,
                2. intel_iommu=on in grub.config
                3. VT-d enabled in bios.

                Scenarios 1:

                • Loaded the ixgbe max_vfs=0, and started the testing.
                  Below are dmesg from the kernel.
                  DRHD: handling fault status reg 2
                  DMAR:[DMA Read] Request device [03:00.0] fault addr 619c00000 DMAR:[fault reason 01] Present bit in root entry is clear

                Scenarios 2:

                • loaded the ixgbe max_vfs=2, and started the testing.
                • Assigned the Vfs to the GuestOs.VFs are working fine.

                Didn't receive any DMAR error.

                 


                If intel_iommu=off or CONFIG_DMAR_TABLE=n,the above issue is not seen and ethernet driver is working fine(only PF driver ).But VFs cannot be assigned to the guest OS.

                 

                Can u please tell where it went worng ......................

                • 5. Re: Guest OS enumeration for VF PCIe device for 82599
                  Patrick_Kutch
                  Patrick_Kutch May 22, 2012 4:17 PM (in response to vaishali)

                  Thanks for trying that and reporting your findings.  It would appear to be some kind of DMAR issue - we are preparing to investigate.  It makes sense if there is a DMAR problem that VF's assigned to a VM would no work.

                   

                  To aid in the investigation, our engineers have asked if you are using the inbox (kernel) drivers, or the latest from sourceforge.

                   

                  thanks,

                  Patrick


                  • 6. Re: Guest OS enumeration for VF PCIe device for 82599
                    vaishali
                    vaishali May 23, 2012 11:41 PM (in response to Patrick_Kutch)

                    I am using the kernel version 3.2.13 rt23.

                    • 7. Re: Guest OS enumeration for VF PCIe device for 82599
                      Patrick_Kutch
                      Patrick_Kutch May 24, 2012 10:10 AM (in response to vaishali)

                      That is a very old driver.  I would recommend getting the latest versions (PF and VF) - which is available here:

                      http://sourceforge.net/projects/e1000/files/ixgbe%20stable/

                       

                      Please give this a try and let us know how it works.

                       

                      thanks,

                       

                      Patrick

                      • 8. Re: Guest OS enumeration for VF PCIe device for 82599
                        vaishali
                        vaishali May 25, 2012 12:03 AM (in response to Patrick_Kutch)

                        Hi Patrick,

                        Thanks for the support................

                        I will download and test the lastest driver and let u know  the result.

                         

                        Veena

                        • 9. Re: Guest OS enumeration for VF PCIe device for 82599
                          vaishali
                          vaishali Jun 13, 2012 10:54 PM (in response to vaishali)

                          It is working with the latest source code Thnaks for the suppot

                           

                           

                          I am curious to know what is reason for the DMAR issue????

                           

                          DRHD: handling fault status reg 2

                          DMAR:[DMA Read] Request device [03:00.0] fault addr 619c00000 DMAR:[fault reason 01] Present bit in root entry is clear

                          • 10. Re: Guest OS enumeration for VF PCIe device for 82599
                            Patrick_Kutch
                            Patrick_Kutch Jun 15, 2012 8:36 AM (in response to vaishali)

                            Great, I am happy it working and to be able to help.

                             

                            I do not know what is happening with this DMAR issue you have described.  We suspect it may be a BIOS issue, however I will pass along the information you provided to our SR-IOV experts and relay back anything they may offer by way of explaination.