3 Replies Latest reply on Jan 27, 2014 11:38 AM by Patrick_Kutch

    Some VFs are assigned three IRQs and some receive only two

    alex@zadarastorage.com

      Greetings all,

       

      We are using Ubunti-Precise 3.2.0-29-generic #46, with KVM version qemu-kvm-1.0 and libvirt 0.9.8. We are using Intel NIC 82599 with ixgbe driver 3.11.33 and in-tree ixgbevf driver 2.2.0-k. We are attaching VFs to KVM instances.

       

      In lspci we see that MSI-X table for each VF has three entries, like this:

      03:17.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

              Subsystem: Intel Corporation Device 7a11

              Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-

              Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

              Latency: 0

              Region 0: [virtual] Memory at df478000 (64-bit, non-prefetchable) [size=16K]

              Region 3: [virtual] Memory at df578000 (64-bit, non-prefetchable) [size=16K]

              Capabilities: [70] MSI-X: Enable+ Count=3 Masked-

       

      But when attaching the VF to KVM, we see that most of the VFs receive only two IRQs, while only a few receive three IRQs.

      For example:

       

      Dec 13 11:04:17 ubuntu-sata-31 kernel: [    4.448999] ixgbevf 0000:03:17.3: irq 359 for MSI/MSI-X

      Dec 13 11:04:17 ubuntu-sata-31 kernel: [    4.449016] ixgbevf 0000:03:17.3: irq 360 for MSI/MSI-X

      Dec 13 11:04:17 ubuntu-sata-31 kernel: [    4.449033] ixgbevf 0000:03:17.3: irq 361 for MSI/MSI-X

       

      Dec 15 10:23:32 ubuntu-sata-31 kernel: [170107.490908] pci-stub 0000:03:12.2: irq 138 for MSI/MSI-X

      Dec 15 10:23:32 ubuntu-sata-31 kernel: [170107.490924] pci-stub 0000:03:12.2: irq 139 for MSI/MSI-X

       

      Dec 15 10:23:32 ubuntu-sata-31 kernel: [170107.730559] pci-stub 0000:03:1c.1: irq 335 for MSI/MSI-X

      Dec 15 10:23:32 ubuntu-sata-31 kernel: [170107.730580] pci-stub 0000:03:1c.1: irq 336 for MSI/MSI-X

       

      Initial debugging shows, that KVM reads the PCI configuration space to determine the number of IRQs to assign:

       

      static int assigned_dev_update_msix_mmio(PCIDevice *pci_dev)

      {

          AssignedDevice *adev = DO_UPCAST(AssignedDevice, dev, pci_dev);

          uint16_t entries_nr = 0, entries_max_nr;

          int pos = 0, i, r = 0;

          uint32_t msg_addr, msg_upper_addr, msg_data, msg_ctrl;

          struct kvm_assigned_msix_nr msix_nr;

          struct kvm_assigned_msix_entry msix_entry;

          void *va = adev->msix_table_page;

       

          pos = pci_find_capability(pci_dev, PCI_CAP_ID_MSIX);

       

          entries_max_nr = *(uint16_t *)(pci_dev->config + pos + 2);

          entries_max_nr &= PCI_MSIX_FLAGS_QSIZE;

          entries_max_nr += 1;

       

      This yields entries_max_nr=3, as expected.

       

      But then:

       

          /* Get the usable entry number for allocating */

          for (i = 0; i < entries_max_nr; i++) {

              memcpy(&msg_ctrl, va + i * 16 + 12, 4);

              memcpy(&msg_data, va + i * 16 + 8, 4);

              /* Ignore unused entry even it's unmasked */

              if (msg_data == 0)

                  continue;

              entries_nr ++;

          }

       

      And adding some prints shows that the third entry yields msg_data==0, so it is skipped.

       

      ALEXL: PCIDEV(pci-assign) entries_max_nr=3

      ALEXL: PCIDEV(pci-assign) entry #0 msg_data=16817

      ALEXL: PCIDEV(pci-assign) entry #1 msg_data=16833

      ALEXL: PCIDEV(pci-assign) entry #2 msg_data=0

       

      ALEXL: PCIDEV(pci-assign) entries_max_nr=3

      ALEXL: PCIDEV(pci-assign) entry #0 msg_data=16849

      ALEXL: PCIDEV(pci-assign) entry #1 msg_data=16865

      ALEXL: PCIDEV(pci-assign) entry #2 msg_data=0

       

      ALEXL: PCIDEV(pci-assign) entries_max_nr=3

      ALEXL: PCIDEV(pci-assign) entry #0 msg_data=16674

      ALEXL: PCIDEV(pci-assign) entry #1 msg_data=16706

      ALEXL: PCIDEV(pci-assign) entry #2 msg_data=0

       

      We don't know what is the implication of having only two IRQs assigned instead of three. From overall perspective, those KVM instances function normally. Can anybody please comment on this?

       

      When using SR-IOV cards from another vendor, we see that those other cards have 4 entries for each VF, and all those entries have IRQs attached. So we see this issue only with Intel cards.

       

      Thanks,

      Alex.

        • 1. Re: Some VFs are assigned three IRQs and some receive only two
          Patrick_Kutch

          Hi Alex,

           

          Our guess is that you have a mis-aligned PF and VF version.  We did some modifications of how many interrupts we use per VF a while back and went from 3 to 2.  Can you try the latest and greatest drivers located at http://sourceforge.net/projects/e1000/files/ and let us know how it went?

           

          The latest IXGBE PF is 3.19.1 and the latest IXGBEvf drivers  is 2.12.1.

           

          thanx,

           

          Patrick

          • 2. Re: Some VFs are assigned three IRQs and some receive only two
            alex@zadarastorage.com

            Hi Patrick,

            with the latest drivers you mentioned, we still see that MSI-X table for each VF has three entries, but each VF receives only two IRQs:

             

            03:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

                    Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-

                    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

                    Latency: 0

                    Region 0: [virtual] Memory at d1300000 (64-bit, prefetchable) [size=16K]

                    Region 3: [virtual] Memory at d1200000 (64-bit, prefetchable) [size=16K]

                    Capabilities: [70] MSI-X: Enable+ Count=3 Masked-

             

            but still PFs and VFs receive two IRQs each:

            Jan 27 02:32:00 la-01-04 kernel: [4.887658] ixgbe 0000:03:00.1: (unregistered net_device): SR-IOV enabled with 32 VFs
            Jan 27 02:32:00 la-01-04 kernel: [4.911463] ixgbe 0000:03:00.1: irq 117 for MSI/MSI-X
            Jan 27 02:32:00 la-01-04 kernel: [4.911481] ixgbe 0000:03:00.1: irq 118 for MSI/MSI-X
            Jan 27 02:32:00 la-01-04 kernel: [4.913246] ixgbe 0000:03:00.1: PCI Express bandwidth of 32GT/s available
            Jan 27 02:32:00 la-01-04 kernel: [4.913251] ixgbe 0000:03:00.1: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
            Jan 27 02:32:00 la-01-04 kernel: [4.913335] ixgbe 0000:03:00.1: eth5: MAC: 2, PHY: 9, SFP+: 4, PBA No: FFFFFF-0FF
            Jan 27 02:32:00 la-01-04 kernel: [4.913339] ixgbe 0000:03:00.1: f8:4a:bf:56:ce:27
            Jan 27 02:32:00 la-01-04 kernel: [4.913342] ixgbe 0000:03:00.1: eth5: Enabled Features: RxQ: 1 TxQ: 1
            Jan 27 02:32:00 la-01-04 kernel: [4.913356] ixgbe 0000:03:00.1: eth5: Intel(R) 10 Gigabit Network Connection
            Jan 27 02:32:00 la-01-04 kernel: [4.913637] ixgbevf 0000:03:10.0: enabling device (0000 -> 0002)
            Jan 27 02:32:00 la-01-04 kernel: [4.913650] ixgbevf 0000:03:10.0: setting latency timer to 64
            Jan 27 02:32:00 la-01-04 kernel: [4.976594] ixgbevf 0000:03:10.0: PF still in reset state.  Is the PF interface up?
            Jan 27 02:32:00 la-01-04 kernel: [4.976600] ixgbevf 0000:03:10.0: Assigning random MAC address
            Jan 27 02:32:00 la-01-04 kernel: [4.976705] ixgbevf 0000:03:10.0: irq 119 for MSI/MSI-X
            Jan 27 02:32:00 la-01-04 kernel: [4.976722] ixgbevf 0000:03:10.0: irq 120 for MSI/MSI-X
            Jan 27 02:32:00 la-01-04 kernel: [4.976729] ixgbevf: eth%d: ixgbevf_init_interrupt_scheme: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1
            Jan 27 02:32:00 la-01-04 kernel: [4.977184] ixgbevf: eth6: ixgbevf_probe: Intel(R) 82599 Virtual Function
            Jan 27 02:32:00 la-01-04 kernel: [4.977188] a2:b2:a3:e9:8e:d3
            Jan 27 02:32:00 la-01-04 kernel: [4.977192] ixgbevf: eth6: ixgbevf_probe: GRO is enabled
            Jan 27 02:32:00 la-01-04 kernel: [4.977195] ixgbevf: eth6: ixgbevf_probe: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver
            Jan 27 02:32:00 la-01-04 kernel: [4.977348] ixgbevf 0000:03:10.2: enabling device (0000 -> 0002)
            Jan 27 02:32:00 la-01-04 kernel: [4.977362] ixgbevf 0000:03:10.2: setting latency timer to 64
            Jan 27 02:32:00 la-01-04 kernel: [5.040503] ixgbevf 0000:03:10.2: PF still in reset state.  Is the PF interface up?
            Jan 27 02:32:00 la-01-04 kernel: [5.040512] ixgbevf 0000:03:10.2: Assigning random MAC address
            Jan 27 02:32:00 la-01-04 kernel: [5.040632] ixgbevf 0000:03:10.2: irq 121 for MSI/MSI-X
            Jan 27 02:32:00 la-01-04 kernel: [5.040649] ixgbevf 0000:03:10.2: irq 122 for MSI/MSI-X
            Jan 27 02:32:00 la-01-04 kernel: [5.040656] ixgbevf: eth%d: ixgbevf_init_interrupt_scheme: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1
            Jan 27 02:32:00 la-01-04 kernel: [5.041162] ixgbevf: eth7: ixgbevf_probe: Intel(R) 82599 Virtual Function
            Jan 27 02:32:00 la-01-04 kernel: [5.041166] 8e:03:15:99:05:5e
            Jan 27 02:32:00 la-01-04 kernel: [5.041170] ixgbevf: eth7: ixgbevf_probe: GRO is enabled
            Jan 27 02:32:00 la-01-04 kernel: [5.041173] ixgbevf: eth7: ixgbevf_probe: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver

             

            Is a firmware upgrade perhaps needed to make MSIX table have two entries?

            • 3. Re: Some VFs are assigned three IRQs and some receive only two
              Patrick_Kutch

              We have recently added Mulit-Queue support to our SR-IOV solution.  Multi-Queue (MQ) uses 3 interrupts, non MQ only uses 2; however the code still requests 3.  I've asked engineering to look into this and see if we can only reqeust 2 if only going to use 2.

               

              To use MQ, and all 3 interrupts, when you compile both the PF and VF use CFLAGS_EXTRA="-DIXGBE_ENABLE_VF_MQ".

               

              Hope this helps,

               

              Patrick