6 Replies Latest reply on Jul 29, 2014 1:53 PM by calvin419

    82599ES SR-IOV on Ubuntu 14.04 - VF behavior

    calvin419

      Hi,

       

      This is my first visit/post here to this community so I hope I'm asking the right questions in the right place.

       

      I am using the 82599ES adapter with SR-IOV enabled under Ubuntu 14.04 under kernel 3.13.0-30 but this same behavior was witness under 3.13.0-24.  The corresponding VFs are created and visible in 'lspci' and 'ip link show'.  I am intending to pass VFs to guest OSs in KVM running libvirt 1.2.2 as 'hostdev' devices in the guest's XML configuration file.

       

      Once the VFs are created, I proceed to set VF parameters such as a MAC and VLAN ID.  I believed MACs were to be automatically generated by the driver but perhaps I'm wrong as it doesn't seem to be the case.  So in this particular case, I set the VF MAC to the auto generated MAC that libvirt assigns to the interface by editing the guest through virsh edit.

       

      root@ubuntu:~# ip link set eth4 up

      root@ubuntu:~# ip link set eth4 vf 0 mac 52:54:00:4d:8b:bc

      root@ubuntu:~# ip link set eth4 vf 0 vlan 100

      root@ubuntu:~# ip link set eth4 vf 1 mac 52:54:00:5e:3b:e2

      root@ubuntu:~# ip link set eth4 vf 1 vlan 200

      root@ubuntu:~# ip link set eth4 vf 2 mac 52:54:00:19:e3:f8

      root@ubuntu:~# ip link set eth4 vf 2 vlan 300

      root@ubuntu:~# ip link set eth4 vf 3 mac 52:54:00:1c:7e:79

      root@ubuntu:~# ip link set eth4 vf 3 vlan 100

      root@ubuntu:~# ip link set eth4 vf 4 mac 52:54:00:45:c9:7d

      root@ubuntu:~# ip link set eth4 vf 4 vlan 200

      root@ubuntu:~# ip link set eth4 vf 5 mac 52:54:00:32:13:0e

      root@ubuntu:~# ip link set eth4 vf 5 vlan 300

       

      This results in something like the following (dmesg output below also).

       

      root@ubuntu:~# ip link show dev eth4

      11: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

          link/ether 90:e2:ba:47:2c:30 brd ff:ff:ff:ff:ff:ff

          vf 0 MAC 52:54:00:4d:8b:bc, vlan 100, spoof checking on, link-state auto

          vf 1 MAC 52:54:00:5e:3b:e2, vlan 200, spoof checking on, link-state auto

          vf 2 MAC 52:54:00:19:e3:f8, vlan 300, spoof checking on, link-state auto

          vf 3 MAC 52:54:00:1c:7e:79, vlan 100, spoof checking on, link-state auto

          vf 4 MAC 52:54:00:45:c9:7d, vlan 200, spoof checking on, link-state auto

          vf 5 MAC 52:54:00:32:13:0e, vlan 300, spoof checking on, link-state auto

          vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 7 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 8 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 9 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 10 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 11 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 12 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 13 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 14 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 15 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

       

      Jul 16 10:29:21 ubuntu kernel: [ 8518.503869] ixgbe 0000:81:00.0: registered PHC device on eth4

      Jul 16 10:29:21 ubuntu kernel: [ 8518.608585] IPv6: ADDRCONF(NETDEV_UP): eth4: link is not ready

      Jul 16 10:29:21 ubuntu kernel: [ 8518.672821] ixgbe 0000:81:00.0 eth4: detected SFP+: 3

      Jul 16 10:29:21 ubuntu kernel: [ 8518.912379] ixgbe 0000:81:00.0 eth4: NIC Link is Up 10 Gbps, Flow Control: RX/TX

      Jul 16 10:29:21 ubuntu kernel: [ 8518.912536] IPv6: ADDRCONF(NETDEV_CHANGE): eth4: link becomes ready

      Jul 16 10:29:28 ubuntu kernel: [ 8526.350238] ixgbe 0000:81:00.0: setting MAC 52:54:00:4d:8b:bc on VF 0

      Jul 16 10:29:28 ubuntu kernel: [ 8526.350244] ixgbe 0000:81:00.0: Reload the VF driver to make this change effective.

      Jul 16 10:29:28 ubuntu kernel: [ 8526.363123] ixgbe 0000:81:00.0: Setting VLAN 100, QOS 0x0 on VF 0

      Jul 16 10:29:28 ubuntu kernel: [ 8526.387115] ixgbe 0000:81:00.0: setting MAC 52:54:00:5e:3b:e2 on VF 1

      Jul 16 10:29:28 ubuntu kernel: [ 8526.387120] ixgbe 0000:81:00.0: Reload the VF driver to make this change effective.

      Jul 16 10:29:28 ubuntu kernel: [ 8526.399143] ixgbe 0000:81:00.0: Setting VLAN 200, QOS 0x0 on VF 1

      Jul 16 10:29:28 ubuntu kernel: [ 8526.432961] ixgbe 0000:81:00.0: setting MAC 52:54:00:19:e3:f8 on VF 2

      Jul 16 10:29:28 ubuntu kernel: [ 8526.432967] ixgbe 0000:81:00.0: Reload the VF driver to make this change effective.

      Jul 16 10:29:28 ubuntu kernel: [ 8526.435878] ixgbe 0000:81:00.0: Setting VLAN 300, QOS 0x0 on VF 2

      Jul 16 10:29:29 ubuntu kernel: [ 8526.468947] ixgbe 0000:81:00.0: setting MAC 52:54:00:1c:7e:79 on VF 3

      Jul 16 10:29:29 ubuntu kernel: [ 8526.468953] ixgbe 0000:81:00.0: Reload the VF driver to make this change effective.

      Jul 16 10:29:29 ubuntu kernel: [ 8526.471825] ixgbe 0000:81:00.0: Setting VLAN 100, QOS 0x0 on VF 3

      Jul 16 10:29:29 ubuntu kernel: [ 8526.495211] ixgbe 0000:81:00.0: setting MAC 52:54:00:45:c9:7d on VF 4

      Jul 16 10:29:29 ubuntu kernel: [ 8526.495217] ixgbe 0000:81:00.0: Reload the VF driver to make this change effective.

      Jul 16 10:29:29 ubuntu kernel: [ 8526.561157] ixgbe 0000:81:00.0: Setting VLAN 200, QOS 0x0 on VF 4

      Jul 16 10:29:29 ubuntu kernel: [ 8526.564616] ixgbe 0000:81:00.0: setting MAC 52:54:00:32:13:0e on VF 5

      Jul 16 10:29:29 ubuntu kernel: [ 8526.564621] ixgbe 0000:81:00.0: Reload the VF driver to make this change effective.

      Jul 16 10:29:31 ubuntu kernel: [ 8528.628424] ixgbe 0000:81:00.0: Setting VLAN 300, QOS 0x0 on VF 5

       

      I am passing VF 0-2 to one guest OS and VF 3-5 to a second OS. 

       

      Guest1

      <interface type='hostdev' managed='yes'>
        <mac address='52:54:00:4d:8b:bc'/>
        <source>
          <address type='pci' domain='0x0000' bus='0x81' slot='0x10' function='0x0'/>
        </source>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
      </interface>
      <interface type='hostdev' managed='yes'>
        <mac address='52:54:00:5e:3b:e2'/>
        <source>
          <address type='pci' domain='0x0000' bus='0x81' slot='0x10' function='0x1'/>
        </source>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
      </interface>
      <interface type='hostdev' managed='yes'>
        <mac address='52:54:00:19:e3:f8'/>
        <source>
          <address type='pci' domain='0x0000' bus='0x81' slot='0x10' function='0x2'/>
        </source>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
      </interface>

       

      Guest2

      <interface type='hostdev' managed='yes'>
        <mac address='52:54:00:1c:7e:79'/>
        <source>
          <address type='pci' domain='0x0000' bus='0x81' slot='0x10' function='0x3'/>
        </source>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
      </interface>
      <interface type='hostdev' managed='yes'>
        <mac address='52:54:00:45:c9:7d'/>
        <source>
          <address type='pci' domain='0x0000' bus='0x81' slot='0x10' function='0x4'/>
        </source>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
      </interface>
      <interface type='hostdev' managed='yes'>
        <mac address='52:54:00:32:13:0e'/>
        <source>
          <address type='pci' domain='0x0000' bus='0x81' slot='0x10' function='0x5'/>
        </source>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
      </interface>


      And then I start the guests.  They boot normal and detect VFs I've passed through but this is where things get hokey.  The guest may sporadically have an all zeros MAC and sometimes not.  But most notably is this difference in the output of 'ip link show dev eth4' compared to before the guest was started.


      Before Guest Start

      root@ubuntu:~# ip link show dev eth4

      11: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

          link/ether 90:e2:ba:47:2c:30 brd ff:ff:ff:ff:ff:ff

          vf 0 MAC 52:54:00:4d:8b:bc, vlan 100, spoof checking on, link-state auto

          vf 1 MAC 52:54:00:5e:3b:e2, vlan 200, spoof checking on, link-state auto

          vf 2 MAC 52:54:00:19:e3:f8, vlan 300, spoof checking on, link-state auto

          vf 3 MAC 52:54:00:1c:7e:79, vlan 100, spoof checking on, link-state auto

          vf 4 MAC 52:54:00:45:c9:7d, vlan 200, spoof checking on, link-state auto

          vf 5 MAC 52:54:00:32:13:0e, vlan 300, spoof checking on, link-state auto

          vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 7 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 8 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 9 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 10 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 11 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 12 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 13 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 14 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 15 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

       

      After Guest Start

      root@ubuntu:~# ip link show dev eth4

      11: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

          link/ether 90:e2:ba:47:2c:30 brd ff:ff:ff:ff:ff:ff

          vf 0 MAC 52:54:00:4d:8b:bc, spoof checking on, link-state auto

          vf 1 MAC 52:54:00:19:e3:f8, spoof checking on, link-state auto

          vf 2 MAC 52:54:00:45:c9:7d, spoof checking on, link-state auto

          vf 3 MAC 52:54:00:1c:7e:79, vlan 100, spoof checking on, link-state auto

          vf 4 MAC 52:54:00:45:c9:7d, vlan 200, spoof checking on, link-state auto

          vf 5 MAC 52:54:00:32:13:0e, vlan 300, spoof checking on, link-state auto

          vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 7 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 8 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 9 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 10 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 11 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 12 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 13 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 14 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 15 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

       

      The VLAN ID is missing for VF0-2.  VF1 has VF2's original MAC and VF2's MAC has become VF4's MAC.

       

      Here is the resulting dmesg output from starting my guests.


      Jul 16 10:37:56 ubuntu kernel: [ 9034.298337] pci-stub 0000:81:10.0: claimed by stub

      Jul 16 10:37:56 ubuntu kernel: [ 9034.298359] pci-stub 0000:81:10.2: claimed by stub

      Jul 16 10:37:56 ubuntu kernel: [ 9034.298369] pci-stub 0000:81:10.4: claimed by stub

      Jul 16 10:37:56 ubuntu kernel: [ 9034.298378] pci-stub 0000:81:10.6: claimed by stub

      Jul 16 10:37:56 ubuntu kernel: [ 9034.298385] pci-stub 0000:81:11.0: claimed by stub

      Jul 16 10:37:56 ubuntu kernel: [ 9034.298411] pci-stub 0000:81:11.2: claimed by stub

      Jul 16 10:37:56 ubuntu kernel: [ 9034.298428] pci-stub 0000:81:11.4: claimed by stub

      Jul 16 10:37:56 ubuntu kernel: [ 9034.298439] pci-stub 0000:81:11.6: claimed by stub

      Jul 16 10:37:56 ubuntu kernel: [ 9034.298453] pci-stub 0000:81:12.0: claimed by stub

      Jul 16 10:37:56 ubuntu kernel: [ 9034.298466] pci-stub 0000:81:12.2: claimed by stub

      Jul 16 10:37:56 ubuntu kernel: [ 9034.298478] pci-stub 0000:81:12.4: claimed by stub

      Jul 16 10:37:56 ubuntu kernel: [ 9034.298489] pci-stub 0000:81:12.6: claimed by stub

      Jul 16 10:37:56 ubuntu kernel: [ 9034.298499] pci-stub 0000:81:13.0: claimed by stub

      Jul 16 10:37:56 ubuntu kernel: [ 9034.298509] pci-stub 0000:81:13.2: claimed by stub

      Jul 16 10:37:56 ubuntu kernel: [ 9034.298519] pci-stub 0000:81:13.4: claimed by stub

      Jul 16 10:37:56 ubuntu kernel: [ 9034.298528] pci-stub 0000:81:13.6: claimed by stub

      Jul 16 10:37:56 ubuntu kernel: [ 9034.298539] pci-stub 0000:81:10.1: claimed by stub

      Jul 16 10:37:56 ubuntu kernel: [ 9034.298550] pci-stub 0000:81:10.3: claimed by stub

      Jul 16 10:37:56 ubuntu kernel: [ 9034.298561] pci-stub 0000:81:10.5: claimed by stub

      Jul 16 10:37:56 ubuntu kernel: [ 9034.298571] pci-stub 0000:81:10.7: claimed by stub

      Jul 16 10:37:56 ubuntu kernel: [ 9034.298580] pci-stub 0000:81:11.1: claimed by stub

      Jul 16 10:37:56 ubuntu kernel: [ 9034.298591] pci-stub 0000:81:11.3: claimed by stub

      Jul 16 10:37:56 ubuntu kernel: [ 9034.298601] pci-stub 0000:81:11.5: claimed by stub

      Jul 16 10:37:56 ubuntu kernel: [ 9034.298611] pci-stub 0000:81:11.7: claimed by stub

      Jul 16 10:37:56 ubuntu kernel: [ 9034.298621] pci-stub 0000:81:12.1: claimed by stub

      Jul 16 10:37:56 ubuntu kernel: [ 9034.298633] pci-stub 0000:81:12.3: claimed by stub

      Jul 16 10:37:56 ubuntu kernel: [ 9034.298643] pci-stub 0000:81:12.5: claimed by stub

      Jul 16 10:37:56 ubuntu kernel: [ 9034.298653] pci-stub 0000:81:12.7: claimed by stub

      Jul 16 10:37:56 ubuntu kernel: [ 9034.298662] pci-stub 0000:81:13.1: claimed by stub

      Jul 16 10:37:56 ubuntu kernel: [ 9034.298672] pci-stub 0000:81:13.3: claimed by stub

      Jul 16 10:37:56 ubuntu kernel: [ 9034.298683] pci-stub 0000:81:13.5: claimed by stub

      Jul 16 10:37:56 ubuntu kernel: [ 9034.298693] pci-stub 0000:81:13.7: claimed by stub

      Jul 16 10:37:56 ubuntu kernel: [ 9034.324752] ixgbe 0000:81:00.0: setting MAC 52:54:00:4d:8b:bc on VF 0

      Jul 16 10:37:56 ubuntu kernel: [ 9034.324759] ixgbe 0000:81:00.0: Reload the VF driver to make this change effective.

      Jul 16 10:37:56 ubuntu kernel: [ 9034.344723] ixgbe 0000:81:00.1: setting MAC 52:54:00:5e:3b:e2 on VF 0

      Jul 16 10:37:56 ubuntu kernel: [ 9034.344730] ixgbe 0000:81:00.1: Reload the VF driver to make this change effective.

      Jul 16 10:37:56 ubuntu kernel: [ 9034.344733] ixgbe 0000:81:00.1: The VF MAC address has been set, but the PF device is not up.

      Jul 16 10:37:56 ubuntu kernel: [ 9034.344736] ixgbe 0000:81:00.1: Bring the PF device up before attempting to use the VF device.

      Jul 16 10:37:56 ubuntu kernel: [ 9034.360702] ixgbe 0000:81:00.0: setting MAC 52:54:00:19:e3:f8 on VF 1

      Jul 16 10:37:56 ubuntu kernel: [ 9034.360708] ixgbe 0000:81:00.0: Reload the VF driver to make this change effective.

      Jul 16 10:37:56 ubuntu kernel: [ 9034.665913] audit_printk_skb: 69 callbacks suppressed

      Jul 16 10:37:56 ubuntu kernel: [ 9034.665915] type=1400 audit(1405532276.635:35): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-b1ce4fac-83d7-3f00-8127-07ab6f1e15e3" pid=2779 comm="apparmor_parser"

      Jul 16 10:37:56 ubuntu kernel: [ 9034.880720] pci-stub 0000:81:10.0: enabling device (0000 -> 0002)

      Jul 16 10:37:58 ubuntu kernel: [ 9036.454955] pci-stub 0000:81:10.0: kvm assign device

      Jul 16 10:37:58 ubuntu kernel: [ 9036.456580] pci-stub 0000:81:10.1: enabling device (0000 -> 0002)

      Jul 16 10:37:58 ubuntu kernel: [ 9036.558625] pci-stub 0000:81:10.1: kvm assign device

      Jul 16 10:37:58 ubuntu kernel: [ 9036.559693] pci-stub 0000:81:10.2: enabling device (0000 -> 0002)

      Jul 16 10:37:58 ubuntu kernel: [ 9036.662765] pci-stub 0000:81:10.2: kvm assign device

      Jul 16 10:38:22 ubuntu kernel: [ 9060.288671] kvm: zapping shadow pages for mmio generation wraparound

      Jul 16 10:38:22 ubuntu kernel: [ 9060.882172] ixgbe 0000:81:00.0 eth4: VF Reset msg received from vf 0

      Jul 16 10:38:22 ubuntu kernel: [ 9060.893544] pci-stub 0000:81:10.0: irq 193 for MSI/MSI-X

      Jul 16 10:38:22 ubuntu kernel: [ 9060.921582] pci-stub 0000:81:10.0: irq 193 for MSI/MSI-X

      Jul 16 10:38:22 ubuntu kernel: [ 9060.921601] pci-stub 0000:81:10.0: irq 194 for MSI/MSI-X

      Jul 16 10:38:22 ubuntu kernel: [ 9060.945515] pci-stub 0000:81:10.0: irq 193 for MSI/MSI-X

      Jul 16 10:38:22 ubuntu kernel: [ 9060.945528] pci-stub 0000:81:10.0: irq 194 for MSI/MSI-X

      Jul 16 10:38:23 ubuntu kernel: [ 9061.053636] pci-stub 0000:81:10.1: irq 195 for MSI/MSI-X

      Jul 16 10:38:23 ubuntu kernel: [ 9061.073718] pci-stub 0000:81:10.1: irq 195 for MSI/MSI-X

      Jul 16 10:38:23 ubuntu kernel: [ 9061.073732] pci-stub 0000:81:10.1: irq 196 for MSI/MSI-X

      Jul 16 10:38:23 ubuntu kernel: [ 9061.101789] pci-stub 0000:81:10.1: irq 195 for MSI/MSI-X

      Jul 16 10:38:23 ubuntu kernel: [ 9061.101803] pci-stub 0000:81:10.1: irq 196 for MSI/MSI-X

      Jul 16 10:38:23 ubuntu kernel: [ 9061.187991] ixgbe 0000:81:00.0 eth4: VF Reset msg received from vf 1

      Jul 16 10:38:23 ubuntu kernel: [ 9061.201800] pci-stub 0000:81:10.2: irq 197 for MSI/MSI-X

      Jul 16 10:38:23 ubuntu kernel: [ 9061.225840] pci-stub 0000:81:10.2: irq 197 for MSI/MSI-X

      Jul 16 10:38:23 ubuntu kernel: [ 9061.225863] pci-stub 0000:81:10.2: irq 198 for MSI/MSI-X

      Jul 16 10:38:23 ubuntu kernel: [ 9061.261958] pci-stub 0000:81:10.2: irq 197 for MSI/MSI-X

      Jul 16 10:38:23 ubuntu kernel: [ 9061.261972] pci-stub 0000:81:10.2: irq 198 for MSI/MSI-X

      Jul 16 10:38:31 ubuntu kernel: [ 9069.898230] ixgbe 0000:81:00.0 eth4: VF Reset msg received from vf 0

      Jul 16 10:38:42 ubuntu kernel: [ 9080.567505] ixgbe 0000:81:00.1: setting MAC 52:54:00:1c:7e:79 on VF 1

      Jul 16 10:38:42 ubuntu kernel: [ 9080.567510] ixgbe 0000:81:00.1: Reload the VF driver to make this change effective.

      Jul 16 10:38:42 ubuntu kernel: [ 9080.567512] ixgbe 0000:81:00.1: The VF MAC address has been set, but the PF device is not up.

      Jul 16 10:38:42 ubuntu kernel: [ 9080.567513] ixgbe 0000:81:00.1: Bring the PF device up before attempting to use the VF device.

      Jul 16 10:38:42 ubuntu kernel: [ 9080.583695] ixgbe 0000:81:00.0: setting MAC 52:54:00:45:c9:7d on VF 2

      Jul 16 10:38:42 ubuntu kernel: [ 9080.583702] ixgbe 0000:81:00.0: Reload the VF driver to make this change effective.

      Jul 16 10:38:42 ubuntu kernel: [ 9080.595660] ixgbe 0000:81:00.1: setting MAC 52:54:00:32:13:0e on VF 2

      Jul 16 10:38:42 ubuntu kernel: [ 9080.595666] ixgbe 0000:81:00.1: Reload the VF driver to make this change effective.

      Jul 16 10:38:42 ubuntu kernel: [ 9080.595669] ixgbe 0000:81:00.1: The VF MAC address has been set, but the PF device is not up.

      Jul 16 10:38:42 ubuntu kernel: [ 9080.595672] ixgbe 0000:81:00.1: Bring the PF device up before attempting to use the VF device.

      Jul 16 10:38:42 ubuntu kernel: [ 9080.866600] type=1400 audit(1405532322.783:36): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-ed80fbe8-a135-6d45-1457-5d9d00f1d5ce" pid=2805 comm="apparmor_parser"

      Jul 16 10:38:42 ubuntu kernel: [ 9081.052820] pci-stub 0000:81:10.3: enabling device (0000 -> 0002)

      Jul 16 10:38:44 ubuntu kernel: [ 9082.503332] pci-stub 0000:81:10.3: kvm assign device

      Jul 16 10:38:44 ubuntu kernel: [ 9082.504917] pci-stub 0000:81:10.4: enabling device (0000 -> 0002)

      Jul 16 10:38:44 ubuntu kernel: [ 9082.605369] pci-stub 0000:81:10.4: kvm assign device

      Jul 16 10:38:44 ubuntu kernel: [ 9082.606482] pci-stub 0000:81:10.5: enabling device (0000 -> 0002)

      Jul 16 10:38:44 ubuntu kernel: [ 9082.709469] pci-stub 0000:81:10.5: kvm assign device

      Jul 16 10:39:08 ubuntu kernel: [ 9106.462615] kvm: zapping shadow pages for mmio generation wraparound

      Jul 16 10:39:08 ubuntu kernel: [ 9107.064370] pci-stub 0000:81:10.3: irq 199 for MSI/MSI-X

      Jul 16 10:39:08 ubuntu kernel: [ 9107.088349] pci-stub 0000:81:10.3: irq 199 for MSI/MSI-X

      Jul 16 10:39:08 ubuntu kernel: [ 9107.088364] pci-stub 0000:81:10.3: irq 200 for MSI/MSI-X

      Jul 16 10:39:09 ubuntu kernel: [ 9107.108372] pci-stub 0000:81:10.3: irq 199 for MSI/MSI-X

      Jul 16 10:39:09 ubuntu kernel: [ 9107.108387] pci-stub 0000:81:10.3: irq 200 for MSI/MSI-X

      Jul 16 10:39:09 ubuntu kernel: [ 9107.209645] ixgbe 0000:81:00.0 eth4: VF Reset msg received from vf 2

      Jul 16 10:39:09 ubuntu kernel: [ 9107.224472] pci-stub 0000:81:10.4: irq 201 for MSI/MSI-X

      Jul 16 10:39:09 ubuntu kernel: [ 9107.252548] pci-stub 0000:81:10.4: irq 201 for MSI/MSI-X

      Jul 16 10:39:09 ubuntu kernel: [ 9107.252563] pci-stub 0000:81:10.4: irq 202 for MSI/MSI-X

      Jul 16 10:39:09 ubuntu kernel: [ 9107.272562] pci-stub 0000:81:10.4: irq 201 for MSI/MSI-X

      Jul 16 10:39:09 ubuntu kernel: [ 9107.272578] pci-stub 0000:81:10.4: irq 202 for MSI/MSI-X

      Jul 16 10:39:09 ubuntu kernel: [ 9107.364623] pci-stub 0000:81:10.5: irq 203 for MSI/MSI-X

      Jul 16 10:39:09 ubuntu kernel: [ 9107.384680] pci-stub 0000:81:10.5: irq 203 for MSI/MSI-X

      Jul 16 10:39:09 ubuntu kernel: [ 9107.384695] pci-stub 0000:81:10.5: irq 204 for MSI/MSI-X

      Jul 16 10:39:09 ubuntu kernel: [ 9107.404698] pci-stub 0000:81:10.5: irq 203 for MSI/MSI-X

      Jul 16 10:39:09 ubuntu kernel: [ 9107.404713] pci-stub 0000:81:10.5: irq 204 for MSI/MSI-X

      Jul 16 10:39:28 ubuntu kernel: [ 9126.991794] ixgbe 0000:81:00.0 eth4: VF Reset msg received from vf 2

       

      root@ubuntu:~# cat /etc/modprobe.d/ixgbe.conf

      options ixgbe max_vfs=16

      options loop max_loop=128

      blacklist ixgbevf

       

      I removed the module and re-added to provide the dmesg output here.

       

      root@ubuntu:~# modprobe ixgbe

      Jul 16 10:20:31 ubuntu kernel: [ 7988.720851] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 3.15.1-k

      Jul 16 10:20:31 ubuntu kernel: [ 7988.720856] ixgbe: Copyright (c) 1999-2013 Intel Corporation.

      Jul 16 10:20:31 ubuntu kernel: [ 7988.721376] ixgbe 0000:81:00.0: Enabling SR-IOV VFs using the max_vfs module parameter is deprecated - please use the pci sysfs interface instead.

      Jul 16 10:20:32 ubuntu kernel: [ 7988.952543] pci 0000:81:10.0: [8086:10ed] type 00 class 0x020000

      Jul 16 10:20:32 ubuntu kernel: [ 7988.952824] pci 0000:81:10.2: [8086:10ed] type 00 class 0x020000

      Jul 16 10:20:32 ubuntu kernel: [ 7988.953029] pci 0000:81:10.4: [8086:10ed] type 00 class 0x020000

      Jul 16 10:20:32 ubuntu kernel: [ 7988.953200] pci 0000:81:10.6: [8086:10ed] type 00 class 0x020000

      Jul 16 10:20:32 ubuntu kernel: [ 7988.953367] pci 0000:81:11.0: [8086:10ed] type 00 class 0x020000

      Jul 16 10:20:32 ubuntu kernel: [ 7988.953533] pci 0000:81:11.2: [8086:10ed] type 00 class 0x020000

      Jul 16 10:20:32 ubuntu kernel: [ 7988.953698] pci 0000:81:11.4: [8086:10ed] type 00 class 0x020000

      Jul 16 10:20:32 ubuntu kernel: [ 7988.953870] pci 0000:81:11.6: [8086:10ed] type 00 class 0x020000

      Jul 16 10:20:32 ubuntu kernel: [ 7988.954039] pci 0000:81:12.0: [8086:10ed] type 00 class 0x020000

      Jul 16 10:20:32 ubuntu kernel: [ 7988.954214] pci 0000:81:12.2: [8086:10ed] type 00 class 0x020000

      Jul 16 10:20:32 ubuntu kernel: [ 7988.954400] pci 0000:81:12.4: [8086:10ed] type 00 class 0x020000

      Jul 16 10:20:32 ubuntu kernel: [ 7988.954629] pci 0000:81:12.6: [8086:10ed] type 00 class 0x020000

      Jul 16 10:20:32 ubuntu kernel: [ 7988.954805] pci 0000:81:13.0: [8086:10ed] type 00 class 0x020000

      Jul 16 10:20:32 ubuntu kernel: [ 7988.954976] pci 0000:81:13.2: [8086:10ed] type 00 class 0x020000

      Jul 16 10:20:32 ubuntu kernel: [ 7988.955148] pci 0000:81:13.4: [8086:10ed] type 00 class 0x020000

      Jul 16 10:20:32 ubuntu kernel: [ 7988.955400] pci 0000:81:13.6: [8086:10ed] type 00 class 0x020000

      Jul 16 10:20:32 ubuntu kernel: [ 7988.955842] ixgbe 0000:81:00.0 (unregistered net_device): SR-IOV enabled with 16 VFs

      Jul 16 10:20:32 ubuntu kernel: [ 7988.979951] ixgbe 0000:81:00.0: irq 173 for MSI/MSI-X

      Jul 16 10:20:32 ubuntu kernel: [ 7988.979989] ixgbe 0000:81:00.0: irq 174 for MSI/MSI-X

      Jul 16 10:20:32 ubuntu kernel: [ 7988.980011] ixgbe 0000:81:00.0: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1

      Jul 16 10:20:32 ubuntu kernel: [ 7988.980144] ixgbe 0000:81:00.0: PCI Express bandwidth of 32GT/s available

      Jul 16 10:20:32 ubuntu kernel: [ 7988.980148] ixgbe 0000:81:00.0: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)

      Jul 16 10:20:32 ubuntu kernel: [ 7988.980234] ixgbe 0000:81:00.0: MAC: 2, PHY: 9, SFP+: 3, PBA No: E66560-006

      Jul 16 10:20:32 ubuntu kernel: [ 7988.980237] ixgbe 0000:81:00.0: 90:e2:ba:47:2c:30

      Jul 16 10:20:32 ubuntu kernel: [ 7988.982095] ixgbe 0000:81:00.0 eth4: IOV is enabled with 16 VFs

      Jul 16 10:20:32 ubuntu kernel: [ 7988.982253] ixgbe 0000:81:00.0: Intel(R) 10 Gigabit Network Connection

      Jul 16 10:20:32 ubuntu kernel: [ 7988.982599] ixgbe 0000:81:00.1: Enabling SR-IOV VFs using the max_vfs module parameter is deprecated - please use the pci sysfs interface instead.

      Jul 16 10:20:32 ubuntu kernel: [ 7989.212846] pci 0000:81:10.1: [8086:10ed] type 00 class 0x020000

      Jul 16 10:20:32 ubuntu kernel: [ 7989.213062] pci 0000:81:10.3: [8086:10ed] type 00 class 0x020000

      Jul 16 10:20:32 ubuntu kernel: [ 7989.213161] pci 0000:81:10.5: [8086:10ed] type 00 class 0x020000

      Jul 16 10:20:32 ubuntu kernel: [ 7989.213257] pci 0000:81:10.7: [8086:10ed] type 00 class 0x020000

      Jul 16 10:20:32 ubuntu kernel: [ 7989.213354] pci 0000:81:11.1: [8086:10ed] type 00 class 0x020000

      Jul 16 10:20:32 ubuntu kernel: [ 7989.213451] pci 0000:81:11.3: [8086:10ed] type 00 class 0x020000

      Jul 16 10:20:32 ubuntu kernel: [ 7989.213547] pci 0000:81:11.5: [8086:10ed] type 00 class 0x020000

      Jul 16 10:20:32 ubuntu kernel: [ 7989.213713] pci 0000:81:11.7: [8086:10ed] type 00 class 0x020000

      Jul 16 10:20:32 ubuntu kernel: [ 7989.214277] pci 0000:81:12.1: [8086:10ed] type 00 class 0x020000

      Jul 16 10:20:32 ubuntu kernel: [ 7989.214454] pci 0000:81:12.3: [8086:10ed] type 00 class 0x020000

      Jul 16 10:20:32 ubuntu kernel: [ 7989.214607] pci 0000:81:12.5: [8086:10ed] type 00 class 0x020000

      Jul 16 10:20:32 ubuntu kernel: [ 7989.214740] pci 0000:81:12.7: [8086:10ed] type 00 class 0x020000

      Jul 16 10:20:32 ubuntu kernel: [ 7989.214924] pci 0000:81:13.1: [8086:10ed] type 00 class 0x020000

      Jul 16 10:20:32 ubuntu kernel: [ 7989.215088] pci 0000:81:13.3: [8086:10ed] type 00 class 0x020000

      Jul 16 10:20:32 ubuntu kernel: [ 7989.215222] pci 0000:81:13.5: [8086:10ed] type 00 class 0x020000

      Jul 16 10:20:32 ubuntu kernel: [ 7989.215398] pci 0000:81:13.7: [8086:10ed] type 00 class 0x020000

      Jul 16 10:20:32 ubuntu kernel: [ 7989.215567] ixgbe 0000:81:00.1 (unregistered net_device): SR-IOV enabled with 16 VFs

      Jul 16 10:20:32 ubuntu kernel: [ 7989.239728] ixgbe 0000:81:00.1: irq 175 for MSI/MSI-X

      Jul 16 10:20:32 ubuntu kernel: [ 7989.239767] ixgbe 0000:81:00.1: irq 176 for MSI/MSI-X

      Jul 16 10:20:32 ubuntu kernel: [ 7989.239793] ixgbe 0000:81:00.1: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1

      Jul 16 10:20:32 ubuntu kernel: [ 7989.239930] ixgbe 0000:81:00.1: PCI Express bandwidth of 32GT/s available

      Jul 16 10:20:32 ubuntu kernel: [ 7989.239934] ixgbe 0000:81:00.1: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)

      Jul 16 10:20:32 ubuntu kernel: [ 7989.240020] ixgbe 0000:81:00.1: MAC: 2, PHY: 9, SFP+: 4, PBA No: E66560-006

      Jul 16 10:20:32 ubuntu kernel: [ 7989.240023] ixgbe 0000:81:00.1: 90:e2:ba:47:2c:31

      Jul 16 10:20:32 ubuntu kernel: [ 7989.242601] ixgbe 0000:81:00.1 eth5: IOV is enabled with 16 VFs

      Jul 16 10:20:32 ubuntu kernel: [ 7989.242852] ixgbe 0000:81:00.1: Intel(R) 10 Gigabit Network Connection

       

      root@ubuntu:~# lspci | grep 82599

      81:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

      81:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

      81:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      81:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      81:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      81:10.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      81:10.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      81:10.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      81:10.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      81:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      81:11.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      81:11.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      81:11.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      81:11.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      81:11.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      81:11.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      81:11.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      81:11.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      81:12.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      81:12.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      81:12.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      81:12.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      81:12.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      81:12.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      81:12.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      81:12.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      81:13.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      81:13.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      81:13.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      81:13.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      81:13.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      81:13.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      81:13.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      81:13.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)


      A few additional notes on what I've done.  I've taken the kernel as mentioned before from 3.13.0-24 to 3.13.0-30.


      I am using the packaged driver 3.15.1-k in Ubuntu 14.04 as it appears there is issues with compiling the latest driver on 14.04 according to bug #415 and #421, so I have been unsucesfull there.


      I've upgrade iproute2 from 3.12.0-2 to 3.15.0

       

      I've performed SR-IOV using another vendor's card under Ubuntu 14.04 and same versions of libvirt and iproute2 and didn't experience this behavior so for that reason I've started here.

       

      Any help or ideas on why things appear to get scrabled is appreciated. 

       

      Thanks!

       

      JC

        • 1. Re: 82599ES SR-IOV on Ubuntu 14.04 - VF behavior
          Patrick_Kutch

          Thanx for visiting.  Sorry for the delayed response - was on vacation :-)

           

          I believe the issues is that you are administratively setting the MAC address for the VF's.  When  you do this, we make the assumption that the administrator does not trust the VMM or the VM to configure the MAC address, all such requests will be rejected.

           

          In general what usually happens is you create a bunch of VF's, your VMM  (such as Virt-Manager) will itself assign a MAC address to the VF when it get assigned to a VM and everything is fine - you should even be able to change the MAC from the VM.  Though you may need to turn of anti-spoofing.

           

          My expert came back witht the following that I am pasting for your reading pleasure.

           

            Recommended setup steps are:

           

          1. Blacklist VF driver if VFs are intended to be assigned toVM(s). 
          2. Load ixgbe driver with desired number of  VFs. 
          3. DO NOT assign MAC address at this time.  Assigning MAC address to VF from the host means that administrator doesn’t trust the VM and VM will not be able to change VF’s MAC address.  
          4. Assign Port VLAN for each VF using IPROUTE2 utility. 
          5. Assign the VF to VM using Linux Virt-Manager or creating/modifying VM configuration XML file. 
          6. Bring up VM to confirm connectivity. 

           

          We  have tested this setup and it works using Ubuntu 14.04 host and guests.   

           

          We believe Linux Virt-Manager and  KVM are getting confused because user is assigning the MAC addresses from within the host.  For Intel 10Gb network drivers this is called an administrative assignment and is a security feature. 

           

           

          Hope this helps!

           

          - Patrick

          • 2. Re: 82599ES SR-IOV on Ubuntu 14.04 - VF behavior
            calvin419

            Hi Patrick,

             

            Thanks so much for your reply.  I hope you enjoyed your vacation time and I'm really sorry you have to deal with me upon your return!

             

            So, I wouldn't normally assign MAC addresses to the VFs, I only went that route because of the random behavior of the VFs after starting the VM that the hostdev interfaces were assigned to.

             

            For this reply, I changed my guest from FreeBSD to CentOS to perhaps simplify things but what I was originally seeing and the reason I even set them was because I was arbitrarily getting an all zeros MAC randomly on the VMs interfaces.

             

            First, in response to your recommended steps.

             

            1)  ixgbevf is being blacklisted from within /etc/modprobe.d/blacklist.conf

                      blacklist ixgbevf

            2)  I am loading ixgbe and VFs from within /etc/modprobe.d/ixgbe.conf

                      options ixgbe max_vfs=16

                      options loop max_loop=128

            3)  Now begging with my new CentOS VM, I have not assigned MAC addresses to the VFs.

                 root@ubuntu:~# ip link show dev eth4

                 5: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000

                link/ether 90:e2:ba:47:2c:30 brd ff:ff:ff:ff:ff:ff

                vf 0 MAC 00:00:00:00:00:00, vlan 100, spoof checking on, link-state auto

                vf 1 MAC 00:00:00:00:00:00, vlan 200, spoof checking on, link-state auto

                vf 2 MAC 00:00:00:00:00:00, vlan 300, spoof checking on, link-state auto

                vf 3 MAC 00:00:00:00:00:00, vlan 100, spoof checking on, link-state auto

                vf 4 MAC 00:00:00:00:00:00, vlan 200, spoof checking on, link-state auto

                vf 5 MAC 00:00:00:00:00:00, vlan 300, spoof checking on, link-state auto

                vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

                vf 7 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

                vf 8 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

                vf 9 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

                vf 10 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

                vf 11 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

                vf 12 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

                vf 13 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

                vf 14 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

                vf 15 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

             

            4)  VLANs are assigned (see output from 3)

             

            5)  Here is the applicable snippet from my guest's XML configuration

            <interface type='hostdev' managed='yes'>
              <mac address='52:54:00:58:5f:0c'/>
              <source>
                <address type='pci' domain='0x0000' bus='0x81' slot='0x10' function='0x0'/>
              </source>
              <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
            </interface>
            <interface type='hostdev' managed='yes'>
              <mac address='52:54:00:d4:63:a7'/>
              <source>
                <address type='pci' domain='0x0000' bus='0x81' slot='0x10' function='0x1'/>
              </source>
              <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
            </interface>
            <interface type='hostdev' managed='yes'>
              <mac address='52:54:00:51:5c:0a'/>
              <source>
                <address type='pci' domain='0x0000' bus='0x81' slot='0x10' function='0x2'/>
              </source>
              <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
            </interface>

             

            6)  And the result from starting the guest which still reflects bizarre behavior.  Notice from the log output below that a MAC was set to VF0 two times, a MAC was set for VF1 but VF2 was never set.

             

                 root@ubuntu:~# ip link show dev eth4

                 5: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000

                link/ether 90:e2:ba:47:2c:30 brd ff:ff:ff:ff:ff:ff

                vf 0 MAC 52:54:00:58:5f:0c, spoof checking on, link-state auto

                vf 1 MAC 52:54:00:51:5c:0a, spoof checking on, link-state auto

                vf 2 MAC 00:00:00:00:00:00, vlan 300, spoof checking on, link-state auto

                vf 3 MAC 00:00:00:00:00:00, vlan 100, spoof checking on, link-state auto

                vf 4 MAC 00:00:00:00:00:00, vlan 200, spoof checking on, link-state auto

                vf 5 MAC 00:00:00:00:00:00, vlan 300, spoof checking on, link-state auto

                vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

                vf 7 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

                vf 8 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

                vf 9 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

                vf 10 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

                vf 11 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

                vf 12 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

                vf 13 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

                vf 14 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

                vf 15 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

             

            Jul 22 12:53:40 ubuntu kernel: [ 3138.012846] pci-stub 0000:81:10.0: claimed by stub

            Jul 22 12:53:40 ubuntu kernel: [ 3138.012936] pci-stub 0000:81:10.2: claimed by stub

            Jul 22 12:53:40 ubuntu kernel: [ 3138.012951] pci-stub 0000:81:10.4: claimed by stub

            Jul 22 12:53:40 ubuntu kernel: [ 3138.012972] pci-stub 0000:81:10.6: claimed by stub

            Jul 22 12:53:40 ubuntu kernel: [ 3138.012988] pci-stub 0000:81:11.0: claimed by stub

            Jul 22 12:53:40 ubuntu kernel: [ 3138.013002] pci-stub 0000:81:11.2: claimed by stub

            Jul 22 12:53:40 ubuntu kernel: [ 3138.013017] pci-stub 0000:81:11.4: claimed by stub

            Jul 22 12:53:40 ubuntu kernel: [ 3138.013031] pci-stub 0000:81:11.6: claimed by stub

            Jul 22 12:53:40 ubuntu kernel: [ 3138.013045] pci-stub 0000:81:12.0: claimed by stub

            Jul 22 12:53:40 ubuntu kernel: [ 3138.013057] pci-stub 0000:81:12.2: claimed by stub

            Jul 22 12:53:40 ubuntu kernel: [ 3138.013068] pci-stub 0000:81:12.4: claimed by stub

            Jul 22 12:53:40 ubuntu kernel: [ 3138.013080] pci-stub 0000:81:12.6: claimed by stub

            Jul 22 12:53:40 ubuntu kernel: [ 3138.013091] pci-stub 0000:81:13.0: claimed by stub

            Jul 22 12:53:40 ubuntu kernel: [ 3138.013102] pci-stub 0000:81:13.2: claimed by stub

            Jul 22 12:53:40 ubuntu kernel: [ 3138.013113] pci-stub 0000:81:13.4: claimed by stub

            Jul 22 12:53:40 ubuntu kernel: [ 3138.013125] pci-stub 0000:81:13.6: claimed by stub

            Jul 22 12:53:40 ubuntu kernel: [ 3138.013136] pci-stub 0000:81:10.1: claimed by stub

            Jul 22 12:53:40 ubuntu kernel: [ 3138.013149] pci-stub 0000:81:10.3: claimed by stub

            Jul 22 12:53:40 ubuntu kernel: [ 3138.013166] pci-stub 0000:81:10.5: claimed by stub

            Jul 22 12:53:40 ubuntu kernel: [ 3138.013178] pci-stub 0000:81:10.7: claimed by stub

            Jul 22 12:53:40 ubuntu kernel: [ 3138.013190] pci-stub 0000:81:11.1: claimed by stub

            Jul 22 12:53:40 ubuntu kernel: [ 3138.013201] pci-stub 0000:81:11.3: claimed by stub

            Jul 22 12:53:40 ubuntu kernel: [ 3138.013213] pci-stub 0000:81:11.5: claimed by stub

            Jul 22 12:53:40 ubuntu kernel: [ 3138.013224] pci-stub 0000:81:11.7: claimed by stub

            Jul 22 12:53:40 ubuntu kernel: [ 3138.013235] pci-stub 0000:81:12.1: claimed by stub

            Jul 22 12:53:40 ubuntu kernel: [ 3138.013246] pci-stub 0000:81:12.3: claimed by stub

            Jul 22 12:53:40 ubuntu kernel: [ 3138.013258] pci-stub 0000:81:12.5: claimed by stub

            Jul 22 12:53:40 ubuntu kernel: [ 3138.013269] pci-stub 0000:81:12.7: claimed by stub

            Jul 22 12:53:40 ubuntu kernel: [ 3138.013280] pci-stub 0000:81:13.1: claimed by stub

            Jul 22 12:53:40 ubuntu kernel: [ 3138.013291] pci-stub 0000:81:13.3: claimed by stub

            Jul 22 12:53:40 ubuntu kernel: [ 3138.013302] pci-stub 0000:81:13.5: claimed by stub

            Jul 22 12:53:40 ubuntu kernel: [ 3138.013314] pci-stub 0000:81:13.7: claimed by stub

            Jul 22 12:53:40 ubuntu kernel: [ 3138.033103] ixgbe 0000:81:00.0: setting MAC 52:54:00:58:5f:0c on VF 0

            Jul 22 12:53:40 ubuntu kernel: [ 3138.033109] ixgbe 0000:81:00.0: Reload the VF driver to make this change effective.

            Jul 22 12:53:40 ubuntu kernel: [ 3138.033112] ixgbe 0000:81:00.0: The VF MAC address has been set, but the PF device is not up.

            Jul 22 12:53:40 ubuntu kernel: [ 3138.033115] ixgbe 0000:81:00.0: Bring the PF device up before attempting to use the VF device.

            Jul 22 12:53:40 ubuntu kernel: [ 3138.045072] ixgbe 0000:81:00.1: setting MAC 52:54:00:d4:63:a7 on VF 0

            Jul 22 12:53:40 ubuntu kernel: [ 3138.045078] ixgbe 0000:81:00.1: Reload the VF driver to make this change effective.

            Jul 22 12:53:40 ubuntu kernel: [ 3138.045081] ixgbe 0000:81:00.1: The VF MAC address has been set, but the PF device is not up.

            Jul 22 12:53:40 ubuntu kernel: [ 3138.045084] ixgbe 0000:81:00.1: Bring the PF device up before attempting to use the VF device.

            Jul 22 12:53:40 ubuntu kernel: [ 3138.053024] ixgbe 0000:81:00.0: setting MAC 52:54:00:51:5c:0a on VF 1

            Jul 22 12:53:40 ubuntu kernel: [ 3138.053030] ixgbe 0000:81:00.0: Reload the VF driver to make this change effective.

            Jul 22 12:53:40 ubuntu kernel: [ 3138.053033] ixgbe 0000:81:00.0: The VF MAC address has been set, but the PF device is not up.

            Jul 22 12:53:40 ubuntu kernel: [ 3138.053036] ixgbe 0000:81:00.0: Bring the PF device up before attempting to use the VF device.

            Jul 22 12:53:40 ubuntu kernel: [ 3138.305853] type=1400 audit(1406058820.760:42): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-d3724bdc-fc97-37d0-437c-33f5d54fb811" pid=2795 comm="apparmor_parser"

            Jul 22 12:53:40 ubuntu kernel: [ 3138.419792] pci-stub 0000:81:10.0: enabling device (0000 -> 0002)

            Jul 22 12:53:42 ubuntu kernel: [ 3139.873334] pci-stub 0000:81:10.0: kvm assign device

            Jul 22 12:53:42 ubuntu kernel: [ 3139.874950] pci-stub 0000:81:10.1: enabling device (0000 -> 0002)

            Jul 22 12:53:42 ubuntu kernel: [ 3139.978660] pci-stub 0000:81:10.1: kvm assign device

            Jul 22 12:53:42 ubuntu kernel: [ 3139.981161] pci-stub 0000:81:10.2: enabling device (0000 -> 0002)

            Jul 22 12:53:42 ubuntu kernel: [ 3140.082788] pci-stub 0000:81:10.2: kvm assign device

            Jul 22 12:53:49 ubuntu kernel: [ 3146.897520] kvm [2797]: vcpu0 disabled perfctr wrmsr: 0xc1 data 0xabcd

            Jul 22 12:53:49 ubuntu kernel: [ 3146.967250] kvm: zapping shadow pages for mmio generation wraparound

            Jul 22 12:53:51 ubuntu kernel: [ 3149.537229] pci-stub 0000:81:10.0: irq 193 for MSI/MSI-X

            Jul 22 12:53:51 ubuntu kernel: [ 3149.537245] pci-stub 0000:81:10.0: irq 194 for MSI/MSI-X

            Jul 22 12:53:52 ubuntu kernel: [ 3149.557331] pci-stub 0000:81:10.0: irq 193 for MSI/MSI-X

            Jul 22 12:53:52 ubuntu kernel: [ 3149.557372] pci-stub 0000:81:10.0: irq 194 for MSI/MSI-X

            Jul 22 12:53:52 ubuntu kernel: [ 3149.577287] pci-stub 0000:81:10.1: irq 195 for MSI/MSI-X

            Jul 22 12:53:52 ubuntu kernel: [ 3149.577329] pci-stub 0000:81:10.1: irq 196 for MSI/MSI-X

            Jul 22 12:53:52 ubuntu kernel: [ 3149.605431] pci-stub 0000:81:10.1: irq 195 for MSI/MSI-X

            Jul 22 12:53:52 ubuntu kernel: [ 3149.605471] pci-stub 0000:81:10.1: irq 196 for MSI/MSI-X

            Jul 22 12:53:52 ubuntu kernel: [ 3149.621296] pci-stub 0000:81:10.2: irq 197 for MSI/MSI-X

            Jul 22 12:53:52 ubuntu kernel: [ 3149.621327] pci-stub 0000:81:10.2: irq 198 for MSI/MSI-X

            Jul 22 12:53:52 ubuntu kernel: [ 3149.645417] pci-stub 0000:81:10.2: irq 197 for MSI/MSI-X

            Jul 22 12:53:52 ubuntu kernel: [ 3149.645458] pci-stub 0000:81:10.2: irq 198 for MSI/MSI-X

             

            I'm not ruling out yet that I could be doing something wrong.

             

            Thanks for your assistance.

             

            JC

            • 3. Re: 82599ES SR-IOV on Ubuntu 14.04 - VF behavior
              Patrick_Kutch

              If I'm reading this correctly, I think the problem is due to the fact that you need to bring the interface up after the driver is loaded.

               

              So make sure you have a cable plugged in, then issue:

               

              ip link set eth4 up

               

              this should bring the link up on the PF, then you can assign the VLAN IDs tothe VF, and then assign the VM to the VM and then bring up the VM.

               

              Give that a try.

               

              - Patrick

              • 4. Re: 82599ES SR-IOV on Ubuntu 14.04 - VF behavior
                calvin419

                Thanks once again for replying.

                 

                I actually had corrected the link issue prior to your response and now I know it's not the PF state.

                 

                I think this actually a number/order issue.  Is there a simple way to establish exactly what VF belongs to which PF?  Lspci or dmesg output doesn't seem to establish the connection.

                 

                In my previous examples, I used pci_0000_81_10_0, 0000_81_10_1 & 0000_81_10_2 but I am finding out that these aren't consecutive VFs of the same PF or something else has gone awry. 

                 

                What I got from looking at virsh nodedev-dumpxml is that these three addresses match to the following PFs/VFs...

                 

                Am I on the right track?

                 

                pci_0000_81_10_0Eth4 VF0
                pci_0000_81_10_1Eth5 VF1
                pci_0000_81_10_2Eth4 VF2

                 

                 

                 

                root@ubuntu:~# virsh nodedev-dumpxml pci_0000_81_10_0

                <device>

                  <name>pci_0000_81_10_0</name>

                  <path>/sys/devices/pci0000:80/0000:80:01.0/0000:81:10.0</path>

                  <parent>pci_0000_80_01_0</parent>

                  <driver>

                    <name>pci-stub</name>

                  </driver>

                  <capability type='pci'>

                    <domain>0</domain>

                    <bus>129</bus>

                    <slot>16</slot>

                    <function>0</function>

                    <product id='0x10ed'>82599 Ethernet Controller Virtual Function</product>

                    <vendor id='0x8086'>Intel Corporation</vendor>

                    <capability type='phys_function'>

                      <address domain='0x0000' bus='0x81' slot='0x00' function='0x0'/>

                    </capability>

                    <iommuGroup number='51'>

                      <address domain='0x0000' bus='0x81' slot='0x10' function='0x0'/>

                    </iommuGroup>

                  </capability>

                </device>

                 

                 

                root@ubuntu:~# virsh nodedev-dumpxml pci_0000_81_10_1

                <device>

                  <name>pci_0000_81_10_1</name>

                  <path>/sys/devices/pci0000:80/0000:80:01.0/0000:81:10.1</path>

                  <parent>pci_0000_80_01_0</parent>

                  <driver>

                    <name>pci-stub</name>

                  </driver>

                  <capability type='pci'>

                    <domain>0</domain>

                    <bus>129</bus>

                    <slot>16</slot>

                    <function>1</function>

                    <product id='0x10ed'>82599 Ethernet Controller Virtual Function</product>

                    <vendor id='0x8086'>Intel Corporation</vendor>

                    <capability type='phys_function'>

                      <address domain='0x0000' bus='0x81' slot='0x00' function='0x1'/>

                    </capability>

                    <iommuGroup number='67'>

                      <address domain='0x0000' bus='0x81' slot='0x10' function='0x1'/>

                    </iommuGroup>

                  </capability>

                </device>

                 

                 

                root@ubuntu:~# virsh nodedev-dumpxml pci_0000_81_10_2

                <device>

                  <name>pci_0000_81_10_2</name>

                  <path>/sys/devices/pci0000:80/0000:80:01.0/0000:81:10.2</path>

                  <parent>pci_0000_80_01_0</parent>

                  <driver>

                    <name>pci-stub</name>

                  </driver>

                  <capability type='pci'>

                    <domain>0</domain>

                    <bus>129</bus>

                    <slot>16</slot>

                    <function>2</function>

                    <product id='0x10ed'>82599 Ethernet Controller Virtual Function</product>

                    <vendor id='0x8086'>Intel Corporation</vendor>

                    <capability type='phys_function'>

                      <address domain='0x0000' bus='0x81' slot='0x00' function='0x0'/>

                    </capability>

                    <iommuGroup number='52'>

                      <address domain='0x0000' bus='0x81' slot='0x10' function='0x2'/>

                    </iommuGroup>

                  </capability>

                </device>

                 

                root@ubuntu:~# lspci | grep 82599

                81:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

                81:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

                81:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

                81:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

                81:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

                81:10.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

                81:10.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

                81:10.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

                81:10.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

                81:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

                81:11.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

                81:11.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

                81:11.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

                81:11.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

                81:11.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

                81:11.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

                81:11.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

                81:11.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

                81:12.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

                81:12.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

                81:12.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

                81:12.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

                81:12.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

                81:12.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

                81:12.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

                81:12.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

                81:13.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

                81:13.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

                81:13.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

                81:13.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

                81:13.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

                81:13.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

                81:13.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

                81:13.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

                • 5. Re: 82599ES SR-IOV on Ubuntu 14.04 - VF behavior
                  Patrick_Kutch

                  The good ole, which PF does the VF belong to challenge :-)

                   

                  For a dual port device, even numbered VF's belongs to PF0, and odd numbered VF's belong to PF1.

                   

                  So...

                   

                  81:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)         ß BELONG TO PF#0 i.e.  PF 81:00.0

                  81:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)         ß BELONG TO PF#1 i.e.  PF 81:00.1

                  81:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)         ß BELONG TO PF#0 i.e.  PF 81:00.0

                  81:10.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)         ß BELONG TO PF#1 i.e.  PF 81:00.1

                  81:10.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)         ß BELONG TO PF#0 i.e.  PF 81:00.0

                  81:10.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)         ß BELONG TO PF#1 i.e.  PF 81:00.1

                   

                  Hope that clears it up.

                   

                  - Patrick

                  • 6. Re: 82599ES SR-IOV on Ubuntu 14.04 - VF behavior
                    calvin419

                    Thanks for the feedback Patrick and clearing that up.  It makes sense now.  Also, sorry for the late response.  I was away for a few days

                     

                    It's evident that I will need to set per VF vlans in the guest XML files and I'm doing that like this as an example:

                     

                      <vlan>
                        <tag id='300'/>
                      </vlan>

                     

                    This may be outside the scope of this thread, but do you know if nativeMode='untagged' is necessary or will it treat this VF like an access-port connecting to the guest natively using vlan 300 to pass untagged frames?

                     

                    JC