3 Replies Latest reply on Oct 8, 2014 3:51 PM by ShivaB

    Intel X520 (82599) Virtual Functions on FreeBSD 8.3/10

    skpio

      I've run into a one-way transmission issue with FreeBSD and Virtual Functions on an Intel 10GbE X520. So far we've tested:

       

      •     Ubuntu 12.04 LTS
      •     CentOS 6.5
      •     pfSense 2.1 / FreeBSD 8.3
      •     pfSense 2.2 / FreeBSD 10.0 (alpha)
      •     FreeBSD 8.3-STABLE

       

      All are instances are running on top of KVM installed on Ubuntu 14, hosted on Intel Grizzly Pass OEM hardware with the above-mentioned card installed. The card's two 10GbE ports are connected back-to-back.

       

      In our test case, all instances above have a VF inside VLAN 100 and are bridged via the 82599 controller's internal L2 'switch.' The Ubuntu and CentOS instances can reach each other on their VF/VLAN100 interfaces, but any FreeBSD instance spun up cannot. tcpdump on a CentOS instance and a FreeBSD instance show the FreeBSD instance send an ARP request, the CentOS instance receive it and respond, but it never reaches the FreeBSD instance. Since this is not an issue on Ubuntu or CentOS, and has persisted across two versions of FreeBSD and two Intel VF driver versions, I have to assume it's unique to FreeBSD. The examples below are between FreeBSD 8.3-STABLE and CentOS 6.5:

       

      freebsd1# tcpdump -i ix0
      tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
      listening on ix0, link-type EN10MB (Ethernet), capture size 96 bytes
      21:24:44.137249 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
      21:24:45.144856 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
      21:24:46.154881 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
      21:24:47.164913 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
      21:24:48.174898 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
      21:24:49.184903 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
      21:24:50.194905 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
      21:24:51.204917 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
      21:24:52.214924 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
      21:24:53.224947 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
      21:24:54.234961 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
      21:24:55.245032 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
      21:24:56.255034 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
      ^C
      13 packets captured
      13 packets received by filter
      0 packets dropped by kernel
      
      

       

      [root@centos1 ~]# tcpdump -i eth1
      tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
      listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
      17:24:44.274997 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
      17:24:44.275053 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
      17:24:45.282609 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
      17:24:45.282651 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
      17:24:46.292589 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
      17:24:46.292602 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
      17:24:47.302563 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
      17:24:47.302576 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
      17:24:48.312541 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
      17:24:48.312553 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
      17:24:49.322596 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
      17:24:49.322609 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
      17:24:50.332542 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
      17:24:50.332553 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
      17:24:51.342602 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
      17:24:51.342614 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
      17:24:52.352554 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
      17:24:52.352566 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
      17:24:53.362614 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
      17:24:53.362627 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
      17:24:54.372599 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
      17:24:54.372611 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
      17:24:55.382681 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
      17:24:55.382802 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
      ^C
      24 packets captured
      24 packets received by filter
      0 packets dropped by kernel
      
      

       

      freebsd1# uname -a
      FreeBSD freebsd1 8.3-RELEASE FreeBSD 8.3-RELEASE #0: Mon Apr 9 21:23:18 UTC 2012 root@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64
      
      

       

      freebsd1# dmesg | grep Virtual
      CPU: QEMU Virtual CPU version 2.0.0 (2793.29-MHz K8-class CPU)
      ix0: <Intel(R) PRO/10GbE Virtual Function Network Driver, Version - 1.1.2> mem 0xfebf0000-0xfebf3fff,0xfebf4000-0xfebf7fff at device 5.0 on pci0
      
      

       

      freebsd1# ifconfig ix0
      ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
      options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
      ether 52:54:00:44:99:6c
      inet6 fe80::5054:ff:fe44:996c%ix0 prefixlen 64 scopeid 0x3
      inet 192.168.100.253 netmask 0xffffff00 broadcast 192.168.100.255
      nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
      media: Ethernet autoselect
      status: active
      
      

       

      Our issue is identical in description to this post. However, there is no "Simulated MSI Support" option in our BIOS and I believe that particular option was unique to his board based on Intel's BIOS release notes. Another individual ran into a similar problem with FreeBSD 10.0.

       

      Note that pfSense runs v1.1.4 of the Intel VF driver. If you look here, Intel provides a link to e1000 FreeBSD VF drivers. However, no VF-specific drivers are mentioned for ixgbe. There is _vf.h and _vf.c source code in the FreeBSD tree, but we have been unable to compile newer drivers and get them to load.

       

      I'm currently at a loss. I can provide whatever additional information may be helpful; I'm not sure what else is useful at this point. I've dumped some additional info below.

       

      On the host, SR-IOV and IOMMU are enabled. The 10GbE card is on PCI bus 81:00:

       

      root@ubuntu:~# dmesg | grep Intel | grep ixgbe
      [ 8.690398] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 3.15.1-k
      [ 8.690398] ixgbe: Copyright (c) 1999-2013 Intel Corporation.
      [ 8.950982] ixgbe 0000:81:00.0: Intel(R) 10 Gigabit Network Connection
      [ 9.211614] ixgbe 0000:81:00.1: Intel(R) 10 Gigabit Network Connection
      
      

       

      VFs are enabled at the host level:

       

      root@ubuntu:~# lspci -s 81:
      81:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
      81:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
      81:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
      81:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
      81:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
      81:10.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
      81:10.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
      81:10.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
      81:10.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
      81:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
      81:11.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
      81:11.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
      81:11.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
      81:11.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
      81:11.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
      81:11.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
      81:11.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
      81:11.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
      81:12.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
      81:12.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
      81:12.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
      81:12.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
      81:12.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
      81:12.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
      81:12.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
      81:12.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
      81:13.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
      81:13.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
      81:13.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
      81:13.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
      81:13.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
      81:13.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
      81:13.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
      81:13.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
      
      

       

      The virtual functions with several instances spun up. VF3 is centos1, VF6 is freebsd1.

       

      root@ubuntu:~# ip link show dev eth4
      5: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000
      link/ether 90:e2:ba:47:2c:30 brd ff:ff:ff:ff:ff:ff
      vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
      vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
      vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
      vf 3 MAC 52:54:00:db:57:b2, vlan 100, spoof checking on, link-state auto
      vf 4 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
      vf 5 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
      vf 6 MAC 52:54:00:44:99:6c, vlan 100, spoof checking on, link-state auto
      vf 7 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
      vf 8 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
      vf 9 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
      vf 10 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
      vf 11 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
      vf 12 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
      vf 13 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
      vf 14 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
      vf 15 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
      
      

       

      KVM is also pleased:

       

      root@ubuntu:~# virsh nodedev-list | grep 81
      pci_0000_81_00_0
      pci_0000_81_00_1
      pci_0000_81_10_0
      pci_0000_81_10_1
      pci_0000_81_10_2
      pci_0000_81_10_3
      pci_0000_81_10_4
      pci_0000_81_10_5
      pci_0000_81_10_6
      pci_0000_81_10_7
      pci_0000_81_11_0
      pci_0000_81_11_1
      pci_0000_81_11_2
      pci_0000_81_11_3
      pci_0000_81_11_4
      pci_0000_81_11_5
      pci_0000_81_11_6
      pci_0000_81_11_7
      pci_0000_81_12_0
      pci_0000_81_12_1
      pci_0000_81_12_2
      pci_0000_81_12_3
      pci_0000_81_12_4
      pci_0000_81_12_5
      pci_0000_81_12_6
      pci_0000_81_12_7
      pci_0000_81_13_0
      pci_0000_81_13_1
      pci_0000_81_13_2
      pci_0000_81_13_3
      pci_0000_81_13_4
      pci_0000_81_13_5
      pci_0000_81_13_6
      pci_0000_81_13_7
      
      

       

      The FreeBSD instance in question is on pci_0000_81_11_4, which maps to VF6 above.

       

      KVM XML for the freebsd1's interface:

       

              <interface type='hostdev' managed='yes'>
                <mac address='52:54:00:44:99:6c'/>
                <source>
                  <address type='pci' domain='0x0000' bus='0x81' slot='0x11' function='0x4'/>
                </source>
                <vlan>
                  <tag id='100'/>
                </vlan>
                <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
              </interface>
      
      

       

      FYI - we allow KVM to assign the MAC, etc. We only supply the bus, slot, and function numbers when adding a VF-based interface to the instance.

       

      Thanks to anyone who reads and responds. I could really use some help. I've made similar posts on the FreeBSD and pfSense forums, as well as in both /r/freebsd and /r/pfsense on reddit. gonzopancho on reddit directed me to this link, which has a number of patches written by Ryan Stone. It seems like this may be a known issue. At this point we're just looking for confirmation that we aren't crazy. I've felt like this for a week.

       

      Thanks!