5 Replies Latest reply on Jun 27, 2017 1:40 AM by Intel Corporation

    X540-T2 VF cannot receive packet with OVS

    Ray_Tracy_

      I have configured host with two PF and 16 x2 VF in debian system. It worked well if I use the PF/VF as standalone NIC, but not functional with OVS bridge.

       

      The testing scenario as below:

       

      eth0 (PF): 10.250.11.103/24

      eth11 (VF 9): 10.250.11.113/24

       

         Bridge "vmbr1"

              Port "eth10"

                  Interface "eth10" (VF 8)

              Port "vmbr1"

                  Interface "vmbr1"

                      type: internal

              Port "in1"

                  Interface "in1"

                      type: internal

          ovs_version: "2.6.0"

       

      in1: 10.250.11.34/24

       

      I try the following icmp test:

       

      ping 10.250.11.254 (reply good)

      ping 10.250.11.254 -I eth11 (reply good)

      ping 10.250.11.254 -I in1 (NO response)

       

      ip link show my VF as:

      -------------------------------

      2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

          link/ether 0c:c4:7a:df:36:c6 brd ff:ff:ff:ff:ff:ff

          vf 0 MAC 6e:74:9e:65:58:eb, spoof checking on, link-state auto

          vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 3 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 4 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 5 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 7 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 8 MAC da:c7:06:0b:5b:1f, spoof checking on, link-state auto

          vf 9 MAC ea:97:12:4e:b4:6b, spoof checking on, link-state auto

          vf 10 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 11 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 12 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 13 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 14 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

          vf 15 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

       

      arp table shown as:

      -------------------------------

      Address                  HWtype  HWaddress           Flags Mask            Iface

      10.250.11.254            ether   6c:3b:6b:e9:7c:e1   C                     eth11

      10.250.11.254            ether   6c:3b:6b:e9:7c:e1   C                     eth0

      10.250.11.254            ether   6c:3b:6b:e9:7c:e1   C                     in1

       

      But arp is not stable, it disappear very quickly:

      ------------------------------------------------------------

      Address                  HWtype  HWaddress           Flags Mask            Iface

      10.250.11.254                   (incomplete)                              in1

      10.250.11.254            ether   6c:3b:6b:e9:7c:e1   C                     eth0

      10.250.11.254            ether   6c:3b:6b:e9:7c:e1   C                     eth11

       

      The strange thing is:

      ping via in1 will not fill up the arp table for in1 Iface (always show incomplete), but ping via other two iface seems fill the mac address into in1 some time.

      I have tcpdump to in1 and found it send out ARP request broadcast, but did not get any ARP reply from host 254.

       

      ovs-appctl fdb/show vmbr1:

      -----------------------------------------

      port  VLAN  MAC                Age

          2     0  72:fc:b4:3a:e6:0f   78

          1     0  6c:3b:6b:e9:7c:e1   38

          1     0  6c:3b:6b:f5:96:e6    3

          1     0  88:43:e1:dd:9c:80    1

      This looks persist and will not change whatever I ping via any Iface.

       

      I have also try to change VF port, but seems none of them work with OVS bridge.

      But the bridge worked if assigned the port to eth0 or eth1 PF.

       

      uname -a:

      Linux pve3 4.4.62-1-pve #1 SMP PVE 4.4.62-88 (Thu, 18 May 2017 09:18:43 +0200) x86_64 GNU/Linux

       

      /etc/network/interfaces:

      ----------------------------------

      auto eth0

      iface eth0 inet static

              address  10.250.11.103

              netmask  255.255.255.0

              gateway  10.250.11.254

       

      auto eth11

      iface eth11 inet static

              address  10.250.11.113

              netmask  255.255.255.0

       

      allow-vmbr1 eth10

      iface eth10 inet manual

              ovs_type OVSPort

              ovs_bridge vmbr1

       

      auto vmbr1

      iface vmbr1 inet manual

              ovs_type OVSBridge

              ovs_ports eth10 in1

       

      allow-vmbr1 in1

      iface in1 inet static

              address  10.250.11.34

              netmask  255.255.255.0

              ovs_type OVSIntPort

              ovs_bridge vmbr1

       

      Here is ethtool info for eth10:

      -------------------------------

      # ethtool -i eth10

      driver: ixgbevf

      version: 2.12.1-k

      firmware-version:

      bus-info: 0000:05:12.0

      supports-statistics: yes

      supports-test: yes

      supports-eeprom-access: no

      supports-register-dump: yes

      supports-priv-flags: no

       

      #ethtool -k eth10

      Features for eth10:

      rx-checksumming: on

      tx-checksumming: on

              tx-checksum-ipv4: off [fixed]

              tx-checksum-ip-generic: on

              tx-checksum-ipv6: off [fixed]

              tx-checksum-fcoe-crc: off [fixed]

              tx-checksum-sctp: on

      scatter-gather: on

              tx-scatter-gather: on

              tx-scatter-gather-fraglist: off [fixed]

      tcp-segmentation-offload: on

              tx-tcp-segmentation: on

              tx-tcp-ecn-segmentation: off [fixed]

              tx-tcp6-segmentation: on

      udp-fragmentation-offload: off [fixed]

      generic-segmentation-offload: on

      generic-receive-offload: on

      large-receive-offload: off [fixed]

      rx-vlan-offload: on [fixed]

      tx-vlan-offload: on [fixed]

      ntuple-filters: off [fixed]

      receive-hashing: off [fixed]

      highdma: on [fixed]

      rx-vlan-filter: on [fixed]

      vlan-challenged: off [fixed]

      tx-lockless: off [fixed]

      netns-local: off [fixed]

      tx-gso-robust: off [fixed]

      tx-fcoe-segmentation: off [fixed]

      tx-gre-segmentation: off [fixed]

      tx-ipip-segmentation: off [fixed]

      tx-sit-segmentation: off [fixed]

      tx-udp_tnl-segmentation: off [fixed]

      fcoe-mtu: off [fixed]

      tx-nocache-copy: off

      loopback: off [fixed]

      rx-fcs: off [fixed]

      rx-all: off [fixed]

      tx-vlan-stag-hw-insert: off [fixed]

      rx-vlan-stag-hw-parse: off [fixed]

      rx-vlan-stag-filter: off [fixed]

      l2-fwd-offload: off [fixed]

      busy-poll: on [fixed]

      hw-tc-offload: off [fixed]

      -------------------------------

       

      Is there any advice?