5 Replies Latest reply on Aug 15, 2017 7:52 PM by Intel Corporation Branched from an earlier discussion.

    Re: SR-IOV with IXGBE - Vlan packets getting spoofed-using 82599ES in Mirantis Fuel 9.2

    dwangoAC

      Hi Sharon and Pratik,

       

      I have the exact same need - I have an 82599ES-based SR-IOV setup using a Mirantis Fuel 9.2 OpenStack deployment where I am trying to pass VLAN-tagged packets out of a KVM VM instance with the VLAN kept intact.  Just like Pratik, I can work with a situation where either the tag applied by the virtual machine is retained with no tag added to it on egress or (preferably) a situation where the inner tag applied by the VM is preserved and an outer tag is added by the network adapter.

       

      I have a full lab environment set up to test this and I will be compiling both drivers. However, I'm curious if there is a specific fix that addresses this issue and if you have any additional information on what behavior I should expect. Will the ixgbe 5.1 driver and the ixgbevf 4.1 driver allow both of the situations I described, i.e. either one tag or double tags leaving the VM?  Thank you for any clarification you can provide,

       

      A.C.

      ******

        • 1. Re: SR-IOV with IXGBE - Vlan packets getting spoofed-using 82599ES in Mirantis Fuel 9.2
          Intel Corporation
          This message was posted on behalf of Intel Corporation

          Hi AC,
           Thank you for the information. Currently I am waiting for further update from Pratik, we will further check on this at our end.

          regards,
          sharon
           

          • 2. Re: SR-IOV with IXGBE - Vlan packets getting spoofed-using 82599ES in Mirantis Fuel 9.2
            dwangoAC

            Hi Sharon,

             

            I have completed compiling both drivers but loading the new modules does not change the behavior; attempting to set a VLAN inside of a VM results in the VLAN being stripped on egress, after which the VLAN associated with the VF is applied as the only tag present.  Here is the system configuration I am working with and a synopsis of the basic steps I took:

             

            # uname -a Linux sriov4-cpu1.bpi.ciena.com 4.4.0-62-generic #83~14.04.1-Ubuntu SMP Wed Jan 18 18:10:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

            # lsb_release -a

            No LSB modules are available.

            Distributor ID: Ubuntu

            Description: Ubuntu 14.04.5 LTS

            Release: 14.04

            Codename: trusty

            ~# rmmod ixgbe ixgbevf

            ~# cd  /ixgbe/ixgbe-5.2.1/src

            ~/ixgbe/ixgbe-5.2.1/src# make install

            ~/ixgbe/ixgbe-5.2.1/src# cd  ~/ixgbevf/ixgbevf-4.2.1/src

            ~/ixgbevf/ixgbevf-4.2.1/src# make install

            ~/ixgbevf/ixgbevf-4.2.1/src# modprobe ixgbe ixgbevf

            ~/ixgbevf/ixgbevf-4.2.1/src# update-initramfs -u

            ~/ixgbevf/ixgbevf-4.2.1/src# ethtool -i enp4s0f0

            driver: ixgbe

            version: 5.2.1

            firmware-version: 0x800007f5, 17.5.10

            bus-info: 0000:04:00.0

            supports-statistics: yes

            supports-test: yes

            supports-eeprom-access: yes

            supports-register-dump: yes

            supports-priv-flags: yes

             

            All steps and commands succeeded as expected and I was able to start a VM (launch a previously created OpenStack instance, i.e. launch a KVM visible with virsh).  I saw the following output from dmesg showing the tail of the modprobe initialization:

             

            [12589154.358105] ixgbevf: eth370: ixgbevf_probe: Intel(R) 82599 Virtual Function
            [12589154.358107] ca:0c:b9:54:2a:b2
            [12589154.358110] ixgbevf: eth370: ixgbevf_probe: GRO is enabled
            [12589154.358112] ixgbevf: eth370: ixgbevf_probe: Intel(R) 10GbE PCI Express Virtual Function Driver
            [12589154.358143] pci 0000:83:1f.3: [8086:10ed] type 00 class 0x020000
            [12589154.358189] pci 0000:83:1f.3: can't set Max Payload Size to 256; if necessary, use "pci=pcie_bus_safe" and report a bug
            [12589154.358550] iommu: Adding device 0000:83:1f.3 to group 426
            [12589154.358617] ixgbevf 0000:83:1f.3: enabling device (0000 -> 0002)
            [12589154.420133] ixgbe 0000:83:00.1 enp131s0f1: VF Reset msg received from vf 61
            [12589154.420472] ixgbe 0000:83:00.1: VF 61 has no MAC address assigned, you may have to assign one manually
            [12589154.436496] ixgbevf 0000:83:1f.3: MAC address not assigned by administrator.
            [12589154.436501] ixgbevf 0000:83:1f.3: Assigning random MAC address
            [12589154.437558] ixgbevf 0000:83:1f.3: Multiqueue Enabled: Rx Queue count = 2, Tx Queue count = 2
            [12589154.438125] ixgbevf: eth371: ixgbevf_probe: Intel(R) 82599 Virtual Function
            [12589154.438127] 22:9d:65:6d:95:83
            [12589154.438131] ixgbevf: eth371: ixgbevf_probe: GRO is enabled
            [12589154.438132] ixgbevf: eth371: ixgbevf_probe: Intel(R) 10GbE PCI Express Virtual Function Driver
            [12589160.674214] audit: type=1400 audit(1501615158.835:173): apparmor="DENIED" operation="open" profile="/usr/sbin/ntpd" name="/usr/local/sbin/" pid=124042 comm="ntpd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
            [12589160.674224] audit: type=1400 audit(1501615158.835:174): apparmor="DENIED" operation="open" profile="/usr/sbin/ntpd" name="/usr/local/bin/" pid=124042 comm="ntpd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
            [12589436.263828] ixgbevf: eth47: ixgbevf_remove: Remove complete
            [12589436.283385] ixgbevf: eth25: ixgbevf_remove: Remove complete
            [12589436.327460] ixgbe 0000:04:00.0: setting MAC fa:16:39:4c:63:d6 on VF 16
            [12589436.327465] ixgbe 0000:04:00.0: Reload the VF driver to make this change effective.
            [12589436.327506] ixgbe 0000:04:00.0: Setting VLAN 1807, QOS 0x0 on VF 16
            [12589436.353284] ixgbe 0000:04:00.1: setting MAC fa:16:39:26:5e:1c on VF 8
            [12589436.353288] ixgbe 0000:04:00.1: Reload the VF driver to make this change effective.
            [12589436.353328] ixgbe 0000:04:00.1: Setting VLAN 1807, QOS 0x0 on VF 8
            [12589436.607825] audit: type=1400 audit(1501615434.749:175): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-e3675296-f81e-4d61-9280-ce0e7eaee8a1" pid=161335 comm="apparmor_parser"
            [12589436.608013] audit: type=1400 audit(1501615434.749:176): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-e3675296-f81e-4d61-9280-ce0e7eaee8a1//qemu_bridge_helper" pid=161335 comm="apparmor_parser"
            [12589436.699374] device tap8643a0ca-f4 entered promiscuous mode
            [12589436.727574] qbr8643a0ca-f4: port 2(tap8643a0ca-f4) entered forwarding state
            [12589436.727583] qbr8643a0ca-f4: port 2(tap8643a0ca-f4) entered forwarding state
            [12589436.929559] audit: type=1400 audit(1501615435.069:177): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-e3675296-f81e-4d61-9280-ce0e7eaee8a1" pid=161558 comm="apparmor_parser"
            [12589436.963500] audit: type=1400 audit(1501615435.105:178): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-e3675296-f81e-4d61-9280-ce0e7eaee8a1//qemu_bridge_helper" pid=161558 comm="apparmor_parser"
            [12589437.857401] vfio-pci 0000:04:14.0: enabling device (0000 -> 0002)
            [12589437.961279] vfio-pci 0000:04:12.1: enabling device (0000 -> 0002)
            [12589438.633981] kvm: zapping shadow pages for mmio generation wraparound
            [12589438.689159] kvm: zapping shadow pages for mmio generation wraparound
            [12589446.280227] kvm [161563]: vcpu2 unhandled rdmsr: 0x606
            [12589446.288856] kvm [161563]: vcpu2 unhandled rdmsr: 0x34
            [12589446.474826] ixgbe 0000:04:00.0 enp4s0f0: VF Reset msg received from vf 16
            [12589446.490905] ixgbe 0000:04:00.1 enp4s0f1: VF Reset msg received from vf 8
            [12589456.762264] ixgbe 0000:04:00.0 enp4s0f0: VF Reset msg received from vf 16
            [12589456.825927] ixgbe 0000:04:00.1 enp4s0f1: VF Reset msg received from vf 8
            [12589457.012825] ixgbe 0000:04:00.0 enp4s0f0: VF 16 attempted to set a new MAC address but it already has an administratively set MAC address FA:16:39:4C:63:D6
            [12589457.012829] ixgbe 0000:04:00.0 enp4s0f0: Check the VF driver and if it is not using the correct MAC address you may need to reload the VF driver
            [12589457.014910] ixgbe 0000:04:00.1 enp4s0f1: VF 8 attempted to set a new MAC address but it already has an administratively set MAC address FA:16:39:26:5E:1C
            [12589457.014913] ixgbe 0000:04:00.1 enp4s0f1: Check the VF driver and if it is not using the correct MAC address you may need to reload the VF driver
            [12589460.455536] ixgbe 0000:04:00.0 enp4s0f0: VF Reset msg received from vf 16
            [12589461.947302] ixgbe 0000:04:00.1 enp4s0f1: VF Reset msg received from vf 8
            [12589465.143491] ixgbe 0000:04:00.0 enp4s0f0: VF 16 attempted to override administratively set VLAN configuration
            [12589465.143491] Reload the VF driver to resume operations
            

             

            It is not clear to me if the second to last message is indicative of the VM attempting to use an inner VLAN tag or not.  In the specific example I'm working with here, the outer VLAN tag on each of the two ports is 1807 and the inner tag is VLAN 100, i.e. VLAN 1807 is being used to send traffic into the VM and the VLAN 1807 tag is stripped (popped) properly on ingress leaving only VLAN 100 left.  However, when the VM attempts to send VLAN 100 out it is dropped entirely and only VLAN tag 1807 is present when the packet is received by external test equipment.

             

            Please advise if there is any way at all through either compilation options or configuration changes to allow the inner VLAN tag to persist, or alternately to not touch the VLAN tag on egress packets at all to allow the VLAN header(s) to pass undisturbed out of the adapter.

             

            If there is any other information I can provide about this specific test environment or if there are any steps you would like me to take please let me know.  This is my highest priority and I will respond as quickly as I am able to any responses from you; I am also available to do a troubleshooting session if it would be helpful.  Without this functionality SR-IOV scale is severely limited and at least some solution is needed to allow double-tagged traffic. Thanks in advance for any assistance you can provide,

             

            A.C.

            ******

            • 3. Re: SR-IOV with IXGBE - Vlan packets getting spoofed-using 82599ES in Mirantis Fuel 9.2
              Intel Corporation
              This message was posted on behalf of Intel Corporation

              Hi A.C.

              Thank you for the information. Let me further check.

              regards,
              sharon
               

              • 4. Re: SR-IOV with IXGBE - Vlan packets getting spoofed-using 82599ES in Mirantis Fuel 9.2
                dwangoAC

                After extensive testing we have determined that the combination of the current driver and firmware for 82599ES chipset adapters do not have the functionality to allow QinQ / double-tagged / stacked VLAN's on egress.  This appears to be a hard limitation with no workaround available.  If a VLAN is associated with a VF to allow ingress traffic to reach the correct VM it is not possible to egress on that same VF with a different VLAN. The only solution, and it's an insane one, is to associate two VF's for each port (one with a VLAN tag and one without) and only egress on the VF without the VLAN, using bizarre methods such as a fake bonded port or placing the virtual interfaces in a virtual switch with rules dictating where traffic can flow.  This is obviously not a viable workaround in a real VNF situation.

                 

                Are there *any* Intel chipsets that work properly in this regard? Mellanox has full QinQ / double tag support as noted at HowTo Configure QinQ Encapsulation per VF in Li... | Mellanox Interconnect Community but I'd like to know if there are any Intel chipsets that will work.  Thanks in advance,

                 

                A.C.

                ******

                • 5. Re: SR-IOV with IXGBE - Vlan packets getting spoofed-using 82599ES in Mirantis Fuel 9.2
                  Intel Corporation
                  This message was posted on behalf of Intel Corporation

                  Hi DwangoAC,

                   Thank you for the further update. You are right double VLAN is not supported on 82599ES products and respective drivers. Our Fortville supports double VLAN (aka QnQ) feature in the hardware, but the software does not support it at this time.  Double VLAN (aka QnQ) support is expected in a future software release but we don't have timeline yet. Hope this clarified.

                  regards,
                  sharon