10 Replies Latest reply on Jul 25, 2016 11:10 PM by ingenium

    Which bonding mode does Intel SR-IOV NIC support?

    henry_wang

      Hi, Expert

       

      I've investigated this for some days.

       

      My purpose is launching a VM which has two VFs come from two physical NICs, and then do the bonding inside the VM.

      I've tried all the 7 bonding modes, only the mode 1(active-backup) works, all the others cannot work.

       

      From the following articles, the reason why other bonding mode cannot work is VM cannot change the VF's MAC address successfully.

      https://bugzilla.redhat.com/show_bug.cgi?id=1164224

      SR-IOV on NIC 82599: Are VF MAC Addresses unique?

      https://communities.intel.com/message/246183#246183

       

      From the igb driver code, I also notice the HOST refuse VM's change VF's MAC address request.

       

      case E1000_VF_SET_MAC_ADDR:

          retval = -EINVAL;

          if (!(vf_data->flags & IGB_VF_FLAG_PF_SET_MAC))

               retval = igb_set_vf_mac_addr(adapter, msgbuf, vf);

          else

               dev_warn(&pdev->dev, "VF %d attempted to override administratively set MAC address\nReload the VF driver to resume operations\n", vf);

          break;

       

      So, My question is

      * Does Intel's SR-IOV NIC only support bonding mode active-backup? only this mode not request VM change it's interface's MAC address.

      * If I want to use active-active bonding mode, such as mode 0/2/4/5/6, any solution/walkaround to make it work?

       

      Thanks.

        • 1. Re: Which bonding mode does Intel SR-IOV NIC support?
          wb_Intel

          Hi henry_wang,

           

          Thank you for the post. Can you tell me are you using an onboard 82599 NIC or an X520 series (82599) network adapter?

           

          For the meantime, I wll check on this.

           

          rgds,

          wb

          • 2. Re: Which bonding mode does Intel SR-IOV NIC support?
            henry_wang

            Thanks for reply quickly.

             

            The NICs I use in my product are 82579 and I350.

             

            # lspci | grep Eth

            07:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

            07:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

            08:10.1 Ethernet controller: Intel Corporation I350 Virtual Function (rev 01)

            08:10.5 Ethernet controller: Intel Corporation I350 Virtual Function (rev 01)

            08:11.1 Ethernet controller: Intel Corporation I350 Virtual Function (rev 01)

            08:11.5 Ethernet controller: Intel Corporation I350 Virtual Function (rev 01)

            08:12.1 Ethernet controller: Intel Corporation I350 Virtual Function (rev 01)

            08:12.5 Ethernet controller: Intel Corporation I350 Virtual Function (rev 01)

            08:13.1 Ethernet controller: Intel Corporation I350 Virtual Function (rev 01)

            82:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

            82:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

            82:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

            82:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

            82:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

            82:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

            82:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

            82:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

            82:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

             

            And here is more info about kernel and driver:

            # uname -r

            2.6.32-504.23.4.el6.centos.plus.x86_64

             

            # ethtool -i eth0

            driver: igb

            version: 5.0.5-k

            firmware-version: 1.61, 0x8000090e

            bus-info: 0000:07:00.0

            supports-statistics: yes

            supports-test: yes

            supports-eeprom-access: yes

            supports-register-dump: yes

            supports-priv-flags: no

             

            # ethtool -i bond0

            driver: bonding

            version: 3.6.0

            firmware-version: 2

            bus-info:

            supports-statistics: no

            supports-test: no

            supports-eeprom-access: no

            supports-register-dump: no

            supports-priv-flags: no


            # ethtool -i eth21

            driver: igbvf

            version: 2.0.2-k

            firmware-version:

            bus-info: 0000:08:11.5

            supports-statistics: yes

            supports-test: yes

            supports-eeprom-access: no

            supports-register-dump: yes

            supports-priv-flags: no

            • 3. Re: Which bonding mode does Intel SR-IOV NIC support?
              wb_Intel

              Hi henry_wang,

               

              Thank you for the additional info.  82579 ethernet controller does not support SR-IOV but I350 does. Here is the website for reference:

              FAQ for Intel® Ethernet Server Adapters with SR-IOV

              Please refer to the Configure QoS with Intel® Flexible Port Partitioning regarding the bonding mode.

               

              rgds,

              wb

              • 4. Re: Which bonding mode does Intel SR-IOV NIC support?
                henry_wang

                I mean 82576 and I350, sorry for the typo error. Both types of NIC support SRIOV, I've already using the VF.

                 

                Thanks for the links share to me, I've also read these links before I asked this question.

                 

                From the reference Configure QoS with Intel® Flexible Port Partitioning  , seems VM is able to change the VF's MAC address, but when I do bonding according to the reference, error message report and bonding not success, the error message is:

                Master "bond0": Error: SIOCBONDENSLAVE failed: Cannot assign requested address

                Check on  the HOST,

                #dmesg | tail

                igb 0000:82:00.0: VF 5 attempted to override administratively set MAC address

                Reload the VF driver to resume operations

                 

                But from the reference SR-IOV on NIC 82599: Are VF MAC Addresses unique?  , Patrick said: "One caveat – User can use IPROUTE2 utility to assign a unique MAC address to a VF from within the host Operating System.  Once the new MAC address is assigned, VM that has this particular VF assigned to will not be able to alter its MAC address.  This is called “administratively assigned MAC” and is a security feature."

                And it's really what the igb driver's behavior:


                case E1000_VF_SET_MAC_ADDR:

                    retval = -EINVAL;

                    if (!(vf_data->flags & IGB_VF_FLAG_PF_SET_MAC))

                         retval = igb_set_vf_mac_addr(adapter, msgbuf, vf);

                    else

                         dev_warn(&pdev->dev, "VF %d attempted to override administratively set MAC address\nReload the VF driver to resume operations\n", vf);

                    break;


                So, I'm confused with these reference and the test result.


                PS. I suppose the hardware NIC should have no limit for the bonding mode selection, but the driver does. 

                My current igb driver is 5.0.5-k and igbvf driver is 2.0.2-k.


                • 6. Re: Which bonding mode does Intel SR-IOV NIC support?
                  henry_wang

                  Thanks for your links.

                   

                  I checked the latest driver code 5.3.3.5, it handle the VF MAC address change request the same as the driver 2.0.2:

                  if the VF's MAC address has been set by the PF from HOST, will not allow VM to set it again.

                   

                  The related code in igb_main.c:

                  6852     case E1000_VF_SET_MAC_ADDR:

                  6853         retval = -EINVAL;

                  6854 #ifndef IGB_DISABLE_VF_MAC_SET

                  6855         if (!(vf_data->flags & IGB_VF_FLAG_PF_SET_MAC))

                  6856             retval = igb_set_vf_mac_addr(adapter, msgbuf, vf);

                  6857         else

                  6858             DPRINTK(DRV, INFO,

                  6859                 "VF %d attempted to override administratively set MAC address\nReload the VF driver to resume operations\n",

                  6860                 vf);

                  6861 #endif

                  6862         break;

                   

                  Also I checked the ixgb driver code, also do the same thing for the change VF MAC address request:

                  if (adapter->vfinfo[vf].pf_set_mac &&

                      !ether_addr_equal(adapter->vfinfo[vf].vf_mac_addresses, new_mac)) {

                          e_warn(drv,

                          "VF %d attempted to override administratively set MAC address\n"

                          "Reload the VF driver to resume operations\n",

                          vf);

                       return -1;

                  }

                   

                  I've tried to modify the code to delete the check, allow change the VF's MAC address, and after load the new driver module,

                  all bonding modes work fine.

                   

                  But as Patrick said, allow VM to modify the VF's MAC address, is a security issue.

                  Do you know whether there any other potential issue except mac spoofing?

                  • 7. Re: Which bonding mode does Intel SR-IOV NIC support?
                    wb_Intel

                    Hi henry_wang,

                     

                    Thank you for the update and good to know the bonding mode works fine.

                     

                    Let me check on you other inquiry about the potential issue.

                     

                    rgds,

                    wb

                    • 8. Re: Which bonding mode does Intel SR-IOV NIC support?
                      ingenium

                      Has this been fixed yet in the Intel driver? I would think that if a vf had spoofchk disabled that it would allow the guest to change the MAC, but it seems to still prevent it.