8 Replies Latest reply on Oct 23, 2012 12:45 AM by Raymond Kopa

    SR-IOV, can I activate VF's on interface already used?

    Raymond Kopa

      Hello,

      For a couple of days, I have been trying to set up SR-IOV on my system, but so far i have been unsuccessful.

      I manage to create the VF's, assign them to the VM, then, when i boot the VM, if i try to do ifconfig ethX up, the OS goes directly into Kernel Panic.

      Note: It also goes into panic mode if, from the host, I remove the module "ixgbevf" with rmmod ... strange.

       

      In my current setup, i have one "Intel 82599 10 Gigabit Ethernet NIC" which has 2 ports: eth0 and eth1. The eth0 is connected to a switch, say switch0, eth1 is connected to switch1. switch1 is the standby of the active switch0. And at the OS level, eth0 and eth1 are forming my bond0.

      My hypervisor is OVM 303, i guess it's Xen 4.x running 2.6 xen-enabled kernel (that allows to load pciback module, i've also set the iommu and msi). The VM is a PVM with Redhat 5.6 (everything is 64 bits).

       

      My question is: is it a problem that the VF that i assign to my VM is from an eth that is part of a bond? In which case, do i have to isolate the eth from the bond and assign the related VF to the VM?

      More generally, is it a problem that the eth that exposes VF to VM is somewhere else used (bond, or just have an ip address) ; or shall it be entirely available so there's no conflict?

      If that is allowed, any idea of what can cause this kernel panic: driver ixgbe on the hypervisor, ixgbevf on the host, kernel version?

       

      Thanks,

      Ray

        • 1. Re: SR-IOV, can I activate VF's on interface already used?
          Patrick_Kutch

          Hi Ray,

           

          Thanx for posting to our site.

           

          The assignment of a VF that is on a PF which is part of a bond will not cause the error you are explaining.  I actually posted a whitepaper just a few months ago about bonding and SR-IOV:

          http://communities.intel.com/community/wired/blog/2012/06/25/latest-flexible-port-partitioning-paper-is-now-available-learn-about-qos-and-sr-iov

           

          You may find that useful.

           

          Regarding your issue - OVM is of course from Oracle.  They have their own version of device drivers (PF).  I would make sure you have the latest and greatest PF driver.  In addition 5.6 is pretty old - if memory serves there were some kernel updates required for SR-IOV.  Please make sure it is as up to date as can be, and get the latest drivers from sourceforge.

           

          Have you tried a newer release lik Redhat 6.2?  Just trying to narrow down the number of variables.

           

          thanx,

           

          Patrick

          1 of 1 people found this helpful
          • 2. Re: SR-IOV, can I activate VF's on interface already used?
            Raymond Kopa

            Hello Patrick,

             

            Thanks for your answer - really appreciated.

            I will try with a newer version of guest: 6.2. If I am still unsuccessful, i will update the ixgbe drivers (is it what you refer as PF driver?).

             

            I will keep you posted.

            Thanks,

            Ray

            • 3. Re: SR-IOV, can I activate VF's on interface already used?
              Patrick_Kutch

              Great, looking forward to seeing how it works out.

               

              Quick decoder:

              ixgbe = 10G PF Driver

              ixgbvf = 10G VF Driver

              • 4. Re: SR-IOV, can I activate VF's on interface already used?
                Raymond Kopa

                Hello Patrick,

                No luck ...

                The OEL6.2 which has a Unbreakable kernel, does not allow me to see the device now.

                It is removed from the list of "assignable devices" when the VM was booted, but cannot see at the VM level.

                It seems on my OEL 56 that I can see it only if i boot on a Xen kernel... strange.

                 

                Any idea?

                • 5. Re: SR-IOV, can I activate VF's on interface already used?
                  Raymond Kopa

                  Same issue when i installed the latest driver PF:

                   

                  # ethtool -i eth0

                  driver: ixgbe

                  version: 3.11.33

                  firmware-version: 0x27e30001

                  bus-info: 0000:19:00.0

                   

                  # ethtool -i eth1

                  driver: ixgbe

                  version: 3.11.33

                  firmware-version: 0x27e30001

                  bus-info: 0000:19:00.1

                  1 of 1 people found this helpful
                  • 6. Re: SR-IOV, can I activate VF's on interface already used?
                    Patrick_Kutch

                    That looks to be the latest.  I assume you also updated the VM (assuming you tried a newer release of RedHat).

                     

                    Most likely something with your hypervisor.  Unfortunately I don't have any experience with Oracle's VMM and their drivers, while based on ours are owned by them.

                     

                    Hopefully somebody else in the community has an idea.  In the meantime - you might try using Red Hat KVM as the VMM for testing  purposes to provide a data point.

                     

                    - Patrick

                    • 7. Re: SR-IOV, can I activate VF's on interface already used?
                      Patrick_Kutch

                      Seems if our blog is having issues today.  I'll try to re-post my reply.

                       

                      You have the latest PF (ixgbe) driver.  Did you try a different VM with the latest VF(ixgvf) driver?

                       

                      I've no experience with Oracle's VMM, and they own their version of the rivers.  I'd try using a known solution - such as RHEL 6.2 KVM and see if that works in order to add a data point.

                       

                      Perhaps somebody else in the community has more information as well.

                       

                      - Patrick

                      • 8. Re: SR-IOV, can I activate VF's on interface already used?
                        Raymond Kopa

                        Hello Patrick,

                         

                        On the same hypervisor, i finally managed to get this pci device visible.

                        I used a more recent linux distro version which has a more recent kernel (2.6.39). This distro comes with an updated version of the VF driver (2.0).

                        thanks for your help! I'm halfway there!

                         

                        Now i am trying to configure the network, and I encounter more issues to use the VF's...

                        Let me recap my current configuration: 1 server with 1 82599 NIC (2 ports: eth0 and eth1), this server:

                        - runs the Xen hypervisor (4.1.2) with ixgbe driver

                        - the NIC mentioned above has 4 VF's per PF, total is 8 VF assignable to my VM

                        - the 2 PF's: eth0 and eth1 are in a bond group (bond0) in active/standby mode

                        - the Xen hypervisor has 3 virtual bridges, each one of them on a different VLAN - the VM's can be attached to one or more of these bridges in order to participate in the VLAN traffic

                        - the virtual bridges use bond0 to "go outside" of the server

                        - the VM is a PVM with 2.6.39 kernel and 2.x ixgbevf driver, it has 1 PCI device successfully assigned

                        However, when I configure the link, it's fine, the module gets loaded in the kernel, but the VF doesn't seem to "go out" (ping to the hypervisor on the same subnet doesn't work).

                         

                        I've read the docs from this website and it seems actually fine to use a VF which PF is part of an active bond, so i'm a bit lost again.

                         

                        Next step will be to break the bond and remove the virtual switches (that are not used anymore since I use the VF), and assign an IP address on the same subnet as the PF eth0, in order to simplify my network setup. Any other suggestion?

                         

                        Thanks.