1 2 Previous Next 16 Replies Latest reply on Apr 3, 2013 9:19 AM by Patrick_Kutch

    SR-IOV vs nPAR

    Kristoffer Egefelt

      Hi,

       

      For a XEN environment I have doubts if SR-IOV will provide the required features compared to nPAR.

      Servers will be Dell M620 with either Intel X520 or Broadcom 57711, running Linux kernel 3.8, XEN 4.2 and openvswitch 1.7

       

      I need separate networks on the hypervisor:

      - Management

      - Storage

      - Openvswitch

      - Passthrough for VMs

       

      So my questions are:

      - Can I use the PF's on the hypervisor?

      - Can I use 4 VFs on the hypervisor and the rest for VM passthrough?

      - Can I bond VFs with LACP?

      - Can I use a VF as a iSCSI HBA?

      - Is DCB supported in above configuration?

      - Can I guarantee 1 VF 50% and let the rest use the default round robin fairness?

       

      Hope someone is able to help me with answers or documentation, as I have had difficulties finding it.

       

      Thanks.

       

      Regards

      Kristoffer

        • 1. Re: SR-IOV vs nPAR
          Patrick_Kutch

          Thanx for posting to our site.

           

          SR-IOV and nPAR are technologies that provide similar capabilities, however they do so in different ways.  SR-IOV is of course a PCI SIG standard, while nPAR is specific to a Server OEM both have their strong and weak points.

          Intel Ethernet devices, including the X520 currently do not support nPAR.  The Intel SR-IOV solution is pretty full-featured however and does support most of what you list below.

          We have quite a bit of documentation available that discusses much of what you have listed below,  they are documented in this blog: http://communities.intel.com/community/wired/blog/2012/08/10/all-you-ever-wanted-to-know-about-intel-sr-iov-and-flexible-port-partitioning

           

          Answers to your specific questions:

          Can I use the PF's on the hypervisor? [Patrick] Yes, absolutely


          Can I use 4 VFs on the hypervisor and the rest for VM passthrough?
          [Patrick] Yes, however only on Linux/KVM
          Can I bond VFs with LACP? [Patrick] Yes – I have a paper listed on blog post above specifically on bonding

          Can I use a VF as a iSCSI HBA? [Patrick] Yes, however iSCSI over DCB is not supported
          Is DCB supported in above configuration?
          [Patrick] Yes, DCB is supported in this configuration.  As long as DBC is configured for Hypervisor use and not in VMs.  It’s only supportedin Linux operating systems and will need kernel upgrade and Intel 10GB driver upgrade.

           

          Can I guarantee 1 VF 50% and let the rest use the default round robin fairness?[Patrick]  remember that rate limiting is for Tx side only, in order to limit Rx, you must have a switch limit the bandwidth.  Please see the papers and videos listed on the above blog for ‘FPP’, should give you aclear picture of the technology.

           

          Again, thanx for visiting our site and asking questions!

           

          - Patrick

          1 of 1 people found this helpful
          • 2. Re: SR-IOV vs nPAR
            Kristoffer Egefelt

            Hi Patrick,

             

            Thanks for your answers and links - I read through it and a few things are still unclear.

            Can I use the PF's on the hypervisor? [Patrick] Yes, absolutely

            So do I understand it correctly that two 10G interfaces with 2 VFs each will look like:

            eth0: PF

            eth1: VF

            eth2: VF

            eth3: PF

            eth4: VF

            eth5: VF

             

            Can I use 4 VFs on the hypervisor and the rest for VM passthrough? [Patrick] Yes, however only on Linux/KVM

            You don't mention XEN - is there a reason that only KVM is supported?

            As I understand it, I can hide certain VFs from the hypervisor and use them on the VMs?

            In the configuration examples I can find, ixgbevf is blacklisted in the hypervisor, which make me think that the hypervisor cannot use the VFs? Do you know if this is the issue with XEN?

             

            Can I bond VFs with LACP? [Patrick] Yes – I have a paper listed on blog post above specifically on bonding

            Reading http://www.intel.com/content/www/us/en/network-adapters/10-gigabit-network-adapters/config-qos-with-flexible-port-partitioning.html it seems that LACP is not working due to some antispoofing issue - do you know if this is solved?

             

            Can I use a VF as a iSCSI HBA? [Patrick] Yes, however iSCSI over DCB is not supported

            Does this mean that I need to run iSCSI on the PF if I want to use DCB, or is DCB not supported at all?

             

            Is DCB supported in above configuration? [Patrick] Yes, DCB is supported in this configuration.  As long as DBC is configured for Hypervisor use and not in VMs.  It’s only supportedin Linux operating systems and will need kernel upgrade and Intel 10GB driver upgrade.

            Now I'm confused ;-) So DCB is supported on a VF if a use it on the hypervisor?

             

            Can I guarantee 1 VF 50% and let the rest use the default round robin fairness?[Patrick]  remember that rate limiting is for Tx side only, in order to limit Rx, you must have a switch limit the bandwidth.  Please see the papers and videos listed on the above blog for ‘FPP’, should give you aclear picture of the technology.

            Reading http://www.intel.com/content/www/us/en/network-adapters/10-gigabit-network-adapters/config-qos-with-flexible-port-partitioning.html it appears that I can only limit bandwidth on interfaces, not guarentee a specific interface a certain amount of bandwidth, unless limiting all the other interfaces?

            As I understand it nPAR can provide bandwidth guarentees pr interface - does SR-IOV not have this option?

            Would DCB handle this?

             

            Thanks

             

            Regards

            Kristoffer

            • 3. Re: SR-IOV vs nPAR
              Patrick_Kutch

              Kristoffer - I haven't forgotten about you.  Still digging into your questions.

               

              - Patrick

              • 4. Re: SR-IOV vs nPAR
                Kristoffer Egefelt

                Great - thanks Patrick.

                • 5. Re: SR-IOV vs nPAR
                  Patrick_Kutch

                  Kristoffer,

                   

                  Let's see if I can answer all your questions one by one.

                   

                  To be clear, there is a difference between Xen and XenServer.  You can do everything on Xen that you can do on KVM - although it is done differently for some things.  XenServer is a flavor of Xen that doesn't support all these goodies.  For example iSCSI is not support by XenServer, but is in OpenSource Xen.

                   

                  You don't want to blacklist ixgbevf, because that will (as you found out) not load the VF driver within the hyervisor.  We have a whitepaper that discusses how to setup VF's for both use with a VM and within the hypervisor that should help you out:  http://www.intel.com/content/www/us/en/network-adapters/10-gigabit-network-adapters/ethernet-x520-suse-linux-xen-tech-brief.html

                   

                  The problem with LACP has been fixed in the latest release of the iproute2 utility (that has not yet made it into the distros from what I understand).  You need to update the utility itself, along with the driver and the kernel.  Use the spoofchk parameter to disable anti-spoofing.

                   

                  Regarding DCB - DCB is not supported in a VF, only in the PF.

                   

                  Lastly, on the topic of bandwidth guarantees.  As you surmised, you can only guarantee a VF to have a guaranteed bandwidth by rate limiting all other VF's.  This is true no matter what kind of partitioning technology you choose.  Flex-10, NPAR or SR-IOV all have to limit the other partitions to ensure bandwidth reservations.  There is only 10G to go around, so if you want to guarantee say 5Gbps to one partition (VF, NPAR, Flex-10 ports) then all the other partitions must never exceed a combined 5Gbps limit and therefore must be limited.

                   

                  You can do a more flexible type of rate limiting using some kind of chron job or daemon if you wanted to get fancy, where you monitor all the other VF's and if the combined bandwidh exceeds, or nears the 5Gbps AND the partition you want to have guaranteed BW for is  using 5Gbps then you can rate limit the other VF's.  This is just an idea, I've not actually done it in a daemon or anything, just manually tweaking with the gui I created for the demo videos.

                   

                  Hope this helps, and thanx for posting!

                   

                  - Patrick

                  1 of 1 people found this helpful
                  • 6. Re: SR-IOV vs nPAR
                    Kristoffer Egefelt

                    Thanks Patrick, then only the QOS problem remains.

                     

                    This is from Qlogic nPAR documentation, stating that bandwidth guarantee is supported and is dynamic across the interfaces. Or am I misunderstanding this?


                    Q: Is one virtual port’s unused bandwidth available for use by other active virtual ports?

                    Yes. The minimum settings are bandwidth guarantees, specified as a percentage of the link speed. If one or more virtual ports aren’t consuming their full allotment, that bandwidth can be temporarily consumed by other virtual ports if they need more than their guaranteed allotment.


                    Also, if the VF is hidden from the hypervisor, I don't see how I can rate limit the traffic outside the VM, which makes it very difficult to guarantee bandwidth to a interface in the hypervisor for storage etc.

                    Is there a way to rate limit from the hypervisor?

                     

                    Thanks for your answers!

                     

                    • 7. Re: SR-IOV vs nPAR
                      Patrick_Kutch

                      Kristoffer,

                       

                      Thanx for your great questions!

                       

                      We do not have a mechansim to automagically do as the Qlogic nPAR documentation claims.  All these technologies have their quirks and issues.  With the Intel SR-IOV solution you can do as you are asking, however you would need to do so with a daemon or something like that you wrote yourself.

                       

                      In regards to being able to manipulate the VF from the hypervisor even though the VF itself is not available to the hypervisor - this is true, however the way you control the VF from the hypervisor is via the PF that owns the VF.  So you specify the PF and then the VF # in order to say rate limit, or assign a VLAN etc.

                       

                      regards,

                       

                      Patrick

                      • 8. Re: SR-IOV vs nPAR
                        tsukser

                        Hi Patrick,

                         

                        You mentioned here that LACP should be supported with SR-IOV, however, I was unable to get it configured properly. I'm running Ubuntu 12.04 64-bit server with latest patches (kernel 3.2.0-39, ixgbe-3.13.10 and iproute_20121211 - Debian release). I'm trying to configure a bond with 2 interfaces in LACP bundle and that works perfectly without SR-IOV and when I bridge virtual machine to this bond. When I turn on SR-IOV, I'm still able to use this bond normally until I get a VM started (KVM QEMU 1.0, libvirt 0.9.8) and try to configure LACP inside it. At that point, connectivity just gets totally broken, like in typical scenario when server and router don't have same LACP configuration set. I've also turned off spoof checking through ip utility. Is there something so obvious what I'm missing?

                         

                        Could you please explain how did you make it to work, which versions did you use, etc.?

                         

                        Thanks in advance!

                        Tomislav

                        • 9. Re: SR-IOV vs nPAR
                          Patrick_Kutch

                          Tomislav,

                           

                          Please explain what all is in your bridge, and where the bridge is - is it in the VM or the hypervisor.  And what all is in the bridge (what interfaces).

                           

                          thanx,

                           

                          Patrick

                          • 10. Re: SR-IOV vs nPAR
                            tsukser

                            I have a router configured LAG 802.3ad aggregation on those 2 ports (eth4 and eth7) on the server, a part from /etc/network/interfaces on Ubuntu:

                             

                            auto eth7

                            iface eth7 inet manual

                            bond-master bond1

                             

                            auto eth4

                            iface eth4 inet manual

                            bond-master bond1

                             

                            auto bond1

                            iface bond1 inet manual

                            bond-mode 802.3ad

                            bond-miimon 100

                            bond-lacp-rate 1

                            bond-slaves none

                             

                            Additionally I created a bridge br1 which has bond1 attached to itself and ports from virtual machines (hypervisor is KVM and ports are simply bridged). This setup works well without SR-IOV. It also works fine when I enable SR-IOV, but before I attach virtual machine to VFs.

                             

                            When I attach a virtual machine to VFs (Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)) of eth4 and eth7 and configure the same bond in the VM, the whole connectivity (to host and to the VMs) is broken. When I don't configure LACP (bond1 and br1) on the host and use VFs in the virtual machine, the connectivity towards VM is broken again. And I've doublechecked the setting for spoof checking - it's off:

                            eth7:

                            ...

                            vf 14 MAC 96:9e:b8:06:0c:2c, spoof checking off

                             

                            Thanks!

                            T.

                            • 11. Re: SR-IOV vs nPAR
                              Kristoffer Egefelt

                              Thanks Patrick,

                               

                              In case of problems with LACP over VFs, can I just use LACP over the PFs, while using the VFs for other tasks?

                              I'm having difficulties understanding how LACP could work over multiple pairs of VFs at the same time - do you know if this is possible?

                               

                              Regards

                              Kristoffer

                              • 12. Re: SR-IOV vs nPAR
                                Patrick_Kutch

                                Kristoffer,

                                 

                                Sorry for the delay - took a few days off.

                                 

                                In the paper written on this topic, I believe we recommend not doing teaming at the PF level.  For load balancing, you will definately want to load balance between VF's from different PF's (or even different physical NICs if you like).  Load balancing between VF's from the same PF won't do you any good, because it would end up using the same physical port (PF).

                                 

                                Does that help?

                                 

                                - Patrick

                                • 13. Re: SR-IOV vs nPAR
                                  Kristoffer Egefelt

                                  Hi Patrick,

                                   

                                  OK - teaming/bonding must be on VFs.

                                  Also I understand that bonds need VFs from different PFs to work.

                                   

                                  What I don't understand is if it's possible to have 2 LACP bonds on the same PFs, like this:

                                   

                                  PFs are eth0 and eth3

                                  eth1+eth4: LACP

                                  eth2+eth5: LACP

                                   

                                  Is this doable ?

                                   

                                  Thanks - this will be the last piece of info for now ;-)

                                   

                                  Regards

                                  Kristoffer

                                  • 14. Re: SR-IOV vs nPAR
                                    Patrick_Kutch

                                    Hi Kristoffer,

                                     

                                    You can assign two VF's from the same PF to a VM and team they anyway you wish, nothing to prevent you from that.  However  you probably won't gain anything.

                                     

                                    You would be load balancing from the same physical port, so there wouldn't really be any balancing occuring, it's all coming and going to the same cable.

                                     

                                    I hope this helps.

                                     

                                    - Patrick

                                    1 2 Previous Next