11 Replies Latest reply on Dec 4, 2016 7:25 PM by Intel Corporation

    Intel X520 82599 sr-iov with bridge issues

    PRoland

      Hello,
      We have a dual nic x520 on a hypervisor, and planning to use sr-iov and bridge as well.
      Nics VF are working fine, however there is also a bridge (on a tagged vlan) on that host and the machines behind sr-iov vf cannot talk to machine in the bridge.
      I turned off mac spoofing to no avail....

      This is Centos 7.2 with all updates.
      The reason for this is that not all machines require sr-iov functions/maintenance so some of them are used classically with a vnet on bridge.
      Both machines (vf and bridged) are in the same vlan, I did a tcpdump session and I can see it does an ARP request but no reply


      Any help would be appreciated, thank you

        • 1. Re: Intel X520 82599 sr-iov with bridge issues
          Intel Corporation
          This message was posted on behalf of Intel Corporation

          Hi PRoland,

                     Thank you for the post. I will check and update you if there is any finding.

          rgds,
          wb
           

          • 2. Re: Intel X520 82599 sr-iov with bridge issues
            PRoland

            Hello, I managed to narrow it down and found a fix.
            Apparently this solves it for the moment:

             

            bridge fdb add 00:10:DB:FF:10:01 dev vlan102

             

            But is there a temp fix for this as it's being lost on each reboot...

            I can put an rc for it but it's difficult to maintain

            • 3. Re: Intel X520 82599 sr-iov with bridge issues
              Intel Corporation
              This message was posted on behalf of Intel Corporation

              Hi PROland,

               Thank you for sharing the information. I will further check for you. 

              rgds,
              wb
               

              • 4. Re: Intel X520 82599 sr-iov with bridge issues
                Intel Corporation
                This message was posted on behalf of Intel Corporation

                Hi Proland,

                  Good day. Can you help provide additional info of the driver used (ixgbe, vf) and tcp dump?

                Rgds,
                wb
                 

                • 5. Re: Intel X520 82599 sr-iov with bridge issues
                  Intel Corporation
                  This message was posted on behalf of Intel Corporation

                  Hi Proland,

                  Good day. Please  help provide additional requested on my previous post.: info of the driver used (ixgbe, vf) and tcp dump. Thanks. 

                  Rgds,
                  wb

                  • 6. Re: Intel X520 82599 sr-iov with bridge issues
                    PRoland

                    I do not understand how will that help in this case.
                    This MAC spoofing is a very wide issue, across all platforms. In Linux at least I was able to remove it from the driver sourcecode, but vmware was not an option.
                    Basically because security the module denies any mac change on the vf (guest) which would be ok but not in case of an active-backup bond where the second nic has the mac rewritten.
                    I did look closely on other topics and see the same type of questions, to put the admins to work a bit more and maybe avoid the problem so people will eventually give up because of the extensive procedures to get an answer that Intel clearly knows.

                    To rephrase the question: is there a simple way to ***DISABLE*** the mac spoofing function ***GLOBALLY*** because when you have many virtual machines and provisioned as well setting spoofing to off for each vf is complicated.

                    I appreciate Intel's concern for security but I don't appreciate the fact that there is no control over that function. Apple style

                    • 7. Re: Intel X520 82599 sr-iov with bridge issues
                      Intel Corporation
                      This message was posted on behalf of Intel Corporation

                      Hi Proland,

                       Thank you for the clarification. Let me further investigate.

                      Rgds,
                      wb
                       

                      • 8. Re: Intel X520 82599 sr-iov with bridge issues
                        Intel Corporation
                        This message was posted on behalf of Intel Corporation

                        Hi Proland,

                         Please refer to below information hopefully can be of help:
                        1) Adding MAC address to the bridge forwarding database is correct approach.
                           You can add the following command to /etc/rc.d/rc.local file so it will always take effect upon reboot. 

                            bridge fdb add 00:10:DB:FF:10:01 dev vlan102 

                        2) Spoof Checking can be turned off for each VF. Use the following command to disable Spoof Checking. 
                        #ip link set vf spoofchk on|off 
                        Example: #ip link set eth2 vf 0 spoofchk off <-- Assuming Eth2 is X520 Port and Spook Checking is being disabled for VF 0. 

                        3) You can add the above command to /etc/rc.d/rc.local file so it will always take effect upon reboot. 

                        4) Intel drivers use industry standard practices to implement security features.These features and their respective configuration behavior implementation depends on Linux community consensus. 
                        In case Spoof Checking cannot be disabled on VMware ESX products, you need to contact VMware to request Spoof check enable/disable feature. 


                        Thanks,
                        wb

                        • 9. Re: Intel X520 82599 sr-iov with bridge issues
                          Intel Corporation
                          This message was posted on behalf of Intel Corporation

                          Hi Proland,

                             Please feel free to update me if you have tried the suggestion.

                          rgds,
                          wb
                           

                          • 10. Re: Intel X520 82599 sr-iov with bridge issues
                            PRoland

                            Hi, the solutions proposed does work, but cause many issues in a fully virtualized  environment.

                             

                            1. Cannot change mac on the guest or option to set that system wide is a problem creating a normal active-failover link in guest. Disabling the mac spoofing per vf requires scripting to do that for each vm before startup or allocation of resources and also determine which vf needs the option (standby nic)

                            2. The vfs unable to communicate with bridge might be an linux issue, no intel, but again altering the bridge database after each failover requires even more scripting than above.

                             

                             

                            Both issues can be easily solved in driver, but they were not although on a simple google search I get many many pages with this. It's even worse in vmware.

                            We did however find a simple solution to all these. We ditched the X520 and got Broadcom/Qlogic/Avago 57810s instead.

                            No offense.

                            • 11. Re: Intel X520 82599 sr-iov with bridge issues
                              Intel Corporation
                              This message was posted on behalf of Intel Corporation

                              Hi Proland,

                               Thank you for the update and sorry to know this caused some issue. 

                               For the meantime, any help needed still?

                              rgds,
                              wb