Ethernet Products
Determine ramifications of Intel® Ethernet products and technologies
4811 Discussions

Intel X520 82599 sr-iov with bridge issues

PPaul15
Novice
3,612 Views

Hello,

 

We have a dual nic x520 on a hypervisor, and planning to use sr-iov and bridge as well.

 

Nics VF are working fine, however there is also a bridge (on a tagged vlan) on that host and the machines behind sr-iov vf cannot talk to machine in the bridge.

 

I turned off mac spoofing to no avail....

This is Centos 7.2 with all updates.

 

The reason for this is that not all machines require sr-iov functions/maintenance so some of them are used classically with a vnet on bridge.

 

Both machines (vf and bridged) are in the same vlan, I did a tcpdump session and I can see it does an ARP request but no reply

 

Any help would be appreciated, thank you
0 Kudos
11 Replies
idata
Employee
1,316 Views

Hi PRoland,

 

 

Thank you for the post. I will check and update you if there is any finding.

 

 

rgds,

 

wb

 

0 Kudos
PPaul15
Novice
1,316 Views

Hello, I managed to narrow it down and found a fix.

 

Apparently this solves it for the moment:

bridge fdb add 00:10:DB:FF:10:01 dev vlan102

But is there a temp fix for this as it's being lost on each reboot...

I can put an rc for it but it's difficult to maintain

0 Kudos
idata
Employee
1,316 Views

Hi PROland,

 

 

Thank you for sharing the information. I will further check for you.

 

 

rgds,

 

wb

 

0 Kudos
idata
Employee
1,316 Views

Hi Proland,

 

 

Good day. Can you help provide additional info of the driver used (ixgbe, vf) and tcp dump?

 

 

Rgds,

 

wb

 

0 Kudos
idata
Employee
1,316 Views

Hi Proland,

 

 

Good day. Please help provide additional requested on my previous post.: info of the driver used (ixgbe, vf) and tcp dump. Thanks.

 

 

Rgds,

 

wb
0 Kudos
PPaul15
Novice
1,316 Views

I do not understand how will that help in this case.

 

This MAC spoofing is a very wide issue, across all platforms. In Linux at least I was able to remove it from the driver sourcecode, but vmware was not an option.

 

Basically because security the module denies any mac change on the vf (guest) which would be ok but not in case of an active-backup bond where the second nic has the mac rewritten.

 

I did look closely on other topics and see the same type of questions, to put the admins to work a bit more and maybe avoid the problem so people will eventually give up because of the extensive procedures to get an answer that Intel clearly knows.

 

 

To rephrase the question: is there a simple way to ***DISABLE*** the mac spoofing function ***GLOBALLY*** because when you have many virtual machines and provisioned as well setting spoofing to off for each vf is complicated.

I appreciate Intel's concern for security but I don't appreciate the fact that there is no control over that function. Apple style

0 Kudos
idata
Employee
1,316 Views

Hi Proland,

 

 

Thank you for the clarification. Let me further investigate.

 

 

Rgds,

 

wb

 

0 Kudos
idata
Employee
1,316 Views

Hi Proland,

 

 

Please refer to below information hopefully can be of help:

 

1) Adding MAC address to the bridge forwarding database is correct approach.

 

You can add the following command to /etc/rc.d/rc.local file so it will always take effect upon reboot.

 

 

bridge fdb add 00:10:DB:FF:10:01 dev vlan102

 

 

2) Spoof Checking can be turned off for each VF. Use the following command to disable Spoof Checking.

 

# ip link set vf spoofchk on|off

 

Example: # ip link set eth2 vf 0 spoofchk off <-- Assuming Eth2 is X520 Port and Spook Checking is being disabled for VF 0. <p> 

 

3) You can add the above command to /etc/rc.d/rc.local file so it will always take effect upon reboot.

 

 

4) Intel drivers use industry standard practices to implement security features.These features and their respective configuration behavior implementation depends on Linux community consensus.

 

In case Spoof Checking cannot be disabled on VMware ESX products, you need to contact VMware to request Spoof check enable/disable feature.

 

 

 

Thanks,

 

wb
0 Kudos
idata
Employee
1,316 Views

Hi Proland,

 

 

Please feel free to update me if you have tried the suggestion.

 

 

rgds,

 

wb

 

0 Kudos
PPaul15
Novice
1,316 Views

Hi, the solutions proposed does work, but cause many issues in a fully virtualized environment.

1. Cannot change mac on the guest or option to set that system wide is a problem creating a normal active-failover link in guest. Disabling the mac spoofing per vf requires scripting to do that for each vm before startup or allocation of resources and also determine which vf needs the option (standby nic)

2. The vfs unable to communicate with bridge might be an linux issue, no intel, but again altering the bridge database after each failover requires even more scripting than above.

Both issues can be easily solved in driver, but they were not although on a simple google search I get many many pages with this. It's even worse in vmware.

We did however find a simple solution to all these. We ditched the X520 and got Broadcom/Qlogic/Avago 57810s instead.

No offense.

0 Kudos
idata
Employee
1,316 Views

Hi Proland,

 

 

Thank you for the update and sorry to know this caused some issue.

 

 

For the meantime, any help needed still?

 

 

rgds,

 

wb

 

0 Kudos
Reply