Thanx for posting to our forum. I encountered the same issue you are explaining when I wrote the FPP papers. I made a mental note to go figure out what the issue was, however I changed jobs before I ever root caused it.
I suspect it has to do with the actual network stack running on your server where the VF's are instantiated. I think the stack gets a bit confused because all the VF's are on the same subnet.
My solution for this was to put each VF on it's own subnet, or to put it on its own VLAN. I bet if you try that, you will get the results you are looking for.
In the meantime I will go pester my co-worker who owns the SR-IOV technology and ask him to go do some digging.
We were able to reproduce your issue. Both with SR-IOV and with plain ports.
My expert came back with this:
This behavior has to do with subnet mask, Linux network stack and Linux routing. Basically, if you put multiple network interfaces virtual or physical, Linux OS will use the first available interface. Linux makes no guarantees which interface it will use for transmit or receive. It may use one interface for both transmit and receive. It may use one interface for transmit only and another one for receive only. The behavior is unpredictable. It is an OS issue as opposed to a NIC or driver one.
As I indicated earlier, I had encounterd this when I setup the demos for the FPP. My solution was to put the VF's on different subnets, or to put them on different VLANs.
many thanks to you and your expert for your support!
I tried to move the interfaces into different subnets as well as to different VLANs (using vconfig) and tagged the outgoing packets on the the sender with the appropriate VLAN tags by sending them throuch virtual IFs as well.
Unfortunately the problem stays the same.
Maybe you can give me some hint, how you did set up those VLANs and subnets.
To configure ip address on a VF, simply use the ifconfig:
#ifconfig ethx 192.168.100.1 netmask 255.255.255.0
where ethx is the eth device associated with the VF
to set a VLAN for a vf, you use the iproute2 utility on the VF via the PF:
#ip link set eth2 vf 0 vlan 123
This sets VF 0 on the PF assigned to eth2 to 123.
There are more examples all available in my whitepaper: