5 Replies Latest reply on Mar 6, 2013 8:53 PM by Patrick_Kutch

    SR-IOV on NIC 82599: cannot dom0 cannot ping local vm (and vice-versa)

    Raymond Kopa

      Hello all,

      I have a server with a 82599 NIC with 2 ports, I have created 31 VF's per port, 62 in total (for some reason I can't get more than this number, but that's not the problem). From each of these port I pick 1 VF that I assign to my guest VM.

      At the guest VM level, I create a bonding, vlan interface and that works well when i try to get in or get out from the outside or from the dom0. No problem.

       

      Okay, now i need to change my MTU to 9K, so i figured that I'd rather install the latest ixgbe and ixgbevf drivers (on dom0 and the vm respectively). Before changing the MTU, everything still works pretty well. Now, I change the MTU on the guest VM, unfortunately i had to also change the MTU of the dom0 PF  to at least the same size, so 9K. My first question is: why would i have to change the PF MTU size since the VF should be independant?

       

      Anywyas, I change the PF as well as the VF, and I can ping any combination: local vm to local vm; local vm to remote vm; local vm to remote server; etc... but there is one combination that doesn't work: local vm to / from dom0 !

      I checked the arp table and sniffed the frames/packets and this is what i observed when vm tried to ping local dom0:

       

      preliminary arp request is sent, vm1 says:

      17:14:54.639029 arp who-has dom0-ip-address tell vm1-ip-address


      Dom0 sees that request and replies:

      17:16:19.884462 arp who-has dom0-ip-address tell vm1-ip-address

      17:16:19.884489 arp reply dom0-ip-address is-at 00:1b:21:d9:64:4c (oui Unknown)


      meanwhile, dom0 received the rquest so it updates its arp table:

      [root@dom0 ~]# arp -a

      ? (vm1-ip-address) at 72:b5:06:8f:b4:ae [ether] on sp

       

      However, the reply seems to never reach the vm1, its arp table remains incomplete:

      [root@vm1 ~]# arp -a

      ? (dom0-ip-address) at <incomplete> on bond0

       

      Trying the other way around gives the same result. Forcing the mac address doesn't help.

       

      Any idea how to make this work?

      And more importantly: is MTU > 1500 even supported for this 82599 NIC?

       

      Thanks

      Ray.