5 Replies Latest reply on May 23, 2012 8:12 AM by Patrick_Kutch

    82599EB: hundreds of strange MAC addresses are seen on the switch

    alex@zadarastorage.com

      Greeting all,

      we are using 82599EB NICs on stock ubuntu natty, kernel 2.6.38-8 with ixgbe version 3.2.9-NAPI. Each NIC spawns SR-IOV VFs. Some of the VFs are attached to virtual machines with KVM PCI pass-through, and other VFs are used on the physical machine.

       

      At some point, one of the PF-interfaces (eth103, specifically) started advertising hundreds of strange MAC addresses, which are seen on the switch. Here are some of them:

         3   0a:00:00:00:00:00 te0/27dynamic
         3   0a:00:00:00:10:11 te0/27dynamic
         3   0a:00:00:01:85:ae te0/27dynamic
         3   0a:00:00:29:38:39 te0/27dynamic
         3   0a:00:00:77:3e:3e te0/27dynamic
         3   0a:00:03:15:52:ef te0/27dynamic
         3   0a:00:03:15:73:3d te0/27dynamic
         3   0a:00:03:15:a7:a2 te0/27dynamic
         3   0a:00:03:1f:32:34 te0/27dynamic
         3   0a:00:03:1f:3d:34 te0/27dynamic
         3   0a:00:03:1f:4a:9d te0/27dynamic
         3   0a:00:03:1f:61:2d te0/27dynamic
         3   0a:00:03:29:38:61 te0/27dynamic
         3   0a:00:03:29:3e:3e te0/27dynamic
         3   0a:00:03:29:52:ef te0/27dynamic
         3   0a:00:03:29:61:2d te0/27dynamic
         3   0a:00:03:29:73:3d te0/27dynamic
         3   0a:00:03:29:a7:a2 te0/27dynamic
         3   0a:00:03:33:00:00 te0/27dynamic
         3   0a:00:03:33:1b:64 te0/27dynamic
         3   0a:00:03:33:28:31 te0/27dynamic
         3   0a:00:03:33:34:36 te0/27dynamic
         3   0a:00:03:33:35:39 te0/27dynamic
         3   0a:00:03:33:3d:34 te0/27dynamic
         3   0a:00:03:33:3e:3e te0/27dynamic
         3   0a:00:03:33:41:3d te0/27dynamic
         3   0a:00:03:33:4a:9d te0/27dynamic
         3   0a:00:03:33:52:ef te0/27dynamic
         3   0a:00:03:33:61:2d te0/27dynamic
         3   0a:00:03:33:73:3d te0/27dynamic
         3   0a:00:03:33:a7:a2 te0/27dynamic
         3   0a:00:03:33:aa:46 te0/27dynamic
         3   0a:00:03:6c:3e:3e te0/27dynamic
         3   0a:00:03:6c:61:2d te0/27dynamic

      ...

      (there are hundreds more).

       

      We don't have such MAC addresses in our environment. Some debugging with switch port monitoring and Wireshark suggests, that switch receives corrupted packets, parses them wrongly, and collects those non-existing MAC addresses. The result is that the interface in question is unable to successfully send any data to the switch port. (The reverse works, however: interface seems to successfully receive data from the switch).

       

      Unfortunately, after some debugging the server was rebooted and the issue went away.

       

      # Has anyone seen such phenomena or can suggest why it can happen?

      # Is there a way to ask the 82599EB PF to dump its internal state, like what MAC addresses its internal L2 switch sees etc.? This can help in debugging future issues.

       

      Thanks!

      Alex.