8 Replies Latest reply on Dec 1, 2014 6:45 AM by snillguttn

    Intel x540-T2 adapter + ESXi 5.5 = Poor Transmit performance

    NicholasFarmer

      I found an issue and I've narrowed it down to the Intel driver.

       

      Just to show the work I've put into this:

      - I used iperf to test the bandwidth on all server types (Physical Linux install and inside a VM)

      - I have an existing VMware support ticket and we've worked it down to the driver.

      - I tested the issue with three different x540-T2 adapters and a x520-DA2 adapter.

       

      Problem: When a server with the intel x540-T2 adapter is connected at 10Gbps and tries to transmit to another server that is connected at 1Gbps, All flows out of the 10Gbps adapter drops to a max of 1Gbps until the 1Gbps flows stop.

      Let me clarify a bit:

       

      Server A has an Intel x540-T2 adapter and is connected at 10Gbps.

      Server B has an Intel x540-T2 adapter and is connected at 10Gbps.

      Server C has a HP Server adapter and is connected at 1Gbps.

       

      I start a Transmit flow from Server A to Server B and I see the iperf session reach full line rate (10Gbps)

      I start a concurrent iperf flow From server A to Server C.

      My expectation is that Server A will continue to transmit at 10Gbps (total), Server B will drop down 1Gbps to (8-9Gbps) to allow bandwidth for the 1Gbps flow to Server C.

      BUT, this does not happen.

      Server A will flop down to 1.5Gbps transmit and Server B and Server C will now receive 500-700Mbps until the flow to Server C stops then the flow from Server A to Server B goes back to 10Gbps.

       

      Driver data:

      If I use driver version 3.7.13.7  this problem does not exist!  Everything works as expected and Server A continues to transmit at 10Gbps while Server B drops to 8Gbps+ and Server C receives at a full 1Gbps.

      If I Update to any version after that, it has the transmit flop.

      Driver Version 3.7.13.7 is the default driver used in the ESXi 5.5 build 1331820 install ISO.

       

      Newer version tested: (these are listed as compatible with ESXi 5.5 on the VMware compatibility guide)

      3.15.1.8

      3.18.7

      3.19.1

       

      Some extra info from ESXi cli:

      ###### This is the working Driver:########

       

      ~ # vmkchdev -l | grep vmnic
      0000:01:00.0 8086:1528 8086:001a vmkernel vmnic0
      0000:01:00.1 8086:1528 8086:001a vmkernel vmnic1

       

      ~ # ethtool -i vmnic0
      driver: ixgbe
      version: 3.7.13.7.14iov-NAPI
      firmware-version: 0x80000389
      bus-info: 0000:01:00.0

       

      ~ # esxcfg-nics -l
      Name    PCI           Driver      Link Speed     Duplex MAC Address       MTU    Description
      vmnic0  0000:01:00.00 ixgbe       Up   10000Mbps Full   a0:36:9f:01:b6:28 9000   Intel Corporation Ethernet Controller X540-AT2

       

      ###### This is one of the broken Driver versions:########

      ~ # vmkchdev -l | grep vmnic
      0000:01:00.0 8086:1528 8086:0001 vmkernel vmnic0
      0000:01:00.1 8086:1528 8086:0001 vmkernel vmnic1

       

      ~ # ethtool -i vmnic0
      driver: ixgbe
      version: 3.15.1.8iov
      firmware-version: 0x80000389
      bus-info: 0000:01:00.0

       

      ~ # esxcfg-nics -l
      Name    PCI           Driver      Link Speed     Duplex MAC Address       MTU    Description
      vmnic0  0000:01:00.00 ixgbe       Up   10000Mbps Full   a0:36:9f:01:b6:28 9000   Intel Corporation Ethernet Controller X540-AT2

       

      Things to note here are the VID and DID are still being detected correctly as 8086 and 1528.

      Any thoughts or questions?

      I have a ton of screen shots if needed to prove the case.