Here are some thoughts that might help narrow things down.
Is the slot a PCI-x 64-bit slot? Is the PCI slot speed set to a lower speed? Do you have another slot you can try to see if another slot makes a difference.
What kind of team and how many and what ports are you teaming with? Have you tried a different team mode to see if that makes any difference?
What model server motherboard do you have? You might be running into a system bottleneck. Are separate interrupts and system resources assigned to each port?