0 Replies Latest reply on Aug 26, 2016 6:26 AM by JensAuer

    Reordering caused by Flow Director

    JensAuer

      Hi,
       
        we are experiencing packet reordering and thus bandwidth drops due to retransmissions on our servers. We are using RedHat 7.2 on HP Servers connected to a HP Aruba 1920 10GB switch. I can easily reproduce the issues by running iperf3 between the two servers. I have found a description of problems caused by Flow Director and migrating processes from Fermilab (https://arxiv.org/pdf/1106.0443.pdf) together with a support case at RedHat (https://access.redhat.com/solutions/2403071). What is offical position concerning this issue? Is it considered as bug?

       

      An iperf test shows the retransmissions every couple of seconds:

       

      iperf3 -c 192.168.120.11 -p 12345 -b200m -P20 -t0
        Connecting to host 192.168.120.11, port 12345
        [  4] local 192.168.120.10 port 59638 connected to 192.168.120.11 port 12345
        [  6] local 192.168.120.10 port 59640 connected to 192.168.120.11 port 12345
        [  8] local 192.168.120.10 port 59642 connected to 192.168.120.11 port 12345
        [ 10] local 192.168.120.10 port 59644 connected to 192.168.120.11 port 12345
        [ 12] local 192.168.120.10 port 59646 connected to 192.168.120.11 port 12345
        [ 14] local 192.168.120.10 port 59648 connected to 192.168.120.11 port 12345
        [ 16] local 192.168.120.10 port 59650 connected to 192.168.120.11 port 12345
        [ 18] local 192.168.120.10 port 59652 connected to 192.168.120.11 port 12345
        [ 20] local 192.168.120.10 port 59654 connected to 192.168.120.11 port 12345
        [ 22] local 192.168.120.10 port 59656 connected to 192.168.120.11 port 12345
        [ 24] local 192.168.120.10 port 59658 connected to 192.168.120.11 port 12345
        [ 26] local 192.168.120.10 port 59660 connected to 192.168.120.11 port 12345
        [ 28] local 192.168.120.10 port 59662 connected to 192.168.120.11 port 12345
        [ 30] local 192.168.120.10 port 59664 connected to 192.168.120.11 port 12345
        [ 32] local 192.168.120.10 port 59666 connected to 192.168.120.11 port 12345
        [ 34] local 192.168.120.10 port 59668 connected to 192.168.120.11 port 12345
        [ 36] local 192.168.120.10 port 59670 connected to 192.168.120.11 port 12345
        [ 38] local 192.168.120.10 port 59672 connected to 192.168.120.11 port 12345
        [ 40] local 192.168.120.10 port 59674 connected to 192.168.120.11 port 12345
        [ 42] local 192.168.120.10 port 59676 connected to 192.168.120.11 port 12345
        [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
        [  4]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    2    225 KBytes
        [  6]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    1    202 KBytes
        [  8]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    8    225 KBytes
        [ 10]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    2    132 KBytes
        [ 12]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    2    133 KBytes
        [ 14]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    2    132 KBytes
        [ 16]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec   12    165 KBytes
        [ 18]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    2    151 KBytes
        [ 20]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    1    150 KBytes
        [ 22]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    4    225 KBytes
        [ 24]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    5    143 KBytes
        [ 26]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    3    132 KBytes
        [ 28]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    3    126 KBytes
        [ 30]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    1    127 KBytes
        [ 32]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    6    136 KBytes
        [ 34]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    2    139 KBytes
        [ 36]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    1    154 KBytes
        [ 38]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    1    136 KBytes
        [ 40]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    2    122 KBytes
        [ 42]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    4    139 KBytes
        [SUM]   0.00-1.00   sec   438 MBytes  3.67 Gbits/sec   64
        - - - - - - - - - - - - - - - - - - - - - - - - -
        [  4]   1.00-2.00   sec  24.0 MBytes   201 Mbits/sec    0    255 KBytes
        [  6]   1.00-2.00   sec  24.0 MBytes   201 Mbits/sec    0    243 KBytes
        [  8]   1.00-2.00   sec  24.0 MBytes   201 Mbits/sec    0    256 KBytes
        [ 10]   1.00-2.00   sec  24.0 MBytes   201 Mbits/sec    0    191 KBytes
        [ 12]   1.00-2.00   sec  24.0 MBytes   201 Mbits/sec    0    192 KBytes
        [ 14]   1.00-2.00   sec  24.0 MBytes   201 Mbits/sec    0    192 KBytes
        [ 16]   1.00-2.00   sec  24.0 MBytes   201 Mbits/sec    0    216 KBytes
        [ 18]   1.00-2.00   sec  24.0 MBytes   201 Mbits/sec    0    208 KBytes
        [ 20]   1.00-2.00   sec  24.0 MBytes   201 Mbits/sec    0    205 KBytes
        [ 22]   1.00-2.00   sec  24.0 MBytes   201 Mbits/sec    0    259 KBytes
        [ 24]   1.00-2.00   sec  24.0 MBytes   201 Mbits/sec    0    202 KBytes
        [ 26]   1.00-2.00   sec  24.0 MBytes   201 Mbits/sec    0    198 KBytes
        [ 28]   1.00-2.00   sec  24.0 MBytes   201 Mbits/sec    0    191 KBytes
        [ 30]   1.00-2.00   sec  24.0 MBytes   201 Mbits/sec    0    192 KBytes
        [ 32]   1.00-2.00   sec  24.0 MBytes   201 Mbits/sec    0    201 KBytes
        [ 34]   1.00-2.00   sec  24.0 MBytes   201 Mbits/sec    0    198 KBytes
        [ 36]   1.00-2.00   sec  24.0 MBytes   201 Mbits/sec    0    211 KBytes
        [ 38]   1.00-2.00   sec  24.0 MBytes   201 Mbits/sec    0    198 KBytes
        [ 40]   1.00-2.00   sec  24.0 MBytes   201 Mbits/sec    0    194 KBytes
        [ 42]   1.00-2.00   sec  24.0 MBytes   201 Mbits/sec    0    198 KBytes
        [SUM]   1.00-2.00   sec   480 MBytes  4.03 Gbits/sec    0
        - - - - - - - - - - - - - - - - - - - - - - - - -
        [  4]   2.00-3.00   sec  23.8 MBytes   199 Mbits/sec    0    267 KBytes
        [  6]   2.00-3.00   sec  23.8 MBytes   199 Mbits/sec    0    250 KBytes
        [  8]   2.00-3.00   sec  23.8 MBytes   199 Mbits/sec    0    269 KBytes
        [ 10]   2.00-3.00   sec  23.8 MBytes   199 Mbits/sec    0    201 KBytes
        [ 12]   2.00-3.00   sec  23.8 MBytes   199 Mbits/sec    0    204 KBytes
        [ 14]   2.00-3.00   sec  23.8 MBytes   199 Mbits/sec    2    202 KBytes
        [ 16]   2.00-3.00   sec  23.8 MBytes   199 Mbits/sec    0    223 KBytes
        [ 18]   2.00-3.00   sec  23.8 MBytes   199 Mbits/sec    0    216 KBytes
        [ 20]   2.00-3.00   sec  23.8 MBytes   199 Mbits/sec    0    215 KBytes
        [ 22]   2.00-3.00   sec  23.8 MBytes   199 Mbits/sec    0    270 KBytes
        [ 24]   2.00-3.00   sec  23.8 MBytes   199 Mbits/sec    0    212 KBytes
        [ 26]   2.00-3.00   sec  23.8 MBytes   199 Mbits/sec    0    208 KBytes
        [ 28]   2.00-3.00   sec  23.8 MBytes   199 Mbits/sec    0    201 KBytes
        [ 30]   2.00-3.00   sec  23.8 MBytes   199 Mbits/sec    0    202 KBytes
        [ 32]   2.00-3.00   sec  23.8 MBytes   199 Mbits/sec    0    211 KBytes
        [ 34]   2.00-3.00   sec  23.8 MBytes   199 Mbits/sec    0    208 KBytes
        [ 36]   2.00-3.00   sec  23.8 MBytes   199 Mbits/sec    0    219 KBytes
        [ 38]   2.00-3.00   sec  23.8 MBytes   199 Mbits/sec    0    209 KBytes
        [ 40]   2.00-3.00   sec  23.8 MBytes   199 Mbits/sec    0    211 KBytes
        [ 42]   2.00-3.00   sec  23.8 MBytes   199 Mbits/sec    0    211 KBytes
        [SUM]   2.00-3.00   sec   475 MBytes  3.99 Gbits/sec    2
        - - - - - - - - - - - - - - - - - - - - - - - - -
        [  4]   3.00-4.00   sec  23.9 MBytes   200 Mbits/sec    5    205 KBytes
        [  6]   3.00-4.00   sec  23.9 MBytes   200 Mbits/sec    0    260 KBytes
        [  8]   3.00-4.00   sec  23.9 MBytes   200 Mbits/sec    0    276 KBytes
        [ 10]   3.00-4.00   sec  23.9 MBytes   200 Mbits/sec    0    221 KBytes
        [ 12]   3.00-4.00   sec  23.9 MBytes   200 Mbits/sec    0    225 KBytes
        [ 14]   3.00-4.00   sec  23.9 MBytes   200 Mbits/sec    0    221 KBytes
        [ 16]   3.00-4.00   sec  23.9 MBytes   200 Mbits/sec    0    243 KBytes
        [ 18]   3.00-4.00   sec  23.9 MBytes   200 Mbits/sec    0    230 KBytes
        [ 20]   3.00-4.00   sec  23.9 MBytes   200 Mbits/sec    0    229 KBytes
        [ 22]   3.00-4.00   sec  23.9 MBytes   200 Mbits/sec    0    281 KBytes
        [ 24]   3.00-4.00   sec  23.9 MBytes   200 Mbits/sec    0    233 KBytes
        [ 26]   3.00-4.00   sec  23.9 MBytes   200 Mbits/sec    0    226 KBytes
        [ 28]   3.00-4.00   sec  23.9 MBytes   200 Mbits/sec    0    222 KBytes
        [ 30]   3.00-4.00   sec  23.9 MBytes   200 Mbits/sec    0    218 KBytes
        [ 32]   3.00-4.00   sec  23.9 MBytes   200 Mbits/sec    0    232 KBytes
        [ 34]   3.00-4.00   sec  23.9 MBytes   200 Mbits/sec    0    229 KBytes
        [ 36]   3.00-4.00   sec  23.9 MBytes   200 Mbits/sec    0    239 KBytes
        [ 38]   3.00-4.00   sec  23.9 MBytes   200 Mbits/sec    0    229 KBytes
        [ 40]   3.00-4.00   sec  23.9 MBytes   200 Mbits/sec    0    226 KBytes
        [ 42]   3.00-4.00   sec  23.9 MBytes   200 Mbits/sec    0    225 KBytes
        [SUM]   3.00-4.00   sec   478 MBytes  4.00 Gbits/sec    5
        - - - - - - - - - - - - - - - - - - - - - - - - -
        [  4]   4.00-5.00   sec  23.8 MBytes   199 Mbits/sec    1    212 KBytes
        [  6]   4.00-5.00   sec  23.8 MBytes   199 Mbits/sec    0    267 KBytes
        [  8]   4.00-5.00   sec  23.8 MBytes   199 Mbits/sec    0    283 KBytes
        [ 10]   4.00-5.00   sec  23.8 MBytes   199 Mbits/sec    0    228 KBytes
        [ 12]   4.00-5.00   sec  23.8 MBytes   199 Mbits/sec    0    232 KBytes
        [ 14]   4.00-5.00   sec  23.8 MBytes   199 Mbits/sec    0    226 KBytes
        [ 16]   4.00-5.00   sec  23.8 MBytes   199 Mbits/sec    0    249 KBytes
        [ 18]   4.00-5.00   sec  23.8 MBytes   199 Mbits/sec    0    236 KBytes
        [ 20]   4.00-5.00   sec  23.8 MBytes   199 Mbits/sec    0    235 KBytes
        [ 22]   4.00-5.00   sec  23.8 MBytes   199 Mbits/sec    0    287 KBytes
        [ 24]   4.00-5.00   sec  23.8 MBytes   199 Mbits/sec    0    239 KBytes
        [ 26]   4.00-5.00   sec  23.8 MBytes   199 Mbits/sec    0    233 KBytes
        [ 28]   4.00-5.00   sec  23.8 MBytes   199 Mbits/sec    0    230 KBytes
        [ 30]   4.00-5.00   sec  23.8 MBytes   199 Mbits/sec    0    232 KBytes
        [ 32]   4.00-5.00   sec  23.8 MBytes   199 Mbits/sec    0    238 KBytes
        [ 34]   4.00-5.00   sec  23.8 MBytes   199 Mbits/sec    0    236 KBytes
        [ 36]   4.00-5.00   sec  23.8 MBytes   199 Mbits/sec    0    247 KBytes
        [ 38]   4.00-5.00   sec  23.8 MBytes   199 Mbits/sec    0    238 KBytes
        [ 40]   4.00-5.00   sec  23.8 MBytes   199 Mbits/sec    0    233 KBytes
        [ 42]   4.00-5.00   sec  23.8 MBytes   199 Mbits/sec    0    232 KBytes
        [SUM]   4.00-5.00   sec   475 MBytes  3.99 Gbits/sec    1
        ^C- - - - - - - - - - - - - - - - - - - - - - - - -
        [  4]   5.00-5.32   sec  9.50 MBytes   251 Mbits/sec    0    212 KBytes
        [  6]   5.00-5.32   sec  9.50 MBytes   251 Mbits/sec    0    267 KBytes
        [  8]   5.00-5.32   sec  9.50 MBytes   251 Mbits/sec    0    283 KBytes
        [ 10]   5.00-5.32   sec  9.50 MBytes   251 Mbits/sec    0    228 KBytes
        [ 12]   5.00-5.32   sec  9.50 MBytes   251 Mbits/sec    0    232 KBytes
        [ 14]   5.00-5.32   sec  9.50 MBytes   251 Mbits/sec    0    226 KBytes
        [ 16]   5.00-5.32   sec  9.50 MBytes   251 Mbits/sec    0    249 KBytes
        [ 18]   5.00-5.32   sec  9.50 MBytes   251 Mbits/sec    0    236 KBytes
        [ 20]   5.00-5.32   sec  9.50 MBytes   251 Mbits/sec    0    235 KBytes
        [ 22]   5.00-5.32   sec  9.50 MBytes   251 Mbits/sec    0    287 KBytes
        [ 24]   5.00-5.32   sec  9.50 MBytes   251 Mbits/sec    0    239 KBytes
        [ 26]   5.00-5.32   sec  9.50 MBytes   251 Mbits/sec    0    233 KBytes
        [ 28]   5.00-5.32   sec  9.50 MBytes   251 Mbits/sec    0    230 KBytes
        [ 30]   5.00-5.32   sec  9.50 MBytes   251 Mbits/sec    0    232 KBytes
        [ 32]   5.00-5.32   sec  9.50 MBytes   251 Mbits/sec    0    238 KBytes
        [ 34]   5.00-5.32   sec  9.50 MBytes   251 Mbits/sec    0    236 KBytes
        [ 36]   5.00-5.32   sec  9.50 MBytes   251 Mbits/sec    0    247 KBytes
        [ 38]   5.00-5.32   sec  9.50 MBytes   251 Mbits/sec    0    238 KBytes
        [ 40]   5.00-5.32   sec  9.50 MBytes   251 Mbits/sec    0    233 KBytes
        [ 42]   5.00-5.32   sec  9.50 MBytes   251 Mbits/sec    0    232 KBytes
        [SUM]   5.00-5.32   sec   190 MBytes  5.02 Gbits/sec    0
        - - - - - - - - - - - - - - - - - - - - - - - - -
        [ ID] Interval           Transfer     Bandwidth       Retr
        [  4]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    8             sender
        [  4]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [  6]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    1             sender
        [  6]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [  8]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    8             sender
        [  8]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 10]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    2             sender
        [ 10]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 12]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    2             sender
        [ 12]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 14]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    4             sender
        [ 14]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 16]   0.00-5.32   sec   127 MBytes   200 Mbits/sec   12             sender
        [ 16]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 18]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    2             sender
        [ 18]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 20]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    1             sender
        [ 20]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 22]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    4             sender
        [ 22]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 24]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    5             sender
        [ 24]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 26]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    3             sender
        [ 26]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 28]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    3             sender
        [ 28]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 30]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    1             sender
        [ 30]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 32]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    6             sender
        [ 32]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 34]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    2             sender
        [ 34]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 36]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    1             sender
        [ 36]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 38]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    1             sender
        [ 38]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 40]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    2             sender
        [ 40]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 42]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    4             sender
        [ 42]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [SUM]   0.00-5.32   sec  2.48 GBytes  4.00 Gbits/sec   72             sender
        [SUM]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        iperf3: interrupt - the client has terminated

      In this test, I get some retransmissions every couple of seconds. We analyzed the retransmissions by capturing a sample with Wireshark and we can confirm that the retransmissions are fast retransmissions caused by reordering of the traffic at the receiving server only.
       
      The RedHat case recommends to either switch to Perfect Filtering or fix CPU assignment of processes. I have tried both solutions, but the first one still produces retransmissions even when pinning both iperf processes to a fixed CPU:

      taskset 0x1 iperf3 -s -p 12345
        -----------------------------------------------------------
        Server listening on 12345
        -----------------------------------------------------------
        Accepted connection from 192.168.120.10, port 59720
        [  5] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59722
        [  7] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59724
        [  9] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59726
        [ 11] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59728
        [ 13] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59730
        [ 15] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59732
        [ 17] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59734
        [ 19] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59736
        [ 21] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59738
        [ 23] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59740
        [ 25] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59742
        [ 27] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59744
        [ 29] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59746
        [ 31] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59748
        [ 33] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59750
        [ 35] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59752
        [ 37] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59754
        [ 39] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59756
        [ 41] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59758
        [ 43] local 192.168.120.11 port 12345 connected to 192.168.120.10 port 59760
        [ ID] Interval           Transfer     Bandwidth
        [  5]   0.00-1.00   sec  21.8 MBytes   182 Mbits/sec
        [  7]   0.00-1.00   sec  21.8 MBytes   182 Mbits/sec
        [  9]   0.00-1.00   sec  21.8 MBytes   182 Mbits/sec
        [ 11]   0.00-1.00   sec  21.8 MBytes   182 Mbits/sec
        [ 13]   0.00-1.00   sec  21.8 MBytes   182 Mbits/sec
        [ 15]   0.00-1.00   sec  21.8 MBytes   182 Mbits/sec
        [ 17]   0.00-1.00   sec  21.8 MBytes   182 Mbits/sec
        [ 19]   0.00-1.00   sec  21.8 MBytes   182 Mbits/sec
        [ 21]   0.00-1.00   sec  21.8 MBytes   182 Mbits/sec
        [ 23]   0.00-1.00   sec  21.8 MBytes   182 Mbits/sec
        [ 25]   0.00-1.00   sec  21.8 MBytes   182 Mbits/sec
        [ 27]   0.00-1.00   sec  21.8 MBytes   182 Mbits/sec
        [ 29]   0.00-1.00   sec  21.8 MBytes   182 Mbits/sec
        [ 31]   0.00-1.00   sec  21.8 MBytes   182 Mbits/sec
        [ 33]   0.00-1.00   sec  21.8 MBytes   182 Mbits/sec
        [ 35]   0.00-1.00   sec  21.8 MBytes   182 Mbits/sec
        [ 37]   0.00-1.00   sec  21.8 MBytes   182 Mbits/sec
        [ 39]   0.00-1.00   sec  21.8 MBytes   182 Mbits/sec
        [ 41]   0.00-1.00   sec  21.8 MBytes   182 Mbits/sec
        [ 43]   0.00-1.00   sec  21.8 MBytes   182 Mbits/sec
        [SUM]   0.00-1.00   sec   435 MBytes  3.65 Gbits/sec
        - - - - - - - - - - - - - - - - - - - - - - - - -
        [  5]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec
        [  7]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec
        [  9]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec
        [ 11]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec
        [ 13]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec
        [ 15]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec
        [ 17]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec
        [ 19]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec
        [ 21]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec
        [ 23]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec
        [ 25]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec
        [ 27]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec
        [ 29]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec
        [ 31]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec
        [ 33]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec
        [ 35]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec
        [ 37]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec
        [ 39]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec
        [ 41]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec
        [ 43]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec
        [SUM]   1.00-2.00   sec   478 MBytes  4.01 Gbits/sec
        - - - - - - - - - - - - - - - - - - - - - - - - -
        [  5]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec
        [  7]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec
        [  9]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec
        [ 11]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec
        [ 13]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec
        [ 15]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec
        [ 17]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec
        [ 19]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec
        [ 21]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec
        [ 23]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec
        [ 25]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec
        [ 27]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec
        [ 29]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec
        [ 31]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec
        [ 33]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec
        [ 35]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec
        [ 37]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec
        [ 39]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec
        [ 41]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec
        [ 43]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec
        [SUM]   2.00-3.00   sec   478 MBytes  4.01 Gbits/sec
        - - - - - - - - - - - - - - - - - - - - - - - - -
        [  5]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec
        [  7]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec
        [  9]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec
        [ 11]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec
        [ 13]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec
        [ 15]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec
        [ 17]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec
        [ 19]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec
        [ 21]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec
        [ 23]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec
        [ 25]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec
        [ 27]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec
        [ 29]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec
        [ 31]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec
        [ 33]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec
        [ 35]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec
        [ 37]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec
        [ 39]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec
        [ 41]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec
        [ 43]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec
        [SUM]   3.00-4.00   sec   475 MBytes  3.98 Gbits/sec
        - - - - - - - - - - - - - - - - - - - - - - - - -
        [  5]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [  7]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [  9]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 11]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 13]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 15]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 17]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 19]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 21]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 23]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 25]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 27]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 29]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 31]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 33]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 35]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 37]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 39]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 41]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 43]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [SUM]   4.00-5.00   sec   478 MBytes  4.01 Gbits/sec
        - - - - - - - - - - - - - - - - - - - - - - - - -
        [  5]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [  7]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [  9]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 11]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 13]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 15]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 17]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 19]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 21]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 23]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 25]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 27]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 29]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 31]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 33]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 35]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 37]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 39]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 41]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [ 43]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec
        [SUM]   4.00-5.00   sec   478 MBytes  4.01 Gbits/sec
        - - - - - - - - - - - - - - - - - - - - - - - - -
        [ ID] Interval           Transfer     Bandwidth
        [  5]   0.00-5.00   sec  0.00 Bytes  0.00 bits/sec                  sender
        [  5]   0.00-5.00   sec   125 MBytes   210 Mbits/sec                  receiver
        [  7]   0.00-5.00   sec  0.00 Bytes  0.00 bits/sec                  sender
        [  7]   0.00-5.00   sec   125 MBytes   210 Mbits/sec                  receiver
        [  9]   0.00-5.00   sec  0.00 Bytes  0.00 bits/sec                  sender
        [  9]   0.00-5.00   sec   125 MBytes   210 Mbits/sec                  receiver
        [ 11]   0.00-5.00   sec  0.00 Bytes  0.00 bits/sec                  sender
        [ 11]   0.00-5.00   sec   125 MBytes   210 Mbits/sec                  receiver
        [ 13]   0.00-5.00   sec  0.00 Bytes  0.00 bits/sec                  sender
        [ 13]   0.00-5.00   sec   125 MBytes   210 Mbits/sec                  receiver
        [ 15]   0.00-5.00   sec  0.00 Bytes  0.00 bits/sec                  sender
        [ 15]   0.00-5.00   sec   125 MBytes   210 Mbits/sec                  receiver
        [ 17]   0.00-5.00   sec  0.00 Bytes  0.00 bits/sec                  sender
        [ 17]   0.00-5.00   sec   125 MBytes   210 Mbits/sec                  receiver
        [ 19]   0.00-5.00   sec  0.00 Bytes  0.00 bits/sec                  sender
        [ 19]   0.00-5.00   sec   125 MBytes   210 Mbits/sec                  receiver
        [ 21]   0.00-5.00   sec  0.00 Bytes  0.00 bits/sec                  sender
        [ 21]   0.00-5.00   sec   125 MBytes   210 Mbits/sec                  receiver
        [ 23]   0.00-5.00   sec  0.00 Bytes  0.00 bits/sec                  sender
        [ 23]   0.00-5.00   sec   125 MBytes   210 Mbits/sec                  receiver
        [ 25]   0.00-5.00   sec  0.00 Bytes  0.00 bits/sec                  sender
        [ 25]   0.00-5.00   sec   125 MBytes   210 Mbits/sec                  receiver
        [ 27]   0.00-5.00   sec  0.00 Bytes  0.00 bits/sec                  sender
        [ 27]   0.00-5.00   sec   125 MBytes   210 Mbits/sec                  receiver
        [ 29]   0.00-5.00   sec  0.00 Bytes  0.00 bits/sec                  sender
        [ 29]   0.00-5.00   sec   125 MBytes   210 Mbits/sec                  receiver
        [ 31]   0.00-5.00   sec  0.00 Bytes  0.00 bits/sec                  sender
        [ 31]   0.00-5.00   sec   125 MBytes   210 Mbits/sec                  receiver
        [ 33]   0.00-5.00   sec  0.00 Bytes  0.00 bits/sec                  sender
        [ 33]   0.00-5.00   sec   125 MBytes   210 Mbits/sec                  receiver
        [ 35]   0.00-5.00   sec  0.00 Bytes  0.00 bits/sec                  sender
        [ 35]   0.00-5.00   sec   125 MBytes   210 Mbits/sec                  receiver
        [ 37]   0.00-5.00   sec  0.00 Bytes  0.00 bits/sec                  sender
        [ 37]   0.00-5.00   sec   125 MBytes   210 Mbits/sec                  receiver
        [ 39]   0.00-5.00   sec  0.00 Bytes  0.00 bits/sec                  sender
        [ 39]   0.00-5.00   sec   125 MBytes   210 Mbits/sec                  receiver
        [ 41]   0.00-5.00   sec  0.00 Bytes  0.00 bits/sec                  sender
        [ 41]   0.00-5.00   sec   125 MBytes   210 Mbits/sec                  receiver
        [ 43]   0.00-5.00   sec  0.00 Bytes  0.00 bits/sec                  sender
        [ 43]   0.00-5.00   sec   125 MBytes   210 Mbits/sec                  receiver
        [SUM]   0.00-5.00   sec  0.00 Bytes  0.00 bits/sec                  sender
        [SUM]   0.00-5.00   sec  2.45 GBytes  4.20 Gbits/sec                  receiver
        iperf3: the client has terminated


       
       

        taskset 0x1 iperf3 -c 192.168.120.11 -p 12345 -b200m -P20 -t0
        Connecting to host 192.168.120.11, port 12345
        [  4] local 192.168.120.10 port 59722 connected to 192.168.120.11 port 12345
        [  6] local 192.168.120.10 port 59724 connected to 192.168.120.11 port 12345
        [  8] local 192.168.120.10 port 59726 connected to 192.168.120.11 port 12345
        [ 10] local 192.168.120.10 port 59728 connected to 192.168.120.11 port 12345
        [ 12] local 192.168.120.10 port 59730 connected to 192.168.120.11 port 12345
        [ 14] local 192.168.120.10 port 59732 connected to 192.168.120.11 port 12345
        [ 16] local 192.168.120.10 port 59734 connected to 192.168.120.11 port 12345
        [ 18] local 192.168.120.10 port 59736 connected to 192.168.120.11 port 12345
        [ 20] local 192.168.120.10 port 59738 connected to 192.168.120.11 port 12345
        [ 22] local 192.168.120.10 port 59740 connected to 192.168.120.11 port 12345
        [ 24] local 192.168.120.10 port 59742 connected to 192.168.120.11 port 12345
        [ 26] local 192.168.120.10 port 59744 connected to 192.168.120.11 port 12345
        [ 28] local 192.168.120.10 port 59746 connected to 192.168.120.11 port 12345
        [ 30] local 192.168.120.10 port 59748 connected to 192.168.120.11 port 12345
        [ 32] local 192.168.120.10 port 59750 connected to 192.168.120.11 port 12345
        [ 34] local 192.168.120.10 port 59752 connected to 192.168.120.11 port 12345
        [ 36] local 192.168.120.10 port 59754 connected to 192.168.120.11 port 12345
        [ 38] local 192.168.120.10 port 59756 connected to 192.168.120.11 port 12345
        [ 40] local 192.168.120.10 port 59758 connected to 192.168.120.11 port 12345
        [ 42] local 192.168.120.10 port 59760 connected to 192.168.120.11 port 12345
        [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
        [  4]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    7    189 KBytes
        [  6]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    7    122 KBytes
        [  8]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    1    134 KBytes
        [ 10]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    0    158 KBytes
        [ 12]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    2    130 KBytes
        [ 14]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    1    168 KBytes
        [ 16]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    5    133 KBytes
        [ 18]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    0    181 KBytes
        [ 20]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    1    132 KBytes
        [ 22]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    5    129 KBytes
        [ 24]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    4    132 KBytes
        [ 26]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    4    133 KBytes
        [ 28]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    2    127 KBytes
        [ 30]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    0    173 KBytes
        [ 32]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    9    132 KBytes
        [ 34]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    3    130 KBytes
        [ 36]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    4    136 KBytes
        [ 38]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    1    130 KBytes
        [ 40]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    0    168 KBytes
        [ 42]   0.00-1.00   sec  21.9 MBytes   183 Mbits/sec    6    127 KBytes
        [SUM]   0.00-1.00   sec   438 MBytes  3.67 Gbits/sec   62
        - - - - - - - - - - - - - - - - - - - - - - - - -
        [  4]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec    0    189 KBytes
        [  6]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec    0    130 KBytes
        [  8]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec    0    134 KBytes
        [ 10]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec    0    158 KBytes
        [ 12]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec    0    134 KBytes
        [ 14]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec    0    168 KBytes
        [ 16]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec    3    148 KBytes
        [ 18]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec    0    191 KBytes
        [ 20]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec    0    141 KBytes
        [ 22]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec    0    143 KBytes
        [ 24]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec    0    134 KBytes
        [ 26]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec    0    139 KBytes
        [ 28]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec    0    129 KBytes
        [ 30]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec    1    173 KBytes
        [ 32]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec    0    133 KBytes
        [ 34]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec    0    141 KBytes
        [ 36]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec    0    137 KBytes
        [ 38]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec    0    141 KBytes
        [ 40]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec    0    168 KBytes
        [ 42]   1.00-2.00   sec  23.9 MBytes   200 Mbits/sec    0    133 KBytes
        [SUM]   1.00-2.00   sec   478 MBytes  4.01 Gbits/sec    4
        - - - - - - - - - - - - - - - - - - - - - - - - -
        [  4]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec    0    189 KBytes
        [  6]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec    0    130 KBytes
        [  8]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec    0    134 KBytes
        [ 10]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec    0    158 KBytes
        [ 12]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec    0    134 KBytes
        [ 14]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec    0    168 KBytes
        [ 16]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec    0    151 KBytes
        [ 18]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec    0    211 KBytes
        [ 20]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec    0    157 KBytes
        [ 22]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec    0    157 KBytes
        [ 24]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec    0    137 KBytes
        [ 26]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec    0    147 KBytes
        [ 28]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec    4    129 KBytes
        [ 30]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec    0    173 KBytes
        [ 32]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec    0    133 KBytes
        [ 34]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec    0    143 KBytes
        [ 36]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec    0    137 KBytes
        [ 38]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec    0    144 KBytes
        [ 40]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec    0    168 KBytes
        [ 42]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec    0    136 KBytes
        [SUM]   2.00-3.00   sec   478 MBytes  4.01 Gbits/sec    4
        - - - - - - - - - - - - - - - - - - - - - - - - -
        [  4]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec    0    189 KBytes
        [  6]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec    0    130 KBytes
        [  8]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec    0    134 KBytes
        [ 10]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec    0    158 KBytes
        [ 12]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec    0    134 KBytes
        [ 14]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec    0    168 KBytes
        [ 16]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec    1    156 KBytes
        [ 18]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec    2    223 KBytes
        [ 20]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec    0    163 KBytes
        [ 22]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec    0    165 KBytes
        [ 24]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec    0    139 KBytes
        [ 26]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec    0    154 KBytes
        [ 28]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec    0    130 KBytes
        [ 30]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec    0    173 KBytes
        [ 32]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec    0    134 KBytes
        [ 34]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec    0    150 KBytes
        [ 36]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec    0    137 KBytes
        [ 38]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec    0    151 KBytes
        [ 40]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec    0    168 KBytes
        [ 42]   3.00-4.00   sec  23.8 MBytes   199 Mbits/sec    0    146 KBytes
        [SUM]   3.00-4.00   sec   475 MBytes  3.98 Gbits/sec    3
        - - - - - - - - - - - - - - - - - - - - - - - - -
        [  4]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec    0    189 KBytes
        [  6]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec    0    130 KBytes
        [  8]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec    0    134 KBytes
        [ 10]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec    1    158 KBytes
        [ 12]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec    0    134 KBytes
        [ 14]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec    0    168 KBytes
        [ 16]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec    1    161 KBytes
        [ 18]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec    0    223 KBytes
        [ 20]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec    0    174 KBytes
        [ 22]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec    0    175 KBytes
        [ 24]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec    0    139 KBytes
        [ 26]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec    0    154 KBytes
        [ 28]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec    0    132 KBytes
        [ 30]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec    0    173 KBytes
        [ 32]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec    0    134 KBytes
        [ 34]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec    0    150 KBytes
        [ 36]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec    0    137 KBytes
        [ 38]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec    0    151 KBytes
        [ 40]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec    0    168 KBytes
        [ 42]   4.00-5.00   sec  23.9 MBytes   200 Mbits/sec    0    146 KBytes
        [SUM]   4.00-5.00   sec   478 MBytes  4.01 Gbits/sec    2
        ^C- - - - - - - - - - - - - - - - - - - - - - - - -
        [  4]   5.00-5.32   sec  9.38 MBytes   248 Mbits/sec    0    189 KBytes
        [  6]   5.00-5.32   sec  9.38 MBytes   248 Mbits/sec    0    130 KBytes
        [  8]   5.00-5.32   sec  9.38 MBytes   248 Mbits/sec    0    134 KBytes
        [ 10]   5.00-5.32   sec  9.38 MBytes   248 Mbits/sec    0    158 KBytes
        [ 12]   5.00-5.32   sec  9.38 MBytes   248 Mbits/sec    0    140 KBytes
        [ 14]   5.00-5.32   sec  9.38 MBytes   248 Mbits/sec    0    168 KBytes
        [ 16]   5.00-5.32   sec  9.38 MBytes   248 Mbits/sec    0    164 KBytes
        [ 18]   5.00-5.32   sec  9.38 MBytes   248 Mbits/sec    0    223 KBytes
        [ 20]   5.00-5.32   sec  9.38 MBytes   248 Mbits/sec    0    174 KBytes
        [ 22]   5.00-5.32   sec  9.38 MBytes   248 Mbits/sec    0    175 KBytes
        [ 24]   5.00-5.32   sec  9.38 MBytes   248 Mbits/sec    0    139 KBytes
        [ 26]   5.00-5.32   sec  9.38 MBytes   248 Mbits/sec    0    154 KBytes
        [ 28]   5.00-5.32   sec  9.38 MBytes   248 Mbits/sec    0    132 KBytes
        [ 30]   5.00-5.32   sec  9.38 MBytes   248 Mbits/sec    0    173 KBytes
        [ 32]   5.00-5.32   sec  9.38 MBytes   248 Mbits/sec    0    134 KBytes
        [ 34]   5.00-5.32   sec  9.38 MBytes   248 Mbits/sec    0    150 KBytes
        [ 36]   5.00-5.32   sec  9.38 MBytes   248 Mbits/sec    0    137 KBytes
        [ 38]   5.00-5.32   sec  9.38 MBytes   248 Mbits/sec    0    151 KBytes
        [ 40]   5.00-5.32   sec  9.38 MBytes   248 Mbits/sec    0    168 KBytes
        [ 42]   5.00-5.32   sec  9.38 MBytes   248 Mbits/sec    0    146 KBytes
        [SUM]   5.00-5.32   sec   188 MBytes  4.96 Gbits/sec    0
        - - - - - - - - - - - - - - - - - - - - - - - - -
        [ ID] Interval           Transfer     Bandwidth       Retr
        [  4]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    7             sender
        [  4]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [  6]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    7             sender
        [  6]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [  8]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    1             sender
        [  8]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 10]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    1             sender
        [ 10]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 12]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    2             sender
        [ 12]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 14]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    1             sender
        [ 14]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 16]   0.00-5.32   sec   127 MBytes   200 Mbits/sec   10             sender
        [ 16]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 18]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    2             sender
        [ 18]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 20]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    1             sender
        [ 20]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 22]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    5             sender
        [ 22]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 24]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    4             sender
        [ 24]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 26]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    4             sender
        [ 26]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 28]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    6             sender
        [ 28]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 30]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    1             sender
        [ 30]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 32]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    9             sender
        [ 32]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 34]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    3             sender
        [ 34]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 36]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    4             sender
        [ 36]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 38]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    1             sender
        [ 38]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 40]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    0             sender
        [ 40]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [ 42]   0.00-5.32   sec   127 MBytes   200 Mbits/sec    6             sender
        [ 42]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        [SUM]   0.00-5.32   sec  2.47 GBytes  3.99 Gbits/sec   75             sender
        [SUM]   0.00-5.32   sec  0.00 Bytes  0.00 bits/sec                  receiver
        iperf3: interrupt - the client has terminated

      Switching to Perfect Forwarding mode fixed the problems and iperf runs without retransmissions. Are there other factors that can cause retransmissions, e.g. irq balancing at the receiver?
       
        Best wishes,
          Jens