1 Reply Latest reply on Mar 21, 2016 8:58 PM by wb_Intel

    X520 10G - UDP packet drops

    emueller

      Hello,

       

      I have a strange problem occurring with my new server which has the intel x520 network cards. This server runs Ubuntu KVM with virtual machines that collect netflow data- these device are virtual appliances. I noticed within the virtual machine that netflow datagrams. I first suspected resource constraints within KVM however after disabling all of the virtual appliances (except one) the packet drops were still occurring. I am able to determine the packet drops by simply taking a pcap and running it in a script with tshark. Each UDP datagram that contains the netflow data has a sequence number in the application layer header. I then ran a pcap on the host bridge interface and found that the issue is occurring outside the virtual machine. I then did a packet capture on physical interface to rule out that it was caused by the bridge and the issue is occurring there too. I then mirrored the interface to a separate server that has a similar 10gig network card and this device is able to capture each packet.

       

      I suspect this is a network card offload /drive issue however I have not been successful in finding a solution. I tried upgrading the driver from 4.0.1 to 4.3.13 with no difference.

       

      I am running out of options.

       

      I tried disabling

      for i in rx tx sg tso ufo gso gro lro; do sudo ethtool -K enp33s0f1 $i off; done

       

       

       

      Tried increasing memory allocation to receive memory buffer:

       

      cat /proc/sys/net/core/rmem_default

      212992

      cat /proc/sys/net/core/rmem_max

      212992

       

      and set the values to:

       

      net.core.rmem_default = 16777216

      net.core.rmem_max = 16777216

       

       

      I noticed packets were incrementing the below counters in ethtool -S so i tried the below modprobe command but it did not help.

       

      hw_rsc_aggregated: 4074

      hw_rsc_flushed: 1837

       

      modprobe ixgbe InterruptThrottleRate=off

       

       

       

       

       


      server with issue:

      lspci |grep Ethernet

      0c:00.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)

      0c:00.1 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)

      21:00.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)

      21:00.1 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)

      62:00.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)

      62:00.1 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)

      63:00.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)

      63:00.1 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)

       

      I first tried with this below driver version.

      version: 4.0.1-k

       

      Then upgraded and tried the below driver version.

      version: 4.3.13

       

       

       

      server without issue:

      [em621d@shdw01vmh ~]$ lspci |grep Ethernet

      0e:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)

      0e:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)

      version: 3.6.7-k

       

       

       

      Traffic characteristics:

       

      Under 150Mbps

      Rate 10k pps

      Packet size 1340 - 1372 bytes

       

       

       

      ethtool -S enp33s0f1

      NIC statistics:

           rx_packets: 728170

           tx_packets: 8

           rx_bytes: 1007244560

           tx_bytes: 648

           rx_errors: 0

           tx_errors: 0

           rx_dropped: 0

           tx_dropped: 0

           multicast: 254

           collisions: 0

           rx_over_errors: 0

           rx_crc_errors: 0

           rx_frame_errors: 0

           rx_fifo_errors: 0

           rx_missed_errors: 0

           tx_aborted_errors: 0

           tx_carrier_errors: 0

           tx_fifo_errors: 0

           tx_heartbeat_errors: 0

           rx_pkts_nic: 728170

           tx_pkts_nic: 8

           rx_bytes_nic: 1010157240

           tx_bytes_nic: 680

           lsc_int: 1

           tx_busy: 0

           non_eop_descs: 0

           broadcast: 63

           rx_no_buffer_count: 0

           tx_timeout_count: 0

           tx_restart_queue: 0

           rx_long_length_errors: 0

           rx_short_length_errors: 0

           tx_flow_control_xon: 0

           rx_flow_control_xon: 0

           tx_flow_control_xoff: 0

           rx_flow_control_xoff: 0

           rx_csum_offload_errors: 0

           alloc_rx_page_failed: 0

           alloc_rx_buff_failed: 0

           lro_aggregated: 0

           lro_flushed: 0

           rx_no_dma_resources: 0

           hw_rsc_aggregated: 0

           hw_rsc_flushed: 0

           fdir_match: 0

           fdir_miss: 727925

           fdir_overflow: 0

           fcoe_bad_fccrc: 0

           fcoe_last_errors: 0

           rx_fcoe_dropped: 0

           rx_fcoe_packets: 0

           rx_fcoe_dwords: 0

           fcoe_noddp: 0

           fcoe_noddp_ext_buff: 0

           tx_fcoe_packets: 0

           tx_fcoe_dwords: 0

           os2bmc_rx_by_bmc: 0

           os2bmc_tx_by_bmc: 0

           os2bmc_tx_by_host: 0

           os2bmc_rx_by_host: 0

           tx_hwtstamp_timeouts: 0

           rx_hwtstamp_cleared: 0

           tx_queue_0_packets: 0

           tx_queue_0_bytes: 0

           tx_queue_0_bp_napi_yield: 0

           tx_queue_0_bp_misses: 0

           tx_queue_0_bp_cleaned: 0

           tx_queue_1_packets: 0

           tx_queue_1_bytes: 0

           tx_queue_1_bp_napi_yield: 0

           tx_queue_1_bp_misses: 0

           tx_queue_1_bp_cleaned: 0

           tx_queue_2_packets: 0

           tx_queue_2_bytes: 0

           tx_queue_2_bp_napi_yield: 0

           tx_queue_2_bp_misses: 0

           tx_queue_2_bp_cleaned: 0

           tx_queue_3_packets: 0

           tx_queue_3_bytes: 0

           tx_queue_3_bp_napi_yield: 0

           tx_queue_3_bp_misses: 0

           tx_queue_3_bp_cleaned: 0

           tx_queue_4_packets: 0

           tx_queue_4_bytes: 0

           tx_queue_4_bp_napi_yield: 0

           tx_queue_4_bp_misses: 0

           tx_queue_4_bp_cleaned: 0

           tx_queue_5_packets: 0

           tx_queue_5_bytes: 0

           tx_queue_5_bp_napi_yield: 0

           tx_queue_5_bp_misses: 0

           tx_queue_5_bp_cleaned: 0

           tx_queue_6_packets: 0

           tx_queue_6_bytes: 0

           tx_queue_6_bp_napi_yield: 0

           tx_queue_6_bp_misses: 0

           tx_queue_6_bp_cleaned: 0

           tx_queue_7_packets: 0

           tx_queue_7_bytes: 0

           tx_queue_7_bp_napi_yield: 0

           tx_queue_7_bp_misses: 0

           tx_queue_7_bp_cleaned: 0

           tx_queue_8_packets: 0

           tx_queue_8_bytes: 0

           tx_queue_8_bp_napi_yield: 0

           tx_queue_8_bp_misses: 0

           tx_queue_8_bp_cleaned: 0

           tx_queue_9_packets: 0

           tx_queue_9_bytes: 0

           tx_queue_9_bp_napi_yield: 0

           tx_queue_9_bp_misses: 0

           tx_queue_9_bp_cleaned: 0

           tx_queue_10_packets: 0

           tx_queue_10_bytes: 0

           tx_queue_10_bp_napi_yield: 0

           tx_queue_10_bp_misses: 0

           tx_queue_10_bp_cleaned: 0

           tx_queue_11_packets: 0

           tx_queue_11_bytes: 0

           tx_queue_11_bp_napi_yield: 0

           tx_queue_11_bp_misses: 0

           tx_queue_11_bp_cleaned: 0

           tx_queue_12_packets: 0

           tx_queue_12_bytes: 0

           tx_queue_12_bp_napi_yield: 0

           tx_queue_12_bp_misses: 0

           tx_queue_12_bp_cleaned: 0

           tx_queue_13_packets: 0

           tx_queue_13_bytes: 0

           tx_queue_13_bp_napi_yield: 0

           tx_queue_13_bp_misses: 0

           tx_queue_13_bp_cleaned: 0

           tx_queue_14_packets: 0

           tx_queue_14_bytes: 0

           tx_queue_14_bp_napi_yield: 0

           tx_queue_14_bp_misses: 0

           tx_queue_14_bp_cleaned: 0

           tx_queue_15_packets: 0

           tx_queue_15_bytes: 0

           tx_queue_15_bp_napi_yield: 0

           tx_queue_15_bp_misses: 0

           tx_queue_15_bp_cleaned: 0

           tx_queue_16_packets: 0

           tx_queue_16_bytes: 0

           tx_queue_16_bp_napi_yield: 0

           tx_queue_16_bp_misses: 0

           tx_queue_16_bp_cleaned: 0

           tx_queue_17_packets: 0

           tx_queue_17_bytes: 0

           tx_queue_17_bp_napi_yield: 0

           tx_queue_17_bp_misses: 0

           tx_queue_17_bp_cleaned: 0

           tx_queue_18_packets: 0

           tx_queue_18_bytes: 0

           tx_queue_18_bp_napi_yield: 0

           tx_queue_18_bp_misses: 0

           tx_queue_18_bp_cleaned: 0

           tx_queue_19_packets: 0

           tx_queue_19_bytes: 0

           tx_queue_19_bp_napi_yield: 0

           tx_queue_19_bp_misses: 0

           tx_queue_19_bp_cleaned: 0

           tx_queue_20_packets: 0

           tx_queue_20_bytes: 0

           tx_queue_20_bp_napi_yield: 0

           tx_queue_20_bp_misses: 0

           tx_queue_20_bp_cleaned: 0

           tx_queue_21_packets: 0

           tx_queue_21_bytes: 0

           tx_queue_21_bp_napi_yield: 0

           tx_queue_21_bp_misses: 0

           tx_queue_21_bp_cleaned: 0

           tx_queue_22_packets: 0

           tx_queue_22_bytes: 0

           tx_queue_22_bp_napi_yield: 0

           tx_queue_22_bp_misses: 0

           tx_queue_22_bp_cleaned: 0

           tx_queue_23_packets: 0

           tx_queue_23_bytes: 0

           tx_queue_23_bp_napi_yield: 0

           tx_queue_23_bp_misses: 0

           tx_queue_23_bp_cleaned: 0

           tx_queue_24_packets: 0

           tx_queue_24_bytes: 0

           tx_queue_24_bp_napi_yield: 0

           tx_queue_24_bp_misses: 0

           tx_queue_24_bp_cleaned: 0

           tx_queue_25_packets: 0

           tx_queue_25_bytes: 0

           tx_queue_25_bp_napi_yield: 0

           tx_queue_25_bp_misses: 0

           tx_queue_25_bp_cleaned: 0

           tx_queue_26_packets: 0

           tx_queue_26_bytes: 0

           tx_queue_26_bp_napi_yield: 0

           tx_queue_26_bp_misses: 0

           tx_queue_26_bp_cleaned: 0

           tx_queue_27_packets: 8

           tx_queue_27_bytes: 648

           tx_queue_27_bp_napi_yield: 0

           tx_queue_27_bp_misses: 0

           tx_queue_27_bp_cleaned: 0

           tx_queue_28_packets: 0

           tx_queue_28_bytes: 0

           tx_queue_28_bp_napi_yield: 0

           tx_queue_28_bp_misses: 0

           tx_queue_28_bp_cleaned: 0

           tx_queue_29_packets: 0

           tx_queue_29_bytes: 0

           tx_queue_29_bp_napi_yield: 0

           tx_queue_29_bp_misses: 0

           tx_queue_29_bp_cleaned: 0

           tx_queue_30_packets: 0

           tx_queue_30_bytes: 0

           tx_queue_30_bp_napi_yield: 0

           tx_queue_30_bp_misses: 0

           tx_queue_30_bp_cleaned: 0

           tx_queue_31_packets: 0

           tx_queue_31_bytes: 0

           tx_queue_31_bp_napi_yield: 0

           tx_queue_31_bp_misses: 0

           tx_queue_31_bp_cleaned: 0

           tx_queue_32_packets: 0

           tx_queue_32_bytes: 0

           tx_queue_32_bp_napi_yield: 0

           tx_queue_32_bp_misses: 0

           tx_queue_32_bp_cleaned: 0

           tx_queue_33_packets: 0

           tx_queue_33_bytes: 0

           tx_queue_33_bp_napi_yield: 0

           tx_queue_33_bp_misses: 0

           tx_queue_33_bp_cleaned: 0

           tx_queue_34_packets: 0

           tx_queue_34_bytes: 0

           tx_queue_34_bp_napi_yield: 0

           tx_queue_34_bp_misses: 0

           tx_queue_34_bp_cleaned: 0

           tx_queue_35_packets: 0

           tx_queue_35_bytes: 0

           tx_queue_35_bp_napi_yield: 0

           tx_queue_35_bp_misses: 0

           tx_queue_35_bp_cleaned: 0

           tx_queue_36_packets: 0

           tx_queue_36_bytes: 0

           tx_queue_36_bp_napi_yield: 0

           tx_queue_36_bp_misses: 0

           tx_queue_36_bp_cleaned: 0

           tx_queue_37_packets: 0

           tx_queue_37_bytes: 0

           tx_queue_37_bp_napi_yield: 0

           tx_queue_37_bp_misses: 0

           tx_queue_37_bp_cleaned: 0

           tx_queue_38_packets: 0

           tx_queue_38_bytes: 0

           tx_queue_38_bp_napi_yield: 0

           tx_queue_38_bp_misses: 0

           tx_queue_38_bp_cleaned: 0

           tx_queue_39_packets: 0

           tx_queue_39_bytes: 0

           tx_queue_39_bp_napi_yield: 0

           tx_queue_39_bp_misses: 0

           tx_queue_39_bp_cleaned: 0

           tx_queue_40_packets: 0

           tx_queue_40_bytes: 0

           tx_queue_40_bp_napi_yield: 0

           tx_queue_40_bp_misses: 0

           tx_queue_40_bp_cleaned: 0

           tx_queue_41_packets: 0

           tx_queue_41_bytes: 0

           tx_queue_41_bp_napi_yield: 0

           tx_queue_41_bp_misses: 0

           tx_queue_41_bp_cleaned: 0

           tx_queue_42_packets: 0

           tx_queue_42_bytes: 0

           tx_queue_42_bp_napi_yield: 0

           tx_queue_42_bp_misses: 0

           tx_queue_42_bp_cleaned: 0

           tx_queue_43_packets: 0

           tx_queue_43_bytes: 0

           tx_queue_43_bp_napi_yield: 0

           tx_queue_43_bp_misses: 0

           tx_queue_43_bp_cleaned: 0

           tx_queue_44_packets: 0

           tx_queue_44_bytes: 0

           tx_queue_44_bp_napi_yield: 0

           tx_queue_44_bp_misses: 0

           tx_queue_44_bp_cleaned: 0

           tx_queue_45_packets: 0

           tx_queue_45_bytes: 0

           tx_queue_45_bp_napi_yield: 0

           tx_queue_45_bp_misses: 0

           tx_queue_45_bp_cleaned: 0

           tx_queue_46_packets: 0

           tx_queue_46_bytes: 0

           tx_queue_46_bp_napi_yield: 0

           tx_queue_46_bp_misses: 0

           tx_queue_46_bp_cleaned: 0

           tx_queue_47_packets: 0

           tx_queue_47_bytes: 0

           tx_queue_47_bp_napi_yield: 0

           tx_queue_47_bp_misses: 0

           tx_queue_47_bp_cleaned: 0

           tx_queue_48_packets: 0

           tx_queue_48_bytes: 0

           tx_queue_48_bp_napi_yield: 0

           tx_queue_48_bp_misses: 0

           tx_queue_48_bp_cleaned: 0

           tx_queue_49_packets: 0

           tx_queue_49_bytes: 0

           tx_queue_49_bp_napi_yield: 0

           tx_queue_49_bp_misses: 0

           tx_queue_49_bp_cleaned: 0

           tx_queue_50_packets: 0

           tx_queue_50_bytes: 0

           tx_queue_50_bp_napi_yield: 0

           tx_queue_50_bp_misses: 0

           tx_queue_50_bp_cleaned: 0

           tx_queue_51_packets: 0

           tx_queue_51_bytes: 0

           tx_queue_51_bp_napi_yield: 0

           tx_queue_51_bp_misses: 0

           tx_queue_51_bp_cleaned: 0

           tx_queue_52_packets: 0

           tx_queue_52_bytes: 0

           tx_queue_52_bp_napi_yield: 0

           tx_queue_52_bp_misses: 0

           tx_queue_52_bp_cleaned: 0

           tx_queue_53_packets: 0

           tx_queue_53_bytes: 0

           tx_queue_53_bp_napi_yield: 0

           tx_queue_53_bp_misses: 0

           tx_queue_53_bp_cleaned: 0

           tx_queue_54_packets: 0

           tx_queue_54_bytes: 0

           tx_queue_54_bp_napi_yield: 0

           tx_queue_54_bp_misses: 0

           tx_queue_54_bp_cleaned: 0

           tx_queue_55_packets: 0

           tx_queue_55_bytes: 0

           tx_queue_55_bp_napi_yield: 0

           tx_queue_55_bp_misses: 0

           tx_queue_55_bp_cleaned: 0

           tx_queue_56_packets: 0

           tx_queue_56_bytes: 0

           tx_queue_56_bp_napi_yield: 0

           tx_queue_56_bp_misses: 0

           tx_queue_56_bp_cleaned: 0

           tx_queue_57_packets: 0

           tx_queue_57_bytes: 0

           tx_queue_57_bp_napi_yield: 0

           tx_queue_57_bp_misses: 0

           tx_queue_57_bp_cleaned: 0

           tx_queue_58_packets: 0

           tx_queue_58_bytes: 0

           tx_queue_58_bp_napi_yield: 0

           tx_queue_58_bp_misses: 0

           tx_queue_58_bp_cleaned: 0

           tx_queue_59_packets: 0

           tx_queue_59_bytes: 0

           tx_queue_59_bp_napi_yield: 0

           tx_queue_59_bp_misses: 0

           tx_queue_59_bp_cleaned: 0

           tx_queue_60_packets: 0

           tx_queue_60_bytes: 0

           tx_queue_60_bp_napi_yield: 0

           tx_queue_60_bp_misses: 0

           tx_queue_60_bp_cleaned: 0

           tx_queue_61_packets: 0

           tx_queue_61_bytes: 0

           tx_queue_61_bp_napi_yield: 0

           tx_queue_61_bp_misses: 0

           tx_queue_61_bp_cleaned: 0

           tx_queue_62_packets: 0

           tx_queue_62_bytes: 0

           tx_queue_62_bp_napi_yield: 0

           tx_queue_62_bp_misses: 0

           tx_queue_62_bp_cleaned: 0

           tx_queue_63_packets: 0

           tx_queue_63_bytes: 0

           tx_queue_63_bp_napi_yield: 0

           tx_queue_63_bp_misses: 0

           tx_queue_63_bp_cleaned: 0

           tx_queue_64_packets: 0

           tx_queue_64_bytes: 0

           tx_queue_64_bp_napi_yield: 0

           tx_queue_64_bp_misses: 0

           tx_queue_64_bp_cleaned: 0

           tx_queue_65_packets: 0

           tx_queue_65_bytes: 0

           tx_queue_65_bp_napi_yield: 0

           tx_queue_65_bp_misses: 0

           tx_queue_65_bp_cleaned: 0

           tx_queue_66_packets: 0

           tx_queue_66_bytes: 0

           tx_queue_66_bp_napi_yield: 0

           tx_queue_66_bp_misses: 0

           tx_queue_66_bp_cleaned: 0

           tx_queue_67_packets: 0

           tx_queue_67_bytes: 0

           tx_queue_67_bp_napi_yield: 0

           tx_queue_67_bp_misses: 0

           tx_queue_67_bp_cleaned: 0

           tx_queue_68_packets: 0

           tx_queue_68_bytes: 0

           tx_queue_68_bp_napi_yield: 0

           tx_queue_68_bp_misses: 0

           tx_queue_68_bp_cleaned: 0

           tx_queue_69_packets: 0

           tx_queue_69_bytes: 0

           tx_queue_69_bp_napi_yield: 0

           tx_queue_69_bp_misses: 0

           tx_queue_69_bp_cleaned: 0

           tx_queue_70_packets: 0

           tx_queue_70_bytes: 0

           tx_queue_70_bp_napi_yield: 0

           tx_queue_70_bp_misses: 0

           tx_queue_70_bp_cleaned: 0

           rx_queue_0_packets: 9482

           rx_queue_0_bytes: 11643129

           rx_queue_0_bp_poll_yield: 0

           rx_queue_0_bp_misses: 0

           rx_queue_0_bp_cleaned: 0

           rx_queue_1_packets: 165724

           rx_queue_1_bytes: 232486518

           rx_queue_1_bp_poll_yield: 0

           rx_queue_1_bp_misses: 0

           rx_queue_1_bp_cleaned: 0

           rx_queue_2_packets: 80657

           rx_queue_2_bytes: 112241119

           rx_queue_2_bp_poll_yield: 0

           rx_queue_2_bp_misses: 0

           rx_queue_2_bp_cleaned: 0

           rx_queue_3_packets: 46645

           rx_queue_3_bytes: 64303361

           rx_queue_3_bp_poll_yield: 0

           rx_queue_3_bp_misses: 0

           rx_queue_3_bp_cleaned: 0

           rx_queue_4_packets: 39907

           rx_queue_4_bytes: 55708359

           rx_queue_4_bp_poll_yield: 0

           rx_queue_4_bp_misses: 0

           rx_queue_4_bp_cleaned: 0

           rx_queue_5_packets: 57077

           rx_queue_5_bytes: 77940273

           rx_queue_5_bp_poll_yield: 0

           rx_queue_5_bp_misses: 0

           rx_queue_5_bp_cleaned: 0

           rx_queue_6_packets: 12543

           rx_queue_6_bytes: 17162117

           rx_queue_6_bp_poll_yield: 0

           rx_queue_6_bp_misses: 0

           rx_queue_6_bp_cleaned: 0

           rx_queue_7_packets: 45791

           rx_queue_7_bytes: 63757988

           rx_queue_7_bp_poll_yield: 0

           rx_queue_7_bp_misses: 0

           rx_queue_7_bp_cleaned: 0

           rx_queue_8_packets: 10118

           rx_queue_8_bytes: 13687646

           rx_queue_8_bp_poll_yield: 0

           rx_queue_8_bp_misses: 0

           rx_queue_8_bp_cleaned: 0

           rx_queue_9_packets: 69990

           rx_queue_9_bytes: 97934717

           rx_queue_9_bp_poll_yield: 0

           rx_queue_9_bp_misses: 0

           rx_queue_9_bp_cleaned: 0

           rx_queue_10_packets: 5588

           rx_queue_10_bytes: 7125456

           rx_queue_10_bp_poll_yield: 0

           rx_queue_10_bp_misses: 0

           rx_queue_10_bp_cleaned: 0

           rx_queue_11_packets: 73009

           rx_queue_11_bytes: 101591101

           rx_queue_11_bp_poll_yield: 0

           rx_queue_11_bp_misses: 0

           rx_queue_11_bp_cleaned: 0

           rx_queue_12_packets: 6666

           rx_queue_12_bytes: 8709930

           rx_queue_12_bp_poll_yield: 0

           rx_queue_12_bp_misses: 0

           rx_queue_12_bp_cleaned: 0

           rx_queue_13_packets: 86238

           rx_queue_13_bytes: 119855689

           rx_queue_13_bp_poll_yield: 0

           rx_queue_13_bp_misses: 0

           rx_queue_13_bp_cleaned: 0

           rx_queue_14_packets: 10094

           rx_queue_14_bytes: 12339328

           rx_queue_14_bp_poll_yield: 0

           rx_queue_14_bp_misses: 0

           rx_queue_14_bp_cleaned: 0

           rx_queue_15_packets: 8641

           rx_queue_15_bytes: 10757829

           rx_queue_15_bp_poll_yield: 0

           rx_queue_15_bp_misses: 0

           rx_queue_15_bp_cleaned: 0

           rx_queue_16_packets: 0

           rx_queue_16_bytes: 0

           rx_queue_16_bp_poll_yield: 0

           rx_queue_16_bp_misses: 0

           rx_queue_16_bp_cleaned: 0

           rx_queue_17_packets: 0

           rx_queue_17_bytes: 0

           rx_queue_17_bp_poll_yield: 0

           rx_queue_17_bp_misses: 0

           rx_queue_17_bp_cleaned: 0

           rx_queue_18_packets: 0

           rx_queue_18_bytes: 0

           rx_queue_18_bp_poll_yield: 0

           rx_queue_18_bp_misses: 0

           rx_queue_18_bp_cleaned: 0

           rx_queue_19_packets: 0

           rx_queue_19_bytes: 0

           rx_queue_19_bp_poll_yield: 0

           rx_queue_19_bp_misses: 0

           rx_queue_19_bp_cleaned: 0

           rx_queue_20_packets: 0

           rx_queue_20_bytes: 0

           rx_queue_20_bp_poll_yield: 0

           rx_queue_20_bp_misses: 0

           rx_queue_20_bp_cleaned: 0

           rx_queue_21_packets: 0

           rx_queue_21_bytes: 0

           rx_queue_21_bp_poll_yield: 0

           rx_queue_21_bp_misses: 0

           rx_queue_21_bp_cleaned: 0

           rx_queue_22_packets: 0

           rx_queue_22_bytes: 0

           rx_queue_22_bp_poll_yield: 0

           rx_queue_22_bp_misses: 0

           rx_queue_22_bp_cleaned: 0

           rx_queue_23_packets: 0

           rx_queue_23_bytes: 0

           rx_queue_23_bp_poll_yield: 0

           rx_queue_23_bp_misses: 0

           rx_queue_23_bp_cleaned: 0

           rx_queue_24_packets: 0

           rx_queue_24_bytes: 0

           rx_queue_24_bp_poll_yield: 0

           rx_queue_24_bp_misses: 0

           rx_queue_24_bp_cleaned: 0

           rx_queue_25_packets: 0

           rx_queue_25_bytes: 0

           rx_queue_25_bp_poll_yield: 0

           rx_queue_25_bp_misses: 0

           rx_queue_25_bp_cleaned: 0

           rx_queue_26_packets: 0

           rx_queue_26_bytes: 0

           rx_queue_26_bp_poll_yield: 0

           rx_queue_26_bp_misses: 0

           rx_queue_26_bp_cleaned: 0

           rx_queue_27_packets: 0

           rx_queue_27_bytes: 0

           rx_queue_27_bp_poll_yield: 0

           rx_queue_27_bp_misses: 0

           rx_queue_27_bp_cleaned: 0

           rx_queue_28_packets: 0

           rx_queue_28_bytes: 0

           rx_queue_28_bp_poll_yield: 0

           rx_queue_28_bp_misses: 0

           rx_queue_28_bp_cleaned: 0

           rx_queue_29_packets: 0

           rx_queue_29_bytes: 0

           rx_queue_29_bp_poll_yield: 0

           rx_queue_29_bp_misses: 0

           rx_queue_29_bp_cleaned: 0

           rx_queue_30_packets: 0

           rx_queue_30_bytes: 0

           rx_queue_30_bp_poll_yield: 0

           rx_queue_30_bp_misses: 0

           rx_queue_30_bp_cleaned: 0

           rx_queue_31_packets: 0

           rx_queue_31_bytes: 0

           rx_queue_31_bp_poll_yield: 0

           rx_queue_31_bp_misses: 0

           rx_queue_31_bp_cleaned: 0

           rx_queue_32_packets: 0

           rx_queue_32_bytes: 0

           rx_queue_32_bp_poll_yield: 0

           rx_queue_32_bp_misses: 0

           rx_queue_32_bp_cleaned: 0

           rx_queue_33_packets: 0

           rx_queue_33_bytes: 0

           rx_queue_33_bp_poll_yield: 0

           rx_queue_33_bp_misses: 0

           rx_queue_33_bp_cleaned: 0

           rx_queue_34_packets: 0

           rx_queue_34_bytes: 0

           rx_queue_34_bp_poll_yield: 0

           rx_queue_34_bp_misses: 0

           rx_queue_34_bp_cleaned: 0

           rx_queue_35_packets: 0

           rx_queue_35_bytes: 0

           rx_queue_35_bp_poll_yield: 0

           rx_queue_35_bp_misses: 0

           rx_queue_35_bp_cleaned: 0

           rx_queue_36_packets: 0

           rx_queue_36_bytes: 0

           rx_queue_36_bp_poll_yield: 0

           rx_queue_36_bp_misses: 0

           rx_queue_36_bp_cleaned: 0

           rx_queue_37_packets: 0

           rx_queue_37_bytes: 0

           rx_queue_37_bp_poll_yield: 0

           rx_queue_37_bp_misses: 0

           rx_queue_37_bp_cleaned: 0

           rx_queue_38_packets: 0

           rx_queue_38_bytes: 0

           rx_queue_38_bp_poll_yield: 0

           rx_queue_38_bp_misses: 0

           rx_queue_38_bp_cleaned: 0

           rx_queue_39_packets: 0

           rx_queue_39_bytes: 0

           rx_queue_39_bp_poll_yield: 0

           rx_queue_39_bp_misses: 0

           rx_queue_39_bp_cleaned: 0

           rx_queue_40_packets: 0

           rx_queue_40_bytes: 0

           rx_queue_40_bp_poll_yield: 0

           rx_queue_40_bp_misses: 0

           rx_queue_40_bp_cleaned: 0

           rx_queue_41_packets: 0

           rx_queue_41_bytes: 0

           rx_queue_41_bp_poll_yield: 0

           rx_queue_41_bp_misses: 0

           rx_queue_41_bp_cleaned: 0

           rx_queue_42_packets: 0

           rx_queue_42_bytes: 0

           rx_queue_42_bp_poll_yield: 0

           rx_queue_42_bp_misses: 0

           rx_queue_42_bp_cleaned: 0

           rx_queue_43_packets: 0

           rx_queue_43_bytes: 0

           rx_queue_43_bp_poll_yield: 0

           rx_queue_43_bp_misses: 0

           rx_queue_43_bp_cleaned: 0

           rx_queue_44_packets: 0

           rx_queue_44_bytes: 0

           rx_queue_44_bp_poll_yield: 0

           rx_queue_44_bp_misses: 0

           rx_queue_44_bp_cleaned: 0

           rx_queue_45_packets: 0

           rx_queue_45_bytes: 0

           rx_queue_45_bp_poll_yield: 0

           rx_queue_45_bp_misses: 0

           rx_queue_45_bp_cleaned: 0

           rx_queue_46_packets: 0

           rx_queue_46_bytes: 0

           rx_queue_46_bp_poll_yield: 0

           rx_queue_46_bp_misses: 0

           rx_queue_46_bp_cleaned: 0

           rx_queue_47_packets: 0

           rx_queue_47_bytes: 0

           rx_queue_47_bp_poll_yield: 0

           rx_queue_47_bp_misses: 0

           rx_queue_47_bp_cleaned: 0

           rx_queue_48_packets: 0

           rx_queue_48_bytes: 0

           rx_queue_48_bp_poll_yield: 0

           rx_queue_48_bp_misses: 0

           rx_queue_48_bp_cleaned: 0

           rx_queue_49_packets: 0

           rx_queue_49_bytes: 0

           rx_queue_49_bp_poll_yield: 0

           rx_queue_49_bp_misses: 0

           rx_queue_49_bp_cleaned: 0

           rx_queue_50_packets: 0

           rx_queue_50_bytes: 0

           rx_queue_50_bp_poll_yield: 0

           rx_queue_50_bp_misses: 0

           rx_queue_50_bp_cleaned: 0

           rx_queue_51_packets: 0

           rx_queue_51_bytes: 0

           rx_queue_51_bp_poll_yield: 0

           rx_queue_51_bp_misses: 0

           rx_queue_51_bp_cleaned: 0

           rx_queue_52_packets: 0

           rx_queue_52_bytes: 0

           rx_queue_52_bp_poll_yield: 0

           rx_queue_52_bp_misses: 0

           rx_queue_52_bp_cleaned: 0

           rx_queue_53_packets: 0

           rx_queue_53_bytes: 0

           rx_queue_53_bp_poll_yield: 0

           rx_queue_53_bp_misses: 0

           rx_queue_53_bp_cleaned: 0

           rx_queue_54_packets: 0

           rx_queue_54_bytes: 0

           rx_queue_54_bp_poll_yield: 0

           rx_queue_54_bp_misses: 0

           rx_queue_54_bp_cleaned: 0

           rx_queue_55_packets: 0

           rx_queue_55_bytes: 0

           rx_queue_55_bp_poll_yield: 0

           rx_queue_55_bp_misses: 0

           rx_queue_55_bp_cleaned: 0

           rx_queue_56_packets: 0

           rx_queue_56_bytes: 0

           rx_queue_56_bp_poll_yield: 0

           rx_queue_56_bp_misses: 0

           rx_queue_56_bp_cleaned: 0

           rx_queue_57_packets: 0

           rx_queue_57_bytes: 0

           rx_queue_57_bp_poll_yield: 0

           rx_queue_57_bp_misses: 0

           rx_queue_57_bp_cleaned: 0

           rx_queue_58_packets: 0

           rx_queue_58_bytes: 0

           rx_queue_58_bp_poll_yield: 0

           rx_queue_58_bp_misses: 0

           rx_queue_58_bp_cleaned: 0

           rx_queue_59_packets: 0

           rx_queue_59_bytes: 0

           rx_queue_59_bp_poll_yield: 0

           rx_queue_59_bp_misses: 0

           rx_queue_59_bp_cleaned: 0

           rx_queue_60_packets: 0

           rx_queue_60_bytes: 0

           rx_queue_60_bp_poll_yield: 0

           rx_queue_60_bp_misses: 0

           rx_queue_60_bp_cleaned: 0

           rx_queue_61_packets: 0

           rx_queue_61_bytes: 0

           rx_queue_61_bp_poll_yield: 0

           rx_queue_61_bp_misses: 0

           rx_queue_61_bp_cleaned: 0

           rx_queue_62_packets: 0

           rx_queue_62_bytes: 0

           rx_queue_62_bp_poll_yield: 0

           rx_queue_62_bp_misses: 0

           rx_queue_62_bp_cleaned: 0

           rx_queue_63_packets: 0

           rx_queue_63_bytes: 0

           rx_queue_63_bp_poll_yield: 0

           rx_queue_63_bp_misses: 0

           rx_queue_63_bp_cleaned: 0

           rx_queue_64_packets: 0

           rx_queue_64_bytes: 0

           rx_queue_64_bp_poll_yield: 0

           rx_queue_64_bp_misses: 0

           rx_queue_64_bp_cleaned: 0

           rx_queue_65_packets: 0

           rx_queue_65_bytes: 0

           rx_queue_65_bp_poll_yield: 0

           rx_queue_65_bp_misses: 0

           rx_queue_65_bp_cleaned: 0

           rx_queue_66_packets: 0

           rx_queue_66_bytes: 0

           rx_queue_66_bp_poll_yield: 0

           rx_queue_66_bp_misses: 0

           rx_queue_66_bp_cleaned: 0

           rx_queue_67_packets: 0

           rx_queue_67_bytes: 0

           rx_queue_67_bp_poll_yield: 0

           rx_queue_67_bp_misses: 0

           rx_queue_67_bp_cleaned: 0

           rx_queue_68_packets: 0

           rx_queue_68_bytes: 0

           rx_queue_68_bp_poll_yield: 0

           rx_queue_68_bp_misses: 0

           rx_queue_68_bp_cleaned: 0

           rx_queue_69_packets: 0

           rx_queue_69_bytes: 0

           rx_queue_69_bp_poll_yield: 0

           rx_queue_69_bp_misses: 0

           rx_queue_69_bp_cleaned: 0

           rx_queue_70_packets: 0

           rx_queue_70_bytes: 0

           rx_queue_70_bp_poll_yield: 0

           rx_queue_70_bp_misses: 0

           rx_queue_70_bp_cleaned: 0

           tx_pb_0_pxon: 0

           tx_pb_0_pxoff: 0

           tx_pb_1_pxon: 0

           tx_pb_1_pxoff: 0

           tx_pb_2_pxon: 0

           tx_pb_2_pxoff: 0

           tx_pb_3_pxon: 0

           tx_pb_3_pxoff: 0

           tx_pb_4_pxon: 0

           tx_pb_4_pxoff: 0

           tx_pb_5_pxon: 0

           tx_pb_5_pxoff: 0

           tx_pb_6_pxon: 0

           tx_pb_6_pxoff: 0

           tx_pb_7_pxon: 0

           tx_pb_7_pxoff: 0

           rx_pb_0_pxon: 0

           rx_pb_0_pxoff: 0

           rx_pb_1_pxon: 0

           rx_pb_1_pxoff: 0

           rx_pb_2_pxon: 0

           rx_pb_2_pxoff: 0

           rx_pb_3_pxon: 0

           rx_pb_3_pxoff: 0

           rx_pb_4_pxon: 0

           rx_pb_4_pxoff: 0

           rx_pb_5_pxon: 0

           rx_pb_5_pxoff: 0

           rx_pb_6_pxon: 0

           rx_pb_6_pxoff: 0

           rx_pb_7_pxon: 0

           rx_pb_7_pxoff: 0