Ethernet Products
Determine ramifications of Intel® Ethernet products and technologies
4810 Discussions

vxlan: non-ECT with TOS=0x02 logs were generated

idata
Employee
5,625 Views

Hi

I use intel 1G and 10G ethernet card with openvswitch under vxlan overlay network.

Some VM started to use ECN and I found weird logs were generated about non-ECT.

I checked vxlan packets, but all packets had no errors. Also, vxlan outer header's ECN was normal acording to RFC6040.

I changed ethernet card to Broadcom, and I found there was no non-ECT log.

I need to stop wrong non-ECT logs. How should I do?

Please see below informations:

[root@compute01 ~]# grep vxlan /var/log/messages | tail

Dec 13 10:32:57 compute01 kernel: vxlan: non-ECT from 192.168.40.55 with TOS=0x2

Dec 13 10:32:57 compute01 kernel: vxlan: non-ECT from 192.168.40.55 with TOS=0x2

Dec 13 10:32:57 compute01 kernel: vxlan: non-ECT from 192.168.40.55 with TOS=0x2

Dec 13 10:32:57 compute01 kernel: vxlan: non-ECT from 192.168.40.55 with TOS=0x2

Dec 13 10:32:57 compute01 kernel: vxlan: non-ECT from 192.168.40.55 with TOS=0x2

Dec 13 10:32:57 compute01 kernel: vxlan: non-ECT from 192.168.40.55 with TOS=0x2

Dec 13 10:32:57 compute01 kernel: vxlan: non-ECT from 192.168.40.55 with TOS=0x2

Dec 13 10:32:57 compute01 kernel: vxlan: non-ECT from 192.168.40.55 with TOS=0x2

Dec 13 10:33:32 compute01 kernel: vxlan: non-ECT from 192.168.40.55 with TOS=0x2

Dec 13 10:33:34 compute01 kernel: vxlan: non-ECT from 192.168.40.55 with TOS=0x2

[root@compute01 ~]# uname -a

Linux compute01 3.10.0-693.5.2.el7.x86_64 # 1 SMP Fri Oct 20 20:32:50 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

[root@compute01 ~]# ethtool -i eth4

driver: ixgbe

version: 4.4.0-k-rh7.4

firmware-version: 0x80000609

expansion-rom-version:

bus-info: 0000:04:00.0

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: no

[root@compute01 ~]# ethtool -k eth4

Features for eth4:

rx-checksumming: on

tx-checksumming: on

tx-checksum-ipv4: off [fixed]

tx-checksum-ip-generic: on

tx-checksum-ipv6: off [fixed]

tx-checksum-fcoe-crc: on [fixed]

tx-checksum-sctp: on

scatter-gather: on

tx-scatter-gather: on

tx-scatter-gather-fraglist: off [fixed]

tcp-segmentation-offload: on

tx-tcp-segmentation: on

tx-tcp-ecn-segmentation: off [fixed]

tx-tcp6-segmentation: on

tx-tcp-mangleid-segmentation: off

udp-fragmentation-offload: off [fixed]

generic-segmentation-offload: on

generic-receive-offload: on

large-receive-offload: off

rx-vlan-offload: on

tx-vlan-offload: on

ntuple-filters: off

receive-hashing: on

highdma: on [fixed]

rx-vlan-filter: on

vlan-challenged: off [fixed]

tx-lockless: off [fixed]

netns-local: off [fixed]

tx-gso-robust: off [fixed]

tx-fcoe-segmentation: on [fixed]

tx-gre-segmentation: on

tx-ipip-segmentation: on

tx-sit-segmentation: on

tx-udp_tnl-segmentation: on

tx-mpls-segmentation: off [fixed]

fcoe-mtu: off [fixed]

tx-nocache-copy: off

loopback: off [fixed]

rx-fcs: off [fixed]

rx-all: off

tx-vlan-stag-hw-insert: off [fixed]

rx-vlan-stag-hw-parse: off [fixed]

rx-vlan-stag-filter: off [fixed]

busy-poll: on [fixed]

tx-gre-csum-segmentation: on

tx-udp_tnl-csum-segmentation: on

tx-gso-partial: on

tx-sctp-segmentation: off [fixed]

l2-fwd-offload: off

hw-tc-offload: off [fixed]

///////////////////////////////////////////////////////////////////

root@oscompute01:~# grep vxlan /var/log/syslog | tail

Dec 13 18:53:50 oscompute01 kernel: [188924.432135] vxlan: non-ECT from 192.168.2.32 with TOS=0x2

Dec 13 18:53:51 oscompute01 kernel: [188925.431957] vxlan: non-ECT from 192.168.2.32 with TOS=0x2

Dec 13 18:53:51 oscompute01 kernel: [188925.432004] vxlan: non-ECT from 192.168.2.32 with TOS=0x2

Dec 13 18:53:51 oscompute01 kernel: [188925.447808] vxlan: non-ECT from 192.168.2.32 with TOS=0x2

Dec 13 18:53:52 oscompute01 kernel: [188926.447743] vxlan: non-ECT from 192.168.2.32 with TOS=0x2

Dec 13 18:53:52 oscompute01 kernel: [188926.463339] vxlan: non-ECT from 192.168.2.32 with TOS=0x2

Dec 13 18:53:54 oscompute01 kernel: [188928.463280] vxlan: non-ECT from 192.168.2.32 with TOS=0x2

Dec 13 18:53:54 oscompute01 kernel: [188928.478883] vxlan: non-ECT from 192.168.2.32 with TOS=0x2

Dec 13 18:53:55 oscompute01 kernel: [188929.463051] vxlan: non-ECT from 192.168.2.32 with TOS=0x2

Dec 13 18:53:55 oscompute01 kernel: [188929.478835] vxlan: non-ECT from 192.168.2.32 with TOS=0x2

root@oscompute01:~# uname -a

Linux oscompute01 4.4.0-103-generic # 126-Ubuntu SMP Mon Dec 4 16:23:28 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

root@oscompute01:~# ethtool -i eth1

driver: igb

version: 5.3.5.12

firmware-version: 1.70, 0x80000f44, 1.1752.0

expansion-rom-version:

bus-info: 0000:04:00.1

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: no

root@oscompute01:~# ethtool -k eth1

Features for eth1:

rx-checksumming: on

tx-checksumming: on

tx-checksum-ipv4: ontx-checksum-ip-generic: off [fixed]tx-checksum-ipv6: ontx-checksum-fcoe-crc: off [fixed]tx-checksum-sctp: off [fixed]

scatter-gather: on

tx-scatter-gather: ontx-scatter-gather-fraglist: off [fixed]

tcp-segmentation-offload: on

tx-tcp-segmentation: ontx-tcp-ecn-segmentation: off [fixed]tx-tcp6-segmentation: on

udp-fragmentation-offload: off [fixed]

generic-segmentation-offload: on

generic-receive-offload: on

large-receive-offload: off [requested on]

rx-vlan-offload: on

tx-vlan-offload: on

ntuple-filters: off [fixed]

receive-hashing: on

highdma: on [fixed]

rx-vlan-filter: on [fixed]

vlan-challenged: off [fixed]

tx-lockless: off [fixed]

netns-local: off [fixed]

tx-gso-robust: off [fixed]

tx-fcoe-segmentation: off [fixed]

tx-gre-segmentation: off [fixed]

tx-ipip-segmentation: off [fixed]

tx-sit-segmentation: off [fixed]

tx-udp_tnl-segmentation: off [fixed]

fcoe-mtu: off [fixed]

<p...
0 Kudos
15 Replies
idata
Employee
3,124 Views

Hi Woobee,

 

 

Thank you for posting in Wired Communities. Can you share what is the model of the Intel NIC you used? Are they onboard NIC or standalone NIC?

 

 

Thanks,

Sharon

 

0 Kudos
idata
Employee
3,124 Views

Hi Sharon,

I used I350 as a standalone NIC, and X540 as a onboard(LOM) NIC.

- HP 669280-001 Ethernet 1Gb 4-Port 366FLR Adapter

- HP 701525-001 ETHERNET 10GB 2-PORT 561FLR-T ADAPTER

If you want to reproduce ECN easily in lab, please refer below url:

https://gist.github.com/bauer1j/1320355/ef7a8aad7373858cc6230f05f2d9c0f02aee5348 https://gist.github.com/bauer1j/1320355/ef7a8aad7373858cc6230f05f2d9c0f02aee5348

Thanks,

Woobee

0 Kudos
idata
Employee
3,124 Views

Hi Woobee,

 

 

Thank you for the information. I will further investigate and update you if there is any finding.

 

 

Regards,

 

Sharon

 

0 Kudos
idata
Employee
3,124 Views

Hi Woobee,

 

 

Please try using the driver below for the Intel NICs. Do let me know the result.

 

 

https://sourceforge.net/projects/e1000/files/?source=navbar

 

 

Thanks,

 

Sharon

 

0 Kudos
idata
Employee
3,124 Views

Hi Sharon,

I've already used recent drivers, but result were same as before.

- igb stable 5.3.5.12 (lastest driver is 5.3.5.12)

- ixgbe stable 5.3.3 (lastest driver is 5.3.4)

Thanks,

Woobee

0 Kudos
idata
Employee
3,124 Views

Hi Woobee,

 

 

Thank you for the update. I will further check on this.

 

 

Regards,

 

Sharon
0 Kudos
idata
Employee
3,124 Views

Hi Woobee,

 

 

Just to double check I found that you are using HP* network adapter as you mentioned you are using I350 and X540 with below part number

 

 

- HP 669280-001 Ethernet 1Gb 4-Port 366FLR Adapter

 

- HP 701525-001 ETHERNET 10GB 2-PORT 561FLR-T ADAPTER

 

 

These are HP* network adapters, have you contacted HP* for the support?

 

 

Thanks,

 

Sharon

 

 

0 Kudos
idata
Employee
3,124 Views

Hi Sharon,

I requested support to HP, and they said:

-They inspected this log, but there were no history about this log.

-They inspected HW log of server, but no problem found.

-They requested support result of OS vendor, but we do not have OS support becase we use CentOS and Ubuntu.

This logs were generated from vxlan driver of Linux kernel, but only generated in Intel's NIC.

https://github.com/torvalds/linux/blob/master/drivers/net/vxlan.c# L1320 linux/vxlan.c at master · torvalds/linux · GitHub

We are considering replace from Intel's NIC to Brodcom's NIC.

I hope this problem is solved as soon as possible.

Thanks,

woobee

0 Kudos
idata
Employee
3,124 Views

Hi Woobee,

 

 

Thank you for the information. What is the Intel NIC model ? The one you provided on the thread are HP* NIC.

 

 

 

Thanks,

 

Sharon

 

0 Kudos
idata
Employee
3,124 Views

Hi Sharon,

The HP NIC that I mentioned uses Intel Processor and Intel network addaptor driver.

Please see following summarized information:

1. HP 669280-001 Ethernet 1Gb 4-Port 366FLR Adapter

 

- Product Information: https://www.hpe.com/us/en/product-catalog/options/pip.specifications.hpe-ethernet-1gb-4-port-366flr-adapter.5288150.html

 

- Network Processor : Intel I350 Controller

 

- NIC driver and firmware information:

 

root@oscompute01:~# ethtool -i eth1

 

driver: igb

 

version: 5.3.5.12

 

firmware-version: 1.70, 0x80000f44, 1.1752.0

2. HP 701525-001 ETHERNET 10GB 2-PORT 561FLR-T ADAPTER

 

- Product Information: https://www.hpe.com/us/en/product-catalog/options/pip.specifications.hpe-ethernet-10gb-2-port-561flr-t-adapter.5404516.html

 

- Network Processor : Intel X540 Controller

 

- NIC driver and firmware information:

 

[root@compute01 ~]# ethtool -i eth4

 

driver: ixgbe

 

version: 4.4.0-k-rh7.4

 

firmware-version: 0x80000609

Thanks,

Woobee

0 Kudos
idata
Employee
3,124 Views

Hi Woobee,

 

 

Thank you for the clarification. It is best to contact HP* still as this is not Intel's network adapter but HP*'s, for OEM network adapter it is recommended to contact the OEM vendor.

 

https://www.intel.com/content/www/us/en/support/articles/000006628/network-and-i-o/ethernet-products.html?wapkw=oem+network

 

 

Please double check with HP* for customized drivers which are more suitable for the HP* network adapter.

 

 

Regards,

 

Sharon

 

0 Kudos
idata
Employee
3,124 Views

Hi Sharon

Even if the NIC is HP*'s, the chipset is Intel's. Therefore, the driver typically uses what Intel provides, such as ixgbe or igb.

In my case, I installed Ubuntu or CentOS and used automatically configured Intel drivers, such as ixgbe or igb.

So I contacted to Intel for driver support, rather than contact HP.

If you can no longer help me, I'll try to find another way.

Regards,

Woobee

0 Kudos
idata
Employee
3,124 Views

Hi Woobee,

 

 

Thank you for the information. The driver we have is generic version for Intel network adapter,

 

https://downloadcenter.intel.com/download/14687/Intel-Network-Adapter-Driver-for-PCIe-Intel-10-Gigabit-Ethernet-Network-Connections-Under-Linux-?product=59787

 

 

I can see you are using the latest version 5.3.5 from sourceforge as well. if you are using HP* network adapter, it is recommended to double check with them for customized driver that are suitable for your OEM HP network adapter.

 

 

Hope this clarifies.

 

 

Regards,

 

Sharon

 

0 Kudos
idata
Employee
3,124 Views

Hi Sharon,

I'll check this issue with HP.

If I need additional help, I'll post in Wired Communities again.

Thanks for your support.

Regards,

Woobee

0 Kudos
idata
Employee
3,124 Views

Hi Woobee,

 

 

You are welcome and feel free to update us.

 

 

Regards,

 

Sharon

 

0 Kudos
Reply