8 Replies Latest reply on Aug 24, 2015 10:20 PM by Sandy_Intel

    Intel X540-AT2 hardware errors / crash on link state change - ixgbe - Kernel 4.1.3

    s_mcleod

       

      [126295.760534] ------------[ cut here ]------------
      
      [126295.760567] WARNING: CPU: 10 PID: 0 at net/sched/sch_generic.c:303 dev_watchdog+0x24f/0x260()
      
      [126295.760572] NETDEV WATCHDOG: eno2 (ixgbe): transmit queue 8 timed out
      
      [126295.760575] Modules linked in: fuse(E) btrfs(E) xor(E) raid6_pq(E) ufs(E) hfsplus(E) hfs(E) vfat(E) msdos(E) fat(E) xfs(E) binfmt_misc(E) target_core_user(E) uio(E) target_core_pscsi(E) target_core_file(E) target_core_iblock(E) iscsi_target_mod(E) drbd(E) lru_cache(E) libcrc32c(E) target_core_mod(E) iptable_filter(E) ip_tables(E) bonding(E) dm_mod(E) iTCO_wdt(E) iTCO_vendor_support(E) x86_pkg_temp_thermal(E) intel_powerclamp(E) coretemp(E) kvm_intel(E) kvm(E) crct10dif_pclmul(E) crc32_pclmul(E) crc32c_intel(E) ghash_clmulni_intel(E) raid10(E) aesni_intel(E) lrw(E) gf128mul(E) glue_helper(E) ablk_helper(E) cryptd(E) pcspkr(E) sb_edac(E) edac_core(E) i2c_i801(E) joydev(E) lpc_ich(E) mei_me(E) ioatdma(E) mfd_core(E) mei(E) shpchp(E) wmi(E) ipmi_devintf(E) ipmi_si(E) 8250_fintek(E) ipmi_msghandler(E)
      
      [126295.760641]  acpi_power_meter(E) acpi_pad(E) ext4(E) mbcache(E) jbd2(E) raid1(E) sd_mod(E) ast(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) i2c_algo_bit(E) drm_kms_helper(E) ttm(E) drm(E) ahci(E) libahci(E) ixgbe(E) mdio(E) libata(E) ptp(E) pps_core(E) nvme(E) dca(E)
      
      [126295.760670] CPU: 10 PID: 0 Comm: swapper/10 Tainted: G            E   4.1.3-1.el7.elrepo.x86_64 #1
      
      [126295.760674] Hardware name: Supermicro X10DRW-E/X10DRW-NT, BIOS 1.0a 01/07/2015
      
      [126295.760679]  0000000000000000 b7a43922beff28d0 ffff88087fd03d28 ffffffff816d6058
      
      [126295.760686]  0000000000000000 ffff88087fd03d80 ffff88087fd03d68 ffffffff8107d51a
      
      [126295.760691]  0000000000000000 0000000000000008 ffff88046a7e0000 0000000000000040
      
      [126295.760696] Call Trace:
      
      [126295.760700]    [] dump_stack+0x45/0x57
      
      [126295.760720]  [] warn_slowpath_common+0x8a/0xc0
      
      [126295.760725]  [] warn_slowpath_fmt+0x55/0x70
      
      [126295.760734]  [] dev_watchdog+0x24f/0x260
      
      [126295.760739]  [] ? dev_graft_qdisc+0x80/0x80
      
      [126295.760750]  [] call_timer_fn+0x39/0x110
      
      [126295.760754]  [] ? dev_graft_qdisc+0x80/0x80
      
      [126295.760760]  [] run_timer_softirq+0x240/0x350
      
      [126295.760771]  [] ? lapic_next_deadline+0x33/0x40
      
      [126295.760777]  [] __do_softirq+0xf4/0x2d0
      
      [126295.760782]  [] irq_exit+0x125/0x130
      
      [126295.760792]  [] smp_apic_timer_interrupt+0x4a/0x60
      
      [126295.760798]  [] apic_timer_interrupt+0x6e/0x80
      
      [126295.760801]    [] ? cpuidle_enter_state+0xa9/0x1f0
      
      [126295.760815]  [] ? cpuidle_enter_state+0x78/0x1f0
      
      [126295.760821]  [] cpuidle_enter+0x17/0x20
      
      [126295.760828]  [] cpu_startup_entry+0x35c/0x3f0
      
      [126295.760835]  [] start_secondary+0x173/0x1e0
      
      [126295.760839] ---[ end trace 3143549a7bfdab83 ]---
      
      [126295.760847] ixgbe 0000:01:00.1 eno2: initiating reset due to tx timeout
      
      [126295.761042] ixgbe 0000:01:00.1 eno2: Reset adapter
      
      
        *-network:0
      
             description: Ethernet interface
      
             product: Ethernet Controller 10-Gigabit X540-AT2
      
             vendor: Intel Corporation
      
             physical id: 0
      
             bus info: pci@0000:01:00.0
      
             logical name: eno1
      
             version: 01
      
             serial: 00:25:90:fa:60:7e
      
             size: 10Gbit/s
      
             width: 64 bits
      
             clock: 33MHz
      
             capabilities: pm msi msix pciexpress bus_master cap_list ethernet physical tp 100bt-fd 1000bt-fd autonegotiation
      
             configuration: autonegotiation=on broadcast=yes driver=ixgbe driverversion=4.0.1-k duplex=full firmware=0x800003e2 latency=0 link=yes multicast=yes port=twisted pair slave=yes speed=10Gbit/s
      
             resources: irq:24 memory:c7800000-c79fffff ioport:6020(size=32) memory:c7a04000-c7a07fff memory:90000000-900fffff memory:90100000-901fffff
      
        *-network:1
      
             description: Ethernet interface
      
             product: Ethernet Controller 10-Gigabit X540-AT2
      
             vendor: Intel Corporation
      
             physical id: 0.1
      
             bus info: pci@0000:01:00.1
      
             logical name: eno2
      
             version: 01
      
             serial: 00:25:90:fa:60:7e
      
             size: 10Gbit/s
      
             width: 64 bits
      
             clock: 33MHz
      
             capabilities: pm msi msix pciexpress bus_master cap_list ethernet physical tp 100bt-fd 1000bt-fd autonegotiation
      
             configuration: autonegotiation=on broadcast=yes driver=ixgbe driverversion=4.0.1-k duplex=full firmware=0x800003e2 latency=0 link=yes multicast=yes port=twisted pair slave=yes speed=10Gbit/s
      
             resources: irq:50 memory:c7600000-c77fffff ioport:6000(size=32) memory:c7a00000-c7a03fff memory:90200000-902fffff memory:90300000-903fffff
      
      

       

      Offtopic here: It's taken me longer to create a login, verify my email 4 times, choose a username twice, accept Intel's broken SSL cert and write this reply than it did for me to find this bug. intel_admin your forums / community is woeful - no wonder no one replies to issues here.

        • 1. Re: Intel X540-AT2 hardware errors / crash on link state change - ixgbe - Kernel 4.1.3
          Sandy_Intel

          Hi s_mcleod,

           

          Thank you for contacting Intel.

           

          We would like to request for additional details in order to understand your environment.  Please fill up information below:

          1. Operating System:

          2. Driver version:

          3. Steps to replicate the issue.

           

          We look forward to your reply.

           

          Sincerely,

           

          Sandy

          • 2. Re: Intel X540-AT2 hardware errors / crash on link state change - ixgbe - Kernel 4.1.3
            s_mcleod

            Hi, that information is already in my post:

             

            1. Operating System: CentOS 7.1 (RHEL7) / Kernel 4.1.3

            2. Driver version: Stock Kernel 4.1.3 driver which is 4.0.1-k

            3. Steps to replicate the issue: Using bonding mode 1 (active/passive) the issue happens intermittently when the link state changes.

            • 3. Re: Intel X540-AT2 hardware errors / crash on link state change - ixgbe - Kernel 4.1.3
              Sandy_Intel

              Hi s_mcleod,


              Thanks for replying.


              I see that you are using the built-in driver in CentOS*.  Please try installing our driver.  You may download the driver here: 

              Intel-Ethernet-Controller-X540-AT2 Drivers


              Alternatively, you can also check for tested driver provided by your system or motherboard manufacturer.


              We await your test results.


              Sincerely,


              Sandy

              • 4. Re: Intel X540-AT2 hardware errors / crash on link state change - ixgbe - Kernel 4.1.3
                s_mcleod

                Hi Sandy,

                 

                Two questions:

                 

                1) Thanks for the link, I see that the driver you've suggest is a few minor revisions ahead of whats in the current Linux kernel, tell me - in those minor revisions are there any updates that could relate to this crash?

                2) The drive you've linked me to only seems to support old Linux kernels 'latest 2.4 series kernel as well as 2.6.x and 3.x versions' - no mention of the current stable Linux Kernels 4/4.1/4.2 etc... ?

                 

                Unfortunately Supermicro's website is very hard to navigate and they're not very good at releasing driver updates.

                • 5. Re: Intel X540-AT2 hardware errors / crash on link state change - ixgbe - Kernel 4.1.3
                  wb_Intel

                  Hi mcleod,

                   

                                  The driver on our website support kernel version from 2.6.18 up through 3.19.4. Based on the release notes http://downloadmirror.intel.com/24941/eng/readmeixgbe.txt below are the changes in this release

                  Changes in this release:

                  * Various Bug fixes

                  * Support for new X550 hardware (new features not enabled yet)

                  * VxLan RX checksump offload support

                  * Ubuntu version check in makefile

                   

                   

                  rgds,

                  wb

                  • 6. Re: Intel X540-AT2 hardware errors / crash on link state change - ixgbe - Kernel 4.1.3
                    s_mcleod

                    Hi wb,

                     

                    The stable kernel version is currently 4.1.3 though?

                     

                    - What does 'Various Bug fixes' entail?

                    - Your source code fails to build:

                     

                    root@test-centos-02:~ 1 # rpmbuild -tb ixgbe-4.0.3.tar.gz
                    Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.pgg0To
                    + umask 022
                    + cd /root/rpmbuild/BUILD
                    + cd /root/rpmbuild/BUILD
                    + rm -rf ixgbe-4.0.3
                    + /usr/bin/gzip -dc /root/ixgbe-4.0.3.tar.gz
                    + /usr/bin/tar -xvvf -
                    drwxr-xr-x root/root         0 2015-04-08 02:52 ixgbe-4.0.3/
                    -rw-r--r-- root/root    412417 2015-08-03 16:11 ixgbe-4.0.3/COPYING
                    -rw-r--r-- root/root      9550 2015-04-08 02:52 ixgbe-4.0.3/ixgbe.spec
                    -rw-r--r-- root/root      5682 2015-04-08 02:52 ixgbe-4.0.3/pci.updates
                    -rw-r--r-- root/root      2048 2015-04-08 02:52 ixgbe-4.0.3/SUMS
                    -rwxr-xr-x root/root      7257 2015-04-08 02:52 ixgbe-4.0.3/ixgbe.7
                    drwxr-xr-x root/root         0 2015-04-08 02:52 ixgbe-4.0.3/scripts/
                    -rwxr-xr-x root/root      6072 2015-04-08 02:52 ixgbe-4.0.3/scripts/set_irq_affinity
                    drwxr-xr-x root/root         0 2015-04-08 02:52 ixgbe-4.0.3/src/
                    -rw-r--r-- root/root     29598 2015-04-08 02:52 ixgbe-4.0.3/src/kcompat_ethtool.c
                    -rw-r--r-- root/root      6549 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_osdep.h
                    -rw-r--r-- root/root     52160 2015-04-08 02:52 ixgbe-4.0.3/src/kcompat.c
                    -rw-r--r-- root/root      2444 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_x540.h
                    -rw-r--r-- root/root    156012 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_type.h
                    -rw-r--r-- root/root    300605 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_main.c
                    -rw-r--r-- root/root     23259 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_dcb_nl.c
                    -rw-r--r-- root/root      3478 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_dcb_82598.h
                    -rw-r--r-- root/root      6343 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_mbx.h
                    -rw-r--r-- root/root     76940 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_82599.c
                    -rw-r--r-- root/root     39690 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_sriov.c
                    -rw-r--r-- root/root      3080 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_sriov.h
                    -rw-r--r-- root/root     20490 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_dcb.c
                    -rw-r--r-- root/root      5627 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_dcb.h
                    -rw-r--r-- root/root     45777 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_api.c
                    -rw-r--r-- root/root      7217 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_sysfs.c
                    -rw-r--r-- root/root      4276 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_dcb_82599.h
                    -rw-r--r-- root/root      7486 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_debugfs.c
                    -rw-r--r-- root/root     12481 2015-04-08 02:52 ixgbe-4.0.3/src/Makefile
                    -rw-r--r-- root/root     74719 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_phy.c
                    -rw-r--r-- root/root    135006 2015-04-08 02:52 ixgbe-4.0.3/src/kcompat.h
                    -rw-r--r-- root/root      4634 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_x550.h
                    -rw-r--r-- root/root        18 2015-04-08 02:52 ixgbe-4.0.3/src/Module.supported
                    -rw-r--r-- root/root     28711 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_x540.c
                    -rw-r--r-- root/root      7782 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_phy.h
                    -rw-r--r-- root/root     40160 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_82598.c
                    -rw-r--r-- root/root      1890 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_osdep2.h
                    -rw-r--r-- root/root     31347 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_param.c
                    -rw-r--r-- root/root     10009 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_dcb_82598.c
                    -rw-r--r-- root/root     35997 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe.h
                    -rw-r--r-- root/root    107371 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_ethtool.c
                    -rw-r--r-- root/root     36880 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_lib.c
                    -rw-r--r-- root/root     28978 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_fcoe.c
                    -rw-r--r-- root/root    136359 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_common.c
                    -rw-r--r-- root/root      2635 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_fcoe.h
                    -rw-r--r-- root/root      2622 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_82599.h
                    -rw-r--r-- root/root     82609 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_x550.c
                    -rw-r--r-- root/root     19527 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_mbx.c
                    -rw-r--r-- root/root      9706 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_api.h
                    -rw-r--r-- root/root     27304 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_procfs.c
                    -rw-r--r-- root/root     16485 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_dcb_82599.c
                    -rw-r--r-- root/root      1923 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_82598.h
                    -rw-r--r-- root/root      7353 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_common.h
                    -rw-r--r-- root/root     48850 2015-04-08 02:52 ixgbe-4.0.3/src/ixgbe_ptp.c
                    -rwxr-xr-x root/root     45909 2015-04-08 02:52 ixgbe-4.0.3/README
                    + STATUS=0
                    + '[' 0 -ne 0 ']'
                    + cd ixgbe-4.0.3
                    + /usr/bin/chmod -Rf a+rX,u+w,g-w,o-w .
                    + exit 0
                    Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.jVb4KF
                    + umask 022
                    + cd /root/rpmbuild/BUILD
                    + cd ixgbe-4.0.3
                    + make -C src clean
                    make: Entering directory `/root/rpmbuild/BUILD/ixgbe-4.0.3/src'
                    make -C /lib/modules/4.1.3-1.el7.elrepo.x86_64/build SUBDIRS=/root/rpmbuild/BUILD/ixgbe-4.0.3/src clean
                    make[1]: Entering directory `/usr/src/kernels/4.1.3-1.el7.elrepo.x86_64'
                    make[1]: Leaving directory `/usr/src/kernels/4.1.3-1.el7.elrepo.x86_64'
                    rm -rf ixgbe.ko ixgbe.o ixgbe.mod.c ixgbe.mod.o ixgbe_main.o ixgbe_common.o ixgbe_api.o ixgbe_param.o ixgbe_lib.o ixgbe_ethtool.o kcompat.o ixgbe_82598.o ixgbe_82599.o ixgbe_x540.o ixgbe_x550.o ixgbe_sriov.o ixgbe_mbx.o ixgbe_dcb.o ixgbe_dcb_82598.o ixgbe_dcb_82599.o ixgbe_sysfs.o ixgbe_procfs.o ixgbe_phy.o ixgbe_dcb_nl.o ixgbe_fcoe.o ixgbe_debugfs.o ixgbe_ptp.o ixgbe.7.gz .*cmd .tmp_versions
                    make: Leaving directory `/root/rpmbuild/BUILD/ixgbe-4.0.3/src'
                    + make -C src
                    make: Entering directory `/root/rpmbuild/BUILD/ixgbe-4.0.3/src'
                    make -C /lib/modules/4.1.3-1.el7.elrepo.x86_64/build SUBDIRS=/root/rpmbuild/BUILD/ixgbe-4.0.3/src modules
                    make[1]: Entering directory `/usr/src/kernels/4.1.3-1.el7.elrepo.x86_64'
                      CC [M]  /root/rpmbuild/BUILD/ixgbe-4.0.3/src/ixgbe_main.o
                    /root/rpmbuild/BUILD/ixgbe-4.0.3/src/ixgbe_main.c: In function ‘ixgbe_ndo_bridge_getlink’:
                    /root/rpmbuild/BUILD/ixgbe-4.0.3/src/ixgbe_main.c:9475:2: error: too few arguments to function ‘ndo_dflt_bridge_getlink’
                      return ndo_dflt_bridge_getlink(skb, pid, seq, dev, mode, 0, 0);
                      ^
                    In file included from include/net/dst.h:13:0,
                                     from include/net/sock.h:67,
                                     from include/linux/tcp.h:22,
                                     from /root/rpmbuild/BUILD/ixgbe-4.0.3/src/ixgbe_main.c:37:
                    include/linux/rtnetlink.h:123:12: note: declared here
                     extern int ndo_dflt_bridge_getlink(struct sk_buff *skb, u32 pid, u32 seq,
                                ^
                    /root/rpmbuild/BUILD/ixgbe-4.0.3/src/ixgbe_main.c: At top level:
                    /root/rpmbuild/BUILD/ixgbe-4.0.3/src/ixgbe_main.c:9563:2: warning: initialization from incompatible pointer type [enabled by default]
                      .ndo_bridge_getlink = ixgbe_ndo_bridge_getlink,
                      ^
                    /root/rpmbuild/BUILD/ixgbe-4.0.3/src/ixgbe_main.c:9563:2: warning: (near initialization for ‘ixgbe_netdev_ops.ndo_bridge_getlink’) [enabled by default]
                    /root/rpmbuild/BUILD/ixgbe-4.0.3/src/ixgbe_main.c: In function ‘ixgbe_ndo_bridge_getlink’:
                    /root/rpmbuild/BUILD/ixgbe-4.0.3/src/ixgbe_main.c:9479:1: warning: control reaches end of non-void function [-Wreturn-type]
                     }
                     ^
                    make[2]: *** [/root/rpmbuild/BUILD/ixgbe-4.0.3/src/ixgbe_main.o] Error 1
                    make[1]: *** [_module_/root/rpmbuild/BUILD/ixgbe-4.0.3/src] Error 2
                    make[1]: Leaving directory `/usr/src/kernels/4.1.3-1.el7.elrepo.x86_64'
                    make: *** [default] Error 2
                    make: Leaving directory `/root/rpmbuild/BUILD/ixgbe-4.0.3/src'
                    error: Bad exit status from /var/tmp/rpm-tmp.jVb4KF (%build)
                    
                    
                    RPM build errors:
                        Bad exit status from /var/tmp/rpm-tmp.jVb4KF (%build)
                    
                    • 7. Re: Intel X540-AT2 hardware errors / crash on link state change - ixgbe - Kernel 4.1.3
                      Sandy_Intel

                      Hi s_mcleod,

                       

                      Thanks for the updates.  I'll check on this.

                       

                      Sincerely,

                       

                      Sandy

                      • 8. Re: Intel X540-AT2 hardware errors / crash on link state change - ixgbe - Kernel 4.1.3
                        Sandy_Intel

                        Hi s_mcleod,

                         

                        The logs attached does not provide complete description of what the issue is.  Please provide information below:

                         

                        1. Describe the issue that you are seeing.

                        2. What is meant by link state change and what is causing that change?

                        3. Step by step procedure on how the issue can be replicated.

                        4. Platform:

                        - Supermicro X10DRW-E/X10DRW-NT

                        - Onboard X540-AT2 Dual Port NIC

                        - Operating Systems used: CentOS 7.1 (RHEL7) / Kernel 4.1.3

                        - Driver version: Stock Kernel 4.1.3 driver which is 4.0.1-k

                         

                        Sincerely,

                         

                        Sandy