4 Replies Latest reply on Sep 15, 2017 9:32 AM by moyes

    Performance Degradation in CPU v4 vs CPU v3

    aasraoui

      We are running IOR benchmarking on a Lustre filesystem and we see performance degradation when upgrading from cpu v3 to cpu v4;

       

      We are seeing cpus overload under IOR benchmark, this is occurring after cpu v3 to v4 upgrade

      As a result, we see performance degradation after the upgrade..

       

      can someone take a look at the syslog to see if this is a known issue:

       

      Syslog messages and snapshot of the cpus utilizations are below:

       

      kernel:NMI watchdog: BUG: soft lockup - CPU#54 stuck for 22s! [ll_ost_io01_107:24645]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:53:49 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_00_13:7854]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:53:49 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#10 stuck for 23s! [ptlrpcd_00_07:7848]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:53:49 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#46 stuck for 22s! [ll_ost_io01_047:49948]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:53:49 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#48 stuck for 22s! [ll_ost_io01_092:24578]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:53:53 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [ptlrpcd_00_19:7860]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:53:53 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#27 stuck for 23s! [ll_ost_io01_086:22338]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_00_08:7849]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#12 stuck for 22s! [ptlrpcd_00_14:7855]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [ll_ost_io01_008:50182]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#17 stuck for 22s! [kiblnd_sd_01_02:7835]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#18 stuck for 22s! [kiblnd_sd_01_01:7834]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#20 stuck for 22s! [ll_ost_io01_016:44925]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#21 stuck for 22s! [ll_ost_io01_076:22069]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#22 stuck for 22s! [kiblnd_sd_01_00:7833]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#23 stuck for 22s! [ll_ost_io01_094:24592]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [kiblnd_sd_01_03:7836]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#25 stuck for 22s! [ll_ost_io01_106:24643]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#26 stuck for 22s! [ll_ost_io01_064:33777]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#36 stuck for 22s! [kiblnd_sd_00_00:7829]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#43 stuck for 22s! [ll_ost_io01_031:42636]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#44 stuck for 22s! [ll_ost_io01_109:24689]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#49 stuck for 22s! [ll_ost_io01_046:50178]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#53 stuck for 22s! [ll_ost_io01_029:37763]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:54:01 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [ptlrpcd_00_09:7850]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:54:01 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#16 stuck for 22s! [ll_ost_io01_098:24613]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:54:01 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#32 stuck for 22s! [ptlrpcd_00_12:7853]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:54:01 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#42 stuck for 22s! [ll_ost_io01_024:39085]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:54:01 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#51 stuck for 22s! [ll_ost_io01_026:34574]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:54:05 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#19 stuck for 23s! [ll_ost_io01_078:22128]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:54:05 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#35 stuck for 23s! [ptlrpcd_00_15:7856]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:54:05 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#41 stuck for 23s! [ptlrpcd_00_10:7851]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:54:05 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#55 stuck for 23s! [ll_ost_io01_118:24748]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:54:09 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [ptlrpcd_00_18:7859]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:54:09 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [kiblnd_connd:7828]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:54:09 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [ll_ost_io01_020:54143]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:54:09 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#45 stuck for 22s! [ll_ost_io01_019:56155]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:54:09 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#47 stuck for 22s! [ll_ost_io01_009:50187]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:54:13 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_00_02:7843]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:54:13 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_00_16:7857]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:54:13 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [ptlrpcd_00_17:7858]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:54:13 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#37 stuck for 22s! [ptlrpcd_00_11:7852]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:54:13 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#50 stuck for 22s! [ll_ost_io01_103:24632]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:54:13 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#52 stuck for 22s! [ll_ost_io01_001:23614]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:54:13 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#54 stuck for 22s! [ll_ost_io01_107:24645]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:54:17 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpcd_00_13:7854]

       

      Message from syslogd@OSS1-A212 at Aug 21 15:54:17 ...

      kernel:NMI watchdog: BUG: soft lockup - CPU#10 stuck for 23s! [ptlrpcd_00_07:784

       

       

       

       

       

      Thanks,

      Abe

        • 1. Re: Performance Degradation in CPU v4 vs CPU v3
          Al Hill

          What are the processor model numbers?

          What motherboard and BIOS are you using?

          What operating system are you using?

           

          Doc

          • 2. Re: Performance Degradation in CPU v4 vs CPU v3
            aasraoui

            Hi Doc,

            Operating System: Centos 7.3

            cpu info:

            ndor_id  : GenuineIntel
            cpu family : 6
            model      : 79
            model name : Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz

            Motheboard and Bios info:

             

            description: Rack Mount Chassis

                product: X10DRH (085915D9)

                vendor: Supermicro

                version: 123456789

                serial: 123456789

                width: 64 bits

                capabilities: smbios-3.0 dmi-3.0 smp vsyscall32

                configuration: boot=normal chassis=rackmount family=SMC X10 sku=085915D9 uuid=00000000-0000-0000-0000-002590F9F7AE

              *-core

                   description: Motherboard

                   product: X10DRH-IT

                   vendor: Supermicro

                   physical id: 0

                   version: 1.00

                   serial: OM14BS004942

                   slot: Default string

                 *-firmware

                      description: BIOS

                      vendor: American Megatrends Inc.

                      physical id: 0

                      version: 2.0

                      date: 12/17/2015

                      size: 64KiB

                      capacity: 8128KiB

                      capabilities: pci upgrade shadowing cdboot bootselect socketedrom edd int13floppy1200 int13floppy720 int13floppy2880 int5printscreen int9keyboard int14serial int17printer acpi usb biosbootspecification uefi

             

            thanks,

            Abe

            • 3. Re: Performance Degradation in CPU v4 vs CPU v3
              Intel Corporation
              This message was posted on behalf of Intel Corporation

              Hello aasraoui,

              I know that you are expecting support here but please be aware that your best contact support is with your system manufacturer (Supermicro). I know that this is an Intel processor but your system manufacturer is more familiar with your PC features, specifications and how all the components interact with each other. At this point for you to get the proper support you can contact here:
              http://www.supermicro.com/products/motherboard/Xeon/C600/X10DRH-iT.cfm
              NOTE: This link is being offered for your convenience and should not be viewed as an endorsement by Intel of the content, products, or services offered there. 

              Regards,


              Ivan U.

              • 4. Re: Performance Degradation in CPU v4 vs CPU v3
                moyes

                Hi Abe

                 

                Were you able to find a root cause or resolution? we are experiencing the same issue with the v4 procs. Our environment has over 50 FC630's running with the v3 proc's with no issues, we added 30 FC630's running with the v4 proc's into the same environment and most of them are having CPU Softlocks issues when running under a heavy load.

                  

                Operating System: Ubuntu 14.04

                Hardware Info: Dell PowerEdge FC630

                CPU Info:

                vendor_id       : GenuineIntel

                cpu family      : 6

                model           : 79

                model name      : Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz

                stepping        : 1

                microcode       : 0xb00001f

                BIOS:

                BIOS Information

                        Vendor: Dell Inc.

                        Version: 2.4.2

                        Release Date: 01/09/2017

                        Address: 0xF0000

                        Runtime Size: 64 kB

                        ROM Size: 16384 kB

                        Characteristics:

                                ISA is supported

                                PCI is supported

                                PNP is supported

                                BIOS is upgradeable

                                BIOS shadowing is allowed

                                Boot from CD is supported

                                Selectable boot is supported

                                EDD is supported

                                Japanese floppy for Toshiba 1.2 MB is supported (int 13h)

                                5.25"/360 kB floppy services are supported (int 13h)

                                5.25"/1.2 MB floppy services are supported (int 13h)

                                3.5"/720 kB floppy services are supported (int 13h)

                                8042 keyboard services are supported (int 9h)

                                Serial services are supported (int 14h)

                                CGA/mono video services are supported (int 10h)

                                ACPI is supported

                                USB legacy is supported

                                BIOS boot specification is supported

                                Function key-initiated network boot is supported

                                Targeted content distribution is supported

                                UEFI is supported

                        BIOS Revision: 2.4