1 2 3 Previous Next 29 Replies Latest reply on Jul 13, 2017 8:03 PM by Intel Corporation

    Issue with setting smp_affinity on ixgbe cards

    KM29

      Hi,

      I am using a Dell PowerEdge R730 with Dual Xeon, each 22 cores, with 6 ixgbe compatible cards, on which I am running Linux with ixgbe driver version 4.4.0-k, using kernel versions both 4.7.10 and 4.9.6.
      I am loading the ixgbe modules at boot time, bringing up the interfaces and setting smp_affinity to the cards, using the set_irq_affinity script, so all the possible RxTx IRQs are distributed between all the available cores.
      The problem is that it happens, random, but quite often that the smp_affinity setting fails, and I need manually to re-run the script one or more times in order desired settings to be applied. There were also several occasions when the settings were not applied at all, and it took me several reboots to script to start working again.
      The problem appears not only randomly as occurrence, but also at random NIC controllers, so I am excluding the possibility of failed HW, since I also changed NICs.

      I added some debug messages to track the affinity setting in Linux kernel, and it turns out that most of the times when the setting fails the error that affinity setting function irq_do_set_affinity returns is EBUSY, but also sometimes it returns ENOSPC.

      More investigation on the topic showed whenever EBUSY was returned the problem could be overcome with re-running the script. But if the error returned was ENOSPC, it takes several reboots for the problem to disappear.

      In order to provide some more details on the system I am attaching two text files with the output of the modinfo of the ixgbe and lspci on the machine.

        • 1. Re: Issue with setting smp_affinity on ixgbe cards
          Intel Corporation
          This message was posted on behalf of Intel Corporation

          Hi KM29,

            What is the exact model of the ixgbe compatible cards?

          Thanks,
          wb
           

          • 2. Re: Issue with setting smp_affinity on ixgbe cards
            KM29

            Hi wb,

             

            Thanks for the swift response .

            In one of the occasion I am using Silicom's PE310G6SPI9 Six port Fiber 10Gbit 82559ES Based card. In other occasions I am using Hotlava's Tambora 80G4S-G3 4 port 10Gbit with a 2 port Intel's Ethernet Converged Network Adapter X520-SR2. As I wrote in the initial mail I rotated different configurations in order to isolate the problem to a specifc card , but though it kept appearing with all configurations.

             

            Cheers,

            Kula Nimbus

            • 3. Re: Issue with setting smp_affinity on ixgbe cards
              Intel Corporation
              This message was posted on behalf of Intel Corporation

              HI KM29,

               Thank you for the information. I will check if we have any information to share. 

              Thanks,
              wb
               

              • 4. Re: Issue with setting smp_affinity on ixgbe cards
                Intel Corporation
                This message was posted on behalf of Intel Corporation

                Hi KM29,

                  Please try use driver version 5.0.4 at 
                https://sourceforge.net/projects/e1000/files/ixgbe%20stable/

                  You can also contact RHEL support to double check.

                rgds,
                wb
                 

                • 5. Re: Issue with setting smp_affinity on ixgbe cards
                  KM29

                  Hi wb,

                   

                  Thanks for checking this once again. I just downloaded the version and built it on my kernel and got pretty much the same result. From the very first attempt my affinity settings were ignored and for almost all the NICs I needed to re-appy it couple of times in order they to take affect.

                  Here is the modinfo now:

                   

                   

                  # modinfo ixgbe

                  filename:       /lib/modules/4.9.6-mine/updates/drivers/net/ethernet/intel/ixgbe/ixgbe.ko

                  version:        5.0.4

                  license:        GPL

                  description:    Intel(R) 10GbE PCI Express Linux Network Driver

                  author:         Intel Corporation, <linux.nics@intel.com>

                  srcversion:     342373434C53E2CDAA7E0DF

                  alias:          pci:v00008086d000015E5sv*sd*bc*sc*i*

                  alias:          pci:v00008086d000015E4sv*sd*bc*sc*i*

                  alias:          pci:v00008086d000015CEsv*sd*bc*sc*i*

                  alias:          pci:v00008086d000015CCsv*sd*bc*sc*i*

                  alias:          pci:v00008086d000015CAsv*sd*bc*sc*i*

                  alias:          pci:v00008086d000015C8sv*sd*bc*sc*i*

                  alias:          pci:v00008086d000015C7sv*sd*bc*sc*i*

                  alias:          pci:v00008086d000015C6sv*sd*bc*sc*i*

                  alias:          pci:v00008086d000015C4sv*sd*bc*sc*i*

                  alias:          pci:v00008086d000015C3sv*sd*bc*sc*i*

                  alias:          pci:v00008086d000015C2sv*sd*bc*sc*i*

                  alias:          pci:v00008086d000015ADsv*sd*bc*sc*i*

                  alias:          pci:v00008086d000015ACsv*sd*bc*sc*i*

                  alias:          pci:v00008086d000015ABsv*sd*bc*sc*i*

                  alias:          pci:v00008086d000015B0sv*sd*bc*sc*i*

                  alias:          pci:v00008086d000015AAsv*sd*bc*sc*i*

                  alias:          pci:v00008086d000015D1sv*sd*bc*sc*i*

                  alias:          pci:v00008086d00001563sv*sd*bc*sc*i*

                  alias:          pci:v00008086d00001560sv*sd*bc*sc*i*

                  alias:          pci:v00008086d00001558sv*sd*bc*sc*i*

                  alias:          pci:v00008086d0000154Asv*sd*bc*sc*i*

                  alias:          pci:v00008086d00001557sv*sd*bc*sc*i*

                  alias:          pci:v00008086d0000154Fsv*sd*bc*sc*i*

                  alias:          pci:v00008086d0000154Dsv*sd*bc*sc*i*

                  alias:          pci:v00008086d00001528sv*sd*bc*sc*i*

                  alias:          pci:v00008086d000010F8sv*sd*bc*sc*i*

                  alias:          pci:v00008086d0000151Csv*sd*bc*sc*i*

                  alias:          pci:v00008086d00001529sv*sd*bc*sc*i*

                  alias:          pci:v00008086d0000152Asv*sd*bc*sc*i*

                  alias:          pci:v00008086d000010F9sv*sd*bc*sc*i*

                  alias:          pci:v00008086d00001514sv*sd*bc*sc*i*

                  alias:          pci:v00008086d00001507sv*sd*bc*sc*i*

                  alias:          pci:v00008086d000010FBsv*sd*bc*sc*i*

                  alias:          pci:v00008086d00001517sv*sd*bc*sc*i*

                  alias:          pci:v00008086d000010FCsv*sd*bc*sc*i*

                  alias:          pci:v00008086d000010F7sv*sd*bc*sc*i*

                  alias:          pci:v00008086d00001508sv*sd*bc*sc*i*

                  alias:          pci:v00008086d000010DBsv*sd*bc*sc*i*

                  alias:          pci:v00008086d000010F4sv*sd*bc*sc*i*

                  alias:          pci:v00008086d000010E1sv*sd*bc*sc*i*

                  alias:          pci:v00008086d000010F1sv*sd*bc*sc*i*

                  alias:          pci:v00008086d000010ECsv*sd*bc*sc*i*

                  alias:          pci:v00008086d000010DDsv*sd*bc*sc*i*

                  alias:          pci:v00008086d0000150Bsv*sd*bc*sc*i*

                  alias:          pci:v00008086d000010C8sv*sd*bc*sc*i*

                  alias:          pci:v00008086d000010C7sv*sd*bc*sc*i*

                  alias:          pci:v00008086d000010C6sv*sd*bc*sc*i*

                  alias:          pci:v00008086d000010B6sv*sd*bc*sc*i*

                  depends:

                  vermagic:       4.9.6-mine SMP mod_unload

                  parm:           EEE:Energy Efficient Ethernet (EEE) ,0=disabled, 1=enabled )default EEE disable (array of int)

                  parm:           InterruptType:Change Interrupt Mode (0=Legacy, 1=MSI, 2=MSI-X), default IntMode (deprecated) (array of int)

                  parm:           IntMode:Change Interrupt Mode (0=Legacy, 1=MSI, 2=MSI-X), default 2 (array of int)

                  parm:           MQ:Disable or enable Multiple Queues, default 1 (array of int)

                  parm:           DCA:Disable or enable Direct Cache Access, 0=disabled, 1=descriptor only, 2=descriptor and data (array of int)

                  parm:           RSS:Number of Receive-Side Scaling Descriptor Queues, default 0=number of cpus (array of int)

                  parm:           VMDQ:Number of Virtual Machine Device Queues: 0/1 = disable (1 queue) 2-16 enable (default=8) (array of int)

                  parm:           max_vfs:Number of Virtual Functions: 0 = disable (default), 1-63 = enable this many VFs (array of int)

                  parm:           VEPA:VEPA Bridge Mode: 0 = VEB (default), 1 = VEPA (array of int)

                  parm:           InterruptThrottleRate:Maximum interrupts per second, per vector, (0,1,956-488281), default 1 (array of int)

                  parm:           LLIPort:Low Latency Interrupt TCP Port (0-65535) (array of int)

                  parm:           LLIPush:Low Latency Interrupt on TCP Push flag (0,1) (array of int)

                  parm:           LLISize:Low Latency Interrupt on Packet Size (0-1500) (array of int)

                  parm:           LLIEType:Low Latency Interrupt Ethernet Protocol Type (array of int)

                  parm:           LLIVLANP:Low Latency Interrupt on VLAN priority threshold (array of int)

                  parm:           FdirPballoc:Flow Director packet buffer allocation level:

                              1 = 8k hash filters or 2k perfect filters

                              2 = 16k hash filters or 4k perfect filters

                              3 = 32k hash filters or 8k perfect filters (array of int)

                  parm:           AtrSampleRate:Software ATR Tx packet sample rate (array of int)

                  parm:           MDD:Malicious Driver Detection: (0,1), default 1 = on (array of int)

                  parm:           LRO:Large Receive Offload (0,1), default 0 = off (array of int)

                  parm:           allow_unsupported_sfp:Allow unsupported and untested SFP+ modules on 82599 based adapters, default 0 = Disable (array of int)

                  parm:           dmac_watchdog:DMA coalescing watchdog in microseconds (0,41-10000), default 0 = off (array of int)

                  parm:           vxlan_rx:VXLAN receive checksum offload (0,1), default 1 = Enable (array of int)

                  • 6. Re: Issue with setting smp_affinity on ixgbe cards
                    Intel Corporation
                    This message was posted on behalf of Intel Corporation

                    Hi KM29,

                     Have you contacted RHEL support ?

                    Thanks,
                    wb
                     

                    • 7. Re: Issue with setting smp_affinity on ixgbe cards
                      KM29

                      Hi wb,

                       

                      I have posted same question on the sourceforge Intel Ethernet Drivers and Utilities / Bugs / #559 Issue with setting smp_affinity on ixgbe cards

                      Since I am not using Red Hat, I have no argument to post at RHEL support site, so I don't see why I should do it, and I did not post anything there.

                       

                      Best Regards,

                      Kula Nimbus

                      • 8. Re: Issue with setting smp_affinity on ixgbe cards
                        Intel Corporation
                        This message was posted on behalf of Intel Corporation

                        Hi KM29,

                        Thank you for the update.  Have you tried turning off the irqbalance daemon? 

                        Thanks,
                        wb
                         

                        • 9. Re: Issue with setting smp_affinity on ixgbe cards
                          KM29

                          Hi wb,

                           

                          Since my target is manual handling of affinity, I don't even have the irqbalance installed.

                          So I guess it removes it out of the equation.

                           

                          Best Regards,

                          Kula Nimbus

                          • 10. Re: Issue with setting smp_affinity on ixgbe cards
                            Intel Corporation
                            This message was posted on behalf of Intel Corporation

                            Hi KM29,

                             

                               Please continue to monitor the sourceforge website for updates on upcoming fixes.

                            Thanks,
                            wb

                            • 11. Re: Issue with setting smp_affinity on ixgbe cards
                              KM29

                              Hi wb,

                               

                              Thanks for replying. Current stage at sourceforge, is that the issue was closed with the explanation that an internal bug was opened, but I did not get any details how to track the "internal bug".

                               

                              Best Regards,

                              Kula Nimbus

                              • 12. Re: Issue with setting smp_affinity on ixgbe cards
                                Intel Corporation
                                This message was posted on behalf of Intel Corporation

                                Hi KM29,

                                 Thanks for the update.

                                Rgds,
                                wb
                                 

                                • 13. Re: Issue with setting smp_affinity on ixgbe cards
                                  KM29

                                  Hi wb,

                                   

                                  I have small update on this issue. Since the issue on sourceforge got closed, I will update only this thread for now.

                                  I think it may be interesting for you or give a major clue. I was able to reproduce the same issue also with an i40e controller (Intel Corporation Ethernet Controller X710) on the same platform I posted originally.

                                   

                                  Best Regards,

                                  Kula Nimbus

                                  • 14. Re: Issue with setting smp_affinity on ixgbe cards
                                    Intel Corporation
                                    This message was posted on behalf of Intel Corporation

                                    Hi KM29,

                                     Thank you for the update. Just to double check what is the driver you used here? 

                                    rgds,
                                    wb
                                     

                                    1 2 3 Previous Next