14 Replies Latest reply on Sep 2, 2016 9:52 AM by allan_intel

    What conditions are necessary for Turbo Boost to be sustainable?

    UncleJoe

      Hi, I'm using dual 6 core Intel Xeon 5670 CPUs for some heavy duty, high stakes, image processing. Intel Turbo Boost is throwing a wrench into our benchmarking. For benchmarking, we're considering to turn it off to reduce to varriance, but for production we still want to use it because performance is very important.

       

      The main question that we need to know is under what circumstances is turbo mode sustainable? I see from wikipedia the xeon 5670 has 2/2/2/2/3/3 Turbo modes. I do get the expected 2.93 GhZ + 2 * 133 MhZ when all 6 cores are loaded and it seems to sustain for 3 minutes. But to do a fair comparisons, I would need to know with 99% certainty that this can be sustained. I also benchmark against other processors, so just knowing sustainability for 1 CPU isn't sufficient.

       

      Please Intel, give us some the conditions under which turbo mode will be sustained. It doesn't have to be exact (i.e. disclose your design), but be correct 95% of the time.

        • 1. Re: What conditions are necessary for Turbo Boost to be sustainable?
          Doc_SilverCreek

          Turbo Boost is not over clocking.

           

          Turbo Boost Technology opportunistically allows the processor to run faster than the marked frequency if the part is operating below power, temperature and current limits.

           

          Turbo Boost operation:

          – Operates under OS control – only entered when OS requests

          higher performance state (P0)

          – Turbo Boost availability is independent of the number of active

          cores

          – Max Turbo Boost frequency is dependent on the number of active

          cores and varies by processor

          – Amount of time the system spends in Turbo Boost will depend on

          workload, operating environment, and platform design

           

          If you are providing a constant work load in your benchmarking, current and power (which are basically the same) are fixed.

          This leaves temperature as the determining factor. As the processor heats (i.e Likely your 3 minutes of sustained operation) it drops out of turbo boost mode.

          If this is the cause of the drop out, you should be able to see it by monitoring the CPU temperature using a tools such as CPUz. http://www.cpuid.com/softwares/cpu-.html 

           

          In this case a better heat sink, or chassis cooling solution would give you an extended turbo boost range.

           

           

          If your bench mark varies work load (which is more typical in the real world), either total or amount across the cores, turbo will kick in and out as required as any of the 3 contributors reach maximums. Since in this case, core usage, current draw / power as well as temperature can all be varying, it leaving you with a multiple variable problem.

           

          Sustaining turbo boost:   Use less than all the cores.

          Not a good solution for most application since what you loose in core processing power is likely more that what you gain in turbo boost unless your application is just not capable of utilizing all cores.

          If you only run 2 cores on a 5670, the processor should stay at 3.33ghz.

          If you only run 5 cores (depending on the Imon calabration. I have seen at least one systems that started dropping out of turbo ~ 90% CPU utilization on 5 the 5th core) the CPU should stay at 3.20Ghz

           

          Extending Turbo Boost mode:   Improve CPU cooling

          Better heat sinks or chassis ventilation.

           

          Improving Imon curve:

          The Imon curve is the input source to the processor and is a hardware level calibration  that tells the processor how much current it is drawing. 

          This adjustment can be touchy to correctly configure which can result in different mother boards responding differently to turbo boost with the same processors as some vendors under report Imon resulting is longer turbo boost at the risk of CPU damage or over reporting which drops the system out of turbo boost mode sooner than necessary.

           

          Processor:

          Not all processors are created equal. Do to varations in CPU batches, some processor will run hotter or draw more power at the same work load than other.

          The difference is negiable under most conditions, but Turbo Boost bench marking is one area where a low batch TDP processor will generate beter results.

          • 2. Re: What conditions are necessary for Turbo Boost to be sustainable?

            That's some useful information Doc SilverCreek, though it presents a more complicated picture than I expected. Thanks.

             

            My work load can be assumed to be fixed because the computer will be scanning images 24/7 and is the bottleneck. Then, I think power and current should be a function of temperature and manufacturing variation. I also assume power and current are the same and Intel only uses current to decide Turbo speed.

             

            Without specific numbers for the temperature and power, it's no good. We get new Intel CPUs almost every generation and certainly don't want to have to test dozens of them to determine if they all can sustain a certain Turbo frequency.

            • 3. Re: What conditions are necessary for Turbo Boost to be sustainable?
              Doc_SilverCreek

              Turbo Boost is a cool feature ( and very miss-understood)

               

              Max power is determined by the processor rating. In the case of the 5670 it is 95w. (the manufacturing variations comes in that 100% CPU should draw 95W. In some processors it will draw less so turbo can boost more.)

              Processor voltage is set by the board VID so can be assumed to be fixed. (It's not, but that is a completely different set of safety features.)

              Processor current is "measured" by a feed back circuit on the mother board called Imon.

              Temp is mostly a function of cooling since it not how much heat that the CPU generates that is it issue, it is how much that gets left in the core.

              Tcase max (81.3) (or more accurately Tcore which is normally ~ 10deg higher) is the limit here.

               

              Increasing the efficiency of the heat sink solution should increase the time a system stays in Turbo Boost, ( I have to try this one of these days. I have some H20 cooler heat sinks someplace....)

               

              From a bench mark perspective, if you disable the turbo boost, you have a "no worst than" bench mark, with the turbo boost , it should get better,

               

              If you need a guaranteed the faster processor frequency, the only way is to go to one of the faster processors. (and then you pick up an even higher Turbo Boost.)

               

              Your Intel rep might be able to help you with other options also.

               

              maxturbo.bmp

              • 4. Re: What conditions are necessary for Turbo Boost to be sustainable?
                parsec

                "Turbo Boost is not over clocking."

                 

                An interesting comment Doc, not that I'm implying it's wrong, but actually brings up an interesting point: What is over clocking?

                 

                I'm not really going there, but with the Intel CPUs I have, Turbo Boost is accomplished (it seems) by increasing the Multiplier over it's usual maximum value by one, ie, from 22 to 23.

                 

                In the world of Over Clocking, increasing the CPU Multiplier is a standard method of "over clocking", while changing the BCLK frequency is another.  Are you not calling Turbo Boost over clocking because it is not a permanent change, being opportunistic, or am I just nit-picking the semantics of your statement?

                 

                Regarding this threads topic, improved cooling is usually not difficult to accomplish, at least in the desktop environment.  I imagine you would be seeking demonstrable and scientifically rigorous verification regarding the additional cooling efficacy of any "solution" you would consider, which you won't generally find in the consumer/retail area.  High performance CPU cooling would be much easier to implement than cherry picking CPUs for performance.

                • 5. Re: What conditions are necessary for Turbo Boost to be sustainable?
                  Doc_SilverCreek

                  Ok, I had to resort to a picture. This is just a depiction. None of the numbers or graph are calculated out and are probably not even too close, but it serves to give you the idea.

                   

                  Turbo Boost mode gets you additional performance (under some conditions) by raising the clock from a base speed in predetermined steps.to keep the processor within max power (and temp) specifications. (Blue Lines)

                   

                  Over clocking raises the processor base speed to get some additional performance which will also alter the slope of the Turbo Boost line unfavorably since the max power is still fixed. (Pink / yellow lines )

                   

                  In my picture, if you only ran a CPU application that used 5 or less cores, Turbo Boost would like get you better (and safer) performance gain over overclocking. Now if you were able to tweak the overclocking up to 3,60 or 3,8 without frying the processors, you just blew the doors off the turbo boost.

                   

                   

                  Or some rough numbers i am making up I.E.

                  If a 6 core processor that uses 95 watts at 100% loading @ 3.333ghz. 95 Watts / 6 cores  = 15 Watts per core.

                  You start running a application that only uses 4 cores.4 cores at 100% @3.33ghz = ~ 60Watts leaving you 35 watts worth of head room going to waste.

                  Turbo boost see this and knows based on the processors boost steps that it can clock up 2 multipliers to 3.60Ghz at 4 cores to get to 95Watts (~ 24W per core) still keeping everything in spec.

                   

                  If I were to over clock this CPU to 3.6gHz and was using all 6 cores, my CPU would be pulling ~ 144W (6cores * 24watts assumed above). Since the processor is only rated at 95 watts, I generate a lot of heat (which hopefully my "parsec" heatsink takes care of) or I create a lump of charcoal out of what use to be my CPU.except that in this case, Thermal Throttling feature built into the processor would likely kick in to throttle the processor when it gets too hot and prevent any damage.

                   

                  Clear as mud yet? Same thing only different. turbo boost.bmp

                  • 6. Re: What conditions are necessary for Turbo Boost to be sustainable?
                    UncleJoe

                    I see you're an Intel engineer, Doc_SilverCreek. That makes me more confident about your claims. But I'm still not convinced if power < 95W is a *sufficient* condition for Turbo Boost. Clear it's necesasry, but TDP is a worst case figure, so I wouldn't be surprised if Turbo gets disabled before. Can you verify if that's the case?

                     

                    Also, your chart presents a worst case scenario. Even when all 6 cores are loaded, + 2(133) MhZ   Turbo Boost is possible as I have verfied. The only thing in doubt is how much the temperature affects the leakage power.

                    • 7. Re: What conditions are necessary for Turbo Boost to be sustainable?
                      Doc_SilverCreek

                      Uncle Joe,

                       

                      Keep up the skepticism!

                       

                      (and I don't care where anyone works, nobodys perfect. I added the job tag line because Intel said I had to for some legal reason. I would rather be one opinion among many than a sole source. I recommend using good judgment and comparison answers.  Only the skeptics will survive!      Maybe)

                       

                       

                          It is more complicated than just wattage TDP but that is the easiest way to summarize it

                      Three factors determine if a processor can use Turbo Boost mode. Temperature, Power and Current

                      (and Mr. Ohm teaches us that Power (wattage) = Voltage X Current so I assume since we already have Imon current as one variable which relates to workload, the power is really voltage and not exceeding the max.wattage).

                       

                      My little picture greatly over simplifies the variables to give a comparison between Overclocking and Turbo Boost. It is not representative of any particular CPU (Have you ever tried to draw a graph with 5 variables? ) Plus each sku (x56xx) of processor will be different.

                       

                      You can run a quick check to see where turbo boost kicks in out by running a CPU stress tool which can be configured to target single cores and incrementing the number of workers and the number of threads while setting the CPU affinity in the worker window to focus all the stress on the target cores.  (Intel has a developers tool for this, but I can't find it. I believe it is at the same link that is in document 1 below) Prime95 may also work.

                       

                      Monitor the CPU speed with a tool like CPUz http://www.cpuid.com/softwares/cpu-z.html

                       

                      and keep a eye on the cpu thermals with a tool such as CPUID HWmonitor http://www.cpuid.com/softwares/hwmonitor.html

                      Remember the core temperature can be a little higher than Tcase temp.

                       

                      Windows interferes with the test a little since it need some CPU time also and you can't specify which CPU windows will use, but it does give a good comparison when any particulate CPU might drop out of TB mode. Only testing stress on one package at a time lets windows use the second CPU mostly. 

                       

                       

                      This gets you down to variables (workload (power (current* voltage) and temperature).that are easily monitored and can be varied to observe their impacts to TB mode.

                       

                       

                       

                       

                      These links go in to Turbo boost much deeper, especially the second for the VRM design guide under Imon.Pg 38-39

                      (After you read these, you will have a better understanding of my inability to comunicate this as clearly as I would like. It igets deep and can be confusing as heck)

                       

                      http://download.intel.com/support/motherboards/server/sb/white_paper_turbo_boost_on_low_power_processor.pdf

                       

                      http://www.intel.com/assets/PDF/designguide/321736.pdf

                       

                      http://www.intel.com/technology/turboboost/index.htm  (oops, looks like this link has changed and is now on the new Turbo Boost 2 in Sandy Bridge. Still good reading)

                       

                      http://www.intel.com/Assets/PDF/manual/248966.pdf?wapkw=(turbo+boost+optimization)

                      • 8. Re: What conditions are necessary for Turbo Boost to be sustainable?

                        Actually, skepticism isn't good for engineers because we want to improve things. Skepticism is good for science to ensure a model fits reality.

                         

                        Strangely, an engineer's faith is the same as a scientist's skepticism in the sense they are both trying to create something better.

                         

                         

                        Anyways, thanks for the documentation. It looks like it will tell everything, but I will still need to verify myself. The server is running Linux, but I can still use Prime 95 to check things. I also need to figure out how to read all those sensors (the server has an IPMI interface) available.

                        • 9. Re: What conditions are necessary for Turbo Boost to be sustainable?
                          Doc_SilverCreek

                          I tend to lean towords window, but here is a link to several linux tools http://www.overclock.net/linux-unix/248684-linux-benchmarks.html

                          which will accomplish the same.IPMItool can also read the therimal sensors. For the CPU stress you need to be able to set an affinity to the indiviual cores and then increment the stress on the remaining core. (ie core 1 =100% core 2 =100% core 3 =100% core 4 = 100% core 5=10% core 6 = idle.) to see each step of the turbo activity. If the load is distrubuted across all the cores, it will drop in and out of turbo (hundreds of times a second) giving you some very comfusing data. (but more typical of what happens in the real world)

                           

                          Skeptic or Realest? I have spent too many years as a debug engineer were each piece of data needs to be evaluated to see if it is reliable and fits into the other supporting data. Too many times data lies . (or liars give me data?)

                           

                          Good Luck

                           

                          Doc

                          .

                          • 10. Re: What conditions are necessary for Turbo Boost to be sustainable?
                            UncleJoe

                            OK, I have verified Turbo Boost is sustainable, even under worst case scenarios:

                             

                            setup:

                             

                            Supermicro X8DTT    (2U twin chassis)

                            Intel Xeon 5670      x 2

                             

                             

                            ---------------------------------------------after 30 minutes of prime 95 torture test for Linux-------------------------------------------

                            Cpu speed from cpuinfo 2933.00Mhz

                            True Frequency (without accounting Turbo) 2933 MHz

                             

                            Socket [0] - [physical cores=6, logical cores=12, max online cores ever=6]

                              CPU Multiplier 22x || Bus clock frequency (BCLK) 133.32 MHz

                              TURBO ENABLED on 6 Cores, Hyper Threading ON

                              True Frequency 3066.32 MHz (133.32 x [23])

                              Max TURBO Multiplier (if Enabled) with 1/2/3/4/5/6 cores is  25x/25x/24x/24x/24x/24x

                              Current Frequency 3153.45 MHz (Max of below)

                                    Core [core-id]  :Actual Freq (Mult.)      C0%   Halt(C1)%  C3 %   C6 %  Temp(in *C)

                                    Core 1 [0]:       3153.45 (23.65x)      99.9       0       0       0    62

                                    Core 2 [1]:       3153.45 (23.65x)      99.9       0       0       0    58

                                    Core 3 [2]:       3153.45 (23.65x)      99.9       0       0       0    56

                                    Core 4 [3]:       3153.44 (23.65x)      99.9       0       0       0    57

                                    Core 5 [4]:       3153.44 (23.65x)      99.9       0       0       0    60

                                    Core 6 [5]:       3153.44 (23.65x)      99.9       0       0       0    59

                            30

                            Socket [1] - [physical cores=6, logical cores=12, max online cores ever=6]

                              CPU Multiplier 22x || Bus clock frequency (BCLK) 133.32 MHz

                              TURBO ENABLED on 6 Cores, Hyper Threading ON

                              True Frequency 3066.32 MHz (133.32 x [23])

                              Max TURBO Multiplier (if Enabled) with 1/2/3/4/5/6 cores is  25x/25x/24x/24x/24x/24x

                              Current Frequency 3131.57 MHz (Max of below)

                                    Core [core-id]  :Actual Freq (Mult.)      C0%   Halt(C1)%  C3 %   C6 %  Temp(in *C)

                                    Core 1 [6]:       3131.54 (23.49x)      99.9       0       0       0    77

                                    Core 2 [7]:       3131.57 (23.49x)      99.9       0       0       0    80

                                    Core 3 [8]:       3131.56 (23.49x)      99.9       0       0       0    72

                                    Core 4 [9]:       3131.56 (23.49x)      99.9       0       0       0    76

                                    Core 5 [10]:      3131.56 (23.49x)      99.9       0       0       0    80

                                    Core 6 [11]:      3131.55 (23.49x)      99.9       0       0       0    79

                             

                            C0 = Processor running without halting

                            C1 = Processor running with halts (States >C0 are power saver)

                            C3 = Cores running with PLL turned off and core cache turned off

                            C6 = Everything in C3 + core state saved to last level cache

                              Above values in table are in percentage over the last 1 sec

                            [core-id] refers to core-id number in /proc/cpuinfo

                             

                             

                            Although it's not exactly 2.93 GhZ  + 2 * 0.166 GhZ = 3.2 GhZ, it's close enough. The high temperatures on the 2nd CPU do seem to be a concern. Before installing an air shroud, the temperatures were even higher.

                            • 11. Re: What conditions are necessary for Turbo Boost to be sustainable?
                              parsec

                              UncleJoe,  Nice work and data to support it.  I don't understand the fractional multipliers, I've never seen that.  I've always thought that small variations in CPU speed when beyond the standard, non-Turbo CPU speed were due to small variations in BCLK frequency.  The hardware monitoring tools I've used will show a drop in BCLK of one or two MHz, with fractional values (ie, 132.3 MHz) seen as well.  The product of those BCLK frequencies and the (IMO) integer multipliers, result in non-exact final CPU clock frequencies, meaning frequencies not in perfect 133MHz steps.  I've seen minor fluctuations in BCLK on both first and second generation i7 CPU mother boards, but those are desktop products, not server boards.  Are fractional multiplier values common on server boards?

                               

                              Readers of this thread should note that Turbo Boost with first generation i7 CPUs (i7-900 series and the Xeon 5000 series) and second generation i7 CPUs (i7-2000 series and Xeon 6000 series) is different, second generation CPUs use Turbo Boost 2.0, which allows higher multipliers but has differing constraints on the time and power dissipation limits.  Settings of those limits may vary between mother board manufacture and products.

                               

                              Obviously your second CPU is not being cooled as well as the first, or does it have more memory used with it, which may add additional stress to the IMC.

                              • 12. Re: What conditions are necessary for Turbo Boost to be sustainable?
                                UncleJoe

                                "I don't understand the fractional multipliers"

                                 

                                I don't think they're fractional multipliers either, or at least they can go that far from the nearest integer. Actually, after quickly studying the i7z code, it's clear those multipliers being reported are derived, not measured directly:

                                 

                                 

                                _FREQ[ii] = THRESHOLD_BETWEEN_0_6000(estimated_mhz * ((long double) CPU_CLK_UNHALTED_CORE / (long double) CPU_CLK_UNHALTED_REF));


                                _MULT[ii] = _FREQ[ii] / BLCK;             // BLCK is a fixed value computed only once at program startup

                                 

                                In that case, it wouldn't be surprising for the fractional values, especially if the base clock drifts as you claim.

                                 

                                 

                                "Obviously your second CPU is not being cooled as well as the first, or does it have more memory used with it"

                                 

                                I don't really know why the 2nd CPU is that much hotter - I'm working on software optimization not the hardware. My guess is because it's deeper inside the chassis, where more heat can build up.


                                • 13. Re: What conditions are necessary for Turbo Boost to be sustainable?
                                  umairshabbir

                                  will some body will also help me?

                                  my turbo boost works fine on ac when pluged in but did not work on battery what should  do?

                                  i7 3635QM

                                  samsung 880Z