3 Replies Latest reply on Nov 30, 2011 7:45 PM by tedk

    Can SCC cores run at different speeds?

    Kitty

      Hi,

       

      I wonder if SCC cores could run at different speeds with different settings. That is, if configured/trained diffently, it might run faster/slower and finish tasks sooner or later? If so, how do I configure them to the fastest speed?

       

      So here is a short summary of what I have observed: We had a workload that run on 4 SCC cores (rck00 to rck03 on marc006), where each core ran a seperate process on one partition of the workload. Each process had some routine work every 100ms (which we can a step).

      • On Nov 22, when I run the workload, each core reported that it finished the route for each step within 20~30ms, or even faster. Overall the result seemed good.
      • We then rebooted marc006. But today when I run the same workload (same program on each core) on the same 4 cores, each core seemed run pretty slow (each step took 100+ms to 900+ms to finish). As a result, the workload didn't run well.
      • I tried to retrain the SCC cores with Tile800_Mesh1600_DDR1066, and rebooted. Still observed the slow cores.

       

      The workload I run was a collision avoidance program (to avoid collisons among moving agents or bots), and 4 cores partitioned the load of 64 agents. Even plus the communication cost of synchorinizing the positions of the agents among the cores, it does not sound like a workload that would overload a SCC core.

       

      Any comments are appreciated.

       

      Thanks.

        • 1. Re: Can SCC cores run at different speeds?
          Kitty

          It seems that it might be the Fdiv tool I used to change clock frequencies on the SCC cores.

           

          I did train the sytem with option Tile800_Mesh1600_DDR1066, then reloaded. But I probably also have run a script to set clock frequencies to be 800Mhz just to make sure clocks are running at the same frequency as the OS thinks. This is the command I used to do that:

          rccerun -nue 48 -f rc.hosts Fdiv 2

           

          So yesterday I tried to train the sytem with option Tile800_Mesh1600_DDR1066, then reloaded and not using Fdiv to set the clock frequency. This time the cores seem to be back to normal and for the same programs, same workload, each step now finishes well below 100ms (interval of each step). Good sign.

           

          Ted thought that it might be that while setting the clock frequency, Fdiv also does something with the power domain. That together, might have some impact on the speed of SCC cores.

          • 2. Re: Can SCC cores run at different speeds?
            tedk

            I looked into Fdiv.c in more detail. I don't think it's doing anything unexpected  under the covers in this case.

             

            I checked the starting frequency of tile 0x00 and looked at all the voltages. Then I changed the frequency in power domain 0 (which contains tiles 0, 1, 6, 7) to 800 MHz ... divider is 2.

             

            Tiles are as follows. Power domains are 2x2 arrays of tiles. Power domain 4 has tiles 0, 1, 6, and 7. Power domains are numbered 4, 5, and 7 along the first row and 0, 1, and 3 along the top row.

             

            18  19     20  21     22  23

            12  13     14  15     16  17

             

            6   7      8   9      10  11
            0   1      2   3      4   5

             

            $ sccDump -c 00 |grep GCB
            GCBCFG   = 0x00a8e2f0 <== this value indicates 533MHz ... see EAS:Table 4
            $ sccBmc -c set
            INFO: openBMCConnection(10.3.16.163:5010): You are participant #1
            INFO: Welcome to sccBmc 1.4.1 (build date Jun 28 2011 - 16:01:43)...
            INFO: Result of BMC command "set":
            Current settings are:

            Default FPGA bitstream: /mnt/flash4/rl_20110624_ab.bit

            Associated MAC addresses:
            - Port A 0x000000000001
            - Port B 0x000000000002
            - Port C 0x000000000003
            - Port D 0x000000000004

            Target trimming voltages:
            0 1.100 V
            1 1.100 V
            2 1.100 V <== power domain 2 = power domain 6 = whole mesh
            3 1.100 V
            4 1.100 V
            5 1.100 V
            6 1.100 V
            7 1.100 V
            $

             

            $ rccerun -nue 1 -f rc.hosts Fdiv 2
            pssh -h PSSH_HOST_FILE.20189 -t -1 -p 1 /shared/tekubasx/mpb.20189 < /dev/null
            [1] 15:54:49 [SUCCESS] rck00
            pssh -h PSSH_HOST_FILE.20189 -t -1 -P -p 1 /shared/tekubasx/Fdiv 1 0.533 00 2 < /dev/null
            rck00: UE 0, Core ID 0; size of V dom 0 is 1, size of F dom 0 is 1
            rck00: Requested fdiv: 2, actual fdiv: 2
            rck00: Clock divider for tile 0 is 2 <== divider is 2 (800 MHz)
            Clock divider for tile 1 is 2 <== divider is 2
            Clock divider for tile 2 is 3
            Clock divider for tile 3 is 3
            Clock divider for tile 4 is 3
            Clock divider for tile 5 is 3
            Clock divider for tile 6 is 2 <== divider is 2
            Clock divider for tile 7 is 2 <== divider is 2
            Clock divider for tile 8 is 3
            Clock divider for tile 9 is 3
            Clock divider for tile 10 is 3
            Clock divider for tile 11 is 3
            Clock divider for tile 12 is 3
            Clock divider for tile 13 is 3
            Clock divider for tile 14 is 3
            Clock divider for tile 15 is 3
            Clock divider for tile 16 is 3
            Clock divider for tile 17 is 3
            Clock divider for tile 18 is 3
            Clock divider for tile 19 is 3
            Clock divider for tile 20 is 3
            Clock divider for tile 21 is 3
            Clock divider for tile 22 is 3
            Clock divider for tile 23 is 3
            [1] 15:54:50 [SUCCESS] rck00
            $

             

            Note that the voltage has not changed.

            $ sccBmc -c set
            INFO: openBMCConnection(10.3.16.163:5010): You are participant #1
            INFO: Welcome to sccBmc 1.4.1 (build date Jun 28 2011 - 16:01:43)...
            INFO: Result of BMC command "set":
            Current settings are:

            Default FPGA bitstream: /mnt/flash4/rl_20110624_ab.bit

            Associated MAC addresses:
            - Port A 0x000000000001
            - Port B 0x000000000002
            - Port C 0x000000000003
            - Port D 0x000000000004

            Target trimming voltages:
            0 1.100 V
            1 1.100 V
            2 1.100 V
            3 1.100 V
            4 1.100 V
            5 1.100 V
            6 1.100 V
            7 1.100 V
            $

             

            What are the frequencies? Use sccDump -c <tile in hex> |grep GCB to dump the clock config registers and see

            GCBCFG   = 0x0070e1f0 <==  Tile 0 is 800 MHz
            GCBCFG   = 0x0070e1f0 <== Tile 1 is 800 MHz
            GCBCFG   = 0x00a8e2f0
            GCBCFG   = 0x00a8e2f0
            GCBCFG   = 0x00a8e2f0
            GCBCFG   = 0x00a8e2f0
            GCBCFG   = 0x0070e1f0 <== Tile 6 is 800 MHz
            GCBCFG   = 0x0070e1f0 <== Tile 7 is 800 MHz
            GCBCFG   = 0x00a8e2f0
            GCBCFG   = 0x00a8e2f0
            GCBCFG   = 0x00a8e2f0
            GCBCFG   = 0x00a8e2f0
            GCBCFG   = 0x00a8e2f0
            GCBCFG   = 0x00a8e2f0
            GCBCFG   = 0x00a8e2f0
            GCBCFG   = 0x00a8e2f0
            GCBCFG   = 0x00a8e2f0
            GCBCFG   = 0x00a8e2f0
            GCBCFG   = 0x00a8e2f0
            GCBCFG   = 0x00a8e2f0
            GCBCFG   = 0x00a8e2f0
            GCBCFG   = 0x00a8e2f0
            GCBCFG   = 0x00a8e2f0
            GCBCFG   = 0x00a8e2f0

            • 3. Re: Can SCC cores run at different speeds?
              tedk

              Oops ... that sccBmc -c set should really be a sccBMC -c status to read the voltages. The set just gives the target voltage. Note that status doesn't even list VCC6 because it is equal to VCC2.

              $ sccBmc -c status |grep OPVR
                OPVR VCC0: 1.0956 V
                OPVR VCC1: 1.0954 V
                OPVR VCC2: 1.0954 V
                OPVR VCC3: 1.0947 V
                OPVR VCC4: 1.0952 V
                OPVR VCC5: 1.0963 V
                OPVR VCC7: 1.0949 V
              $

               

              Message was edited by: Ted Kubaska .. typo