6 Replies Latest reply on Jun 24, 2011 9:16 AM by tedk

    Booting custom kernels on the SCC

    viveks

      Hello,

       

      I am trying to boot custom kernels on the SCC cores. I was attempting to boot different cores with different images and having some problems.

       

      I have two images - kernel1.obj and kernel2.obj. These are the steps that I carried out:

       

      1) sccBoot -l kernel1.obj 0..47  - this booted the kernel on all 48 cores and they were all ping-reachable and working.

       

      2) sccBoot -l kernel2.obj 1 - this should boot kernel2 on just core 1. What happened was that after this core 1 was reachable and running, while all the other cores went down - neither was I able to ssh to the cores nor ping them.

       

      Am I missing out something or is this not something that the SCC supports?

       

      Thanks in advance.

       

      Vivek

        • 1. Re: Booting custom kernels on the SCC
          andrea.bartolini

          Just one quick question is kernel2 dfferent from kernel1? Maybe there are errrors in kernel2 image.

           

          To debug it I suggest you a trick to see the log output of the kernel at boot time. This will allow you to debug the kernel, i there are errors.

          You need simply to check the address in System.map of the __log_buf variable. To do that you can uncomment in the makeRCKS.sh in rckos/bin folder the line 61 ...cat .....System.map | grep __log_buf .

          This variable contains the starting address of the circular buffer tha contains the kernel log. Is the one you flush by issuing "dmesg" command.

           

          Now you can check in the LuT configuration the mapping of this address in the dram address space. You need to open the file in /tmp/sccKit_username/obj/lut_init.vi.

           

          Be carefull that the address you find in the System.map is not the true one I discover you need to subtrack C0000000 to the address in the System.map.

          example in my case the address was :

          c1eb4260 b __log_buf

          c1ed4260 b logged_chars

           

          thus the address used in the kernel is c1eb4260 - c0000000 = 01eb4260

           

          Now in the lut_init.vi  you can recognize that for the core 0 the address 0x01 is mapped in 0x001 memory space whereas for the core 1 it is mapped to 0x02a. Both of them in the 00 memory controller.

           

          the buffer lenght is instead obtainable by the difference in address in between __log_buf and the next one, in my case 0x2000 = 131072

           

          said that you can dump the log buffer using the sccDump command

           

          sccDump -f log.txt -d 00 01eb2260 131072

           

          the -d 00 is the memory controller at which the ddr is connected.

           

          Hopefully this will help you in debug your modification.

          Best

          Andrea

          1 of 1 people found this helpful
          • 2. Re: Booting custom kernels on the SCC
            viveks

            Yes, the kernels are different. The thing is that if I boot kernel1 and kernel2 on all the cores (as usual) they seem to work fine.

             

            I will try your method with the experiment I was trying and see what happens on the kernel that goes down. Thank you for the debug method.

             

            Regards,

            vivek

            • 3. Re: Booting custom kernels on the SCC
              andrea.bartolini

              Ok if both in all the core work fine you can try to boot it in a single core by using sccGui. If  it is working, probably sccBoot needs to be used in a different way. Unfortunately I do not have experience in using it by command line.

               

              Best

              Andrea

              • 4. Re: Booting custom kernels on the SCC
                tedk

                Vivek,

                Have you resolved this issue?

                 

                You might consider using sccBoot -g and sccMerge to control what obj goes on what core. For example, if you use the default linux and put the linux object somewhere other than in /opt/sccKit/current/resources and boot as sccBoot -l linux.obj 0..47, you'll see some files under /tmp/sccKit_yourname on the MCPC. You'll see a linux.obj and a linux.mt there.

                 

                I tried just copying those files into another directory and trying out our  sccMerge as folllows.

                $ sccMerge -m 8 -n 12 -noimage -force ./linux.mt

                 

                Successfully created memory images...

                Successfully created LUT configuration...

                $


                 

                This also created an directory called obj which contained the following.

                $ ls obj

                arguments.txt     memory_07.32.obj  memory_11.asm     memory_1c.32.obj  memory_26.asm

                lut_init.dat      memory_07.asm     memory_12.32.obj  memory_1c.asm     memory_27.32.obj

                lut_init.vi       memory_08.32.obj  memory_12.asm     memory_1d.32.obj  memory_27.asm

                mch_0_0.32.obj    memory_08.asm     memory_13.32.obj  memory_1d.asm     memory_28.32.obj

                mch_0_2.32.obj    memory_09.32.obj  memory_13.asm     memory_1e.32.obj  memory_28.asm

                mch_5_0.32.obj    memory_09.asm     memory_14.32.obj  memory_1e.asm     memory_29.32.obj

                mch_5_2.32.obj    memory_0a.32.obj  memory_14.asm     memory_1f.32.obj  memory_29.asm

                memory_00.32.obj  memory_0a.asm     memory_15.32.obj  memory_1f.asm     memory_2a.32.obj

                memory_00.asm     memory_0b.32.obj  memory_15.asm     memory_20.32.obj  memory_2a.asm

                memory_01.32.obj  memory_0b.asm     memory_16.32.obj  memory_20.asm     memory_2b.32.obj

                memory_01.asm     memory_0c.32.obj  memory_16.asm     memory_21.32.obj  memory_2b.asm

                memory_02.32.obj  memory_0c.asm     memory_17.32.obj  memory_21.asm     memory_2c.32.obj

                memory_02.asm     memory_0d.32.obj  memory_17.asm     memory_22.32.obj  memory_2c.asm

                memory_03.32.obj  memory_0d.asm     memory_18.32.obj  memory_22.asm     memory_2d.32.obj

                memory_03.asm     memory_0e.32.obj  memory_18.asm     memory_23.32.obj  memory_2d.asm

                memory_04.32.obj  memory_0e.asm     memory_19.32.obj  memory_23.asm     memory_2e.32.obj

                memory_04.asm     memory_0f.32.obj  memory_19.asm     memory_24.32.obj  memory_2e.asm

                memory_05.32.obj  memory_0f.asm     memory_1a.32.obj  memory_24.asm     memory_2f.32.obj

                memory_05.asm     memory_10.32.obj  memory_1a.asm     memory_25.32.obj  memory_2f.asm

                memory_06.32.obj  memory_10.asm     memory_1b.32.obj  memory_25.asm     testcase.txt

                memory_06.asm     memory_11.32.obj  memory_1b.asm     memory_26.32.obj

                $


                The memory_* obj are links to the linux.obj. But the mch*.obj are the merged files, one for each memory controller. They haven't been loaded yet. To load them first reset the cores, then boot, then release the reset.

                 

                $ sccReset -g

                INFO: Welcome to sccReset 1.4.0 (build date Mar 21 2011 - 18:40:25)...

                INFO: Applying global software reset to SCC (cores & CRB registers)...

                INFO: (Re-)configuring GRB registers...

                 

                $ ls

                linux.mt  linux.obj  obj

                 

                $ sccBoot -g ./obj

                INFO: Welcome to sccBoot 1.4.0 (build date Mar 21 2011 - 18:39:01)...

                INFO: Pulling reset of all cores...

                INFO: Preloading Memory with object file...

                INFO: Found object for MC x=0, y=0: "./obj/mch_0_0.32.obj"...

                INFO: writeMemFromOBJ(...): Configuration of memory done!

                INFO: Successfully (re-)loaded object file "./obj/mch_0_0.32.obj"...

                INFO: Found object for MC x=5, y=0: "./obj/mch_5_0.32.obj"...

                INFO: writeMemFromOBJ(...): Configuration of memory done!

                INFO: Successfully (re-)loaded object file "./obj/mch_5_0.32.obj"...

                INFO: Found object for MC x=0, y=2: "./obj/mch_0_2.32.obj"...

                INFO: writeMemFromOBJ(...): Configuration of memory done!

                INFO: Successfully (re-)loaded object file "./obj/mch_0_2.32.obj"...

                INFO: Found object for MC x=5, y=2: "./obj/mch_5_2.32.obj"...

                INFO: writeMemFromOBJ(...): Configuration of memory done!

                INFO: Successfully (re-)loaded object file "./obj/mch_5_2.32.obj"...

                INFO: Preloading LUTs...

                INFO: Configuring LUTs with content of file "./obj/lut_init.dat"...

                INFO: -> Configuration of LUTs done!

                INFO: Image is now pre-loaded. Release resets to start individual cores...

                $

                 

                $ sccReset -r 0..47

                INFO: Welcome to sccReset 1.4.0 (build date Mar 21 2011 - 18:40:25)...

                INFO: Resets have been released: All cores!

                 

                $ ping rck00

                PING rck00.in.rck.net (192.168.26.1) 56(84) bytes of data.

                64 bytes from rck00.ex.rck.net (192.168.26.1): icmp_seq=1 ttl=64 time=0.308 ms

                64 bytes from rck00.ex.rck.net (192.168.26.1): icmp_seq=2 ttl=64 time=0.100 ms

                64 bytes from rck00.ex.rck.net (192.168.26.1): icmp_seq=3 ttl=64 time=0.140 ms

                ^C

                --- rck00.in.rck.net ping statistics ---

                3 packets transmitted, 3 received, 0% packet loss, time 1998ms

                rtt min/avg/max/mdev = 0.100/0.182/0.308/0.091 ms

                 

                Look inside linux.mt to see what linux goes on which cores. For example,

                 

                # pid mch-route mch-dest-id mch-offset-base testcase
                0x00 0x00 6 0x00 /tmp/sccKit_tekubas/linux.obj
                0x01 0x00 6 0x01 /tmp/sccKit_tekubas/linux.obj
                0x02 0x00 6 0x02 /tmp/sccKit_tekubas/linux.obj
                0x03 0x00 6 0x03 /tmp/sccKit_tekubas/linux.obj
                0x04 0x00 6 0x04 /tmp/sccKit_tekubas/linux.obj
                0x05 0x00 6 0x05 /tmp/sccKit_tekubas/linux.obj
                0x06 0x05 4 0x00 /tmp/sccKit_tekubas/linux.obj
                0x07 0x05 4 0x01 /tmp/sccKit_tekubas/linux.obj
                0x08 0x05 4 0x02 /tmp/sccKit_tekubas/linux.obj
                0x09 0x05 4 0x03 /tmp/sccKit_tekubas/linux.obj
                0x0a 0x05 4 0x04 /tmp/sccKit_tekubas/linux.obj
                0x0b 0x05 4 0x05 /tmp/sccKit_tekubas/linux.obj
                0x0c 0x00 6 0x06 /tmp/sccKit_tekubas/linux.obj
                0x0d 0x00 6 0x07 /tmp/sccKit_tekubas/linux.obj
                0x0e 0x00 6 0x08 /tmp/sccKit_tekubas/linux.obj

                Note that the memory controllers are at 0x00, 0x05, 0x20, and 0x25. The first column is the core id; the second column identifies the memory controller, the third column is the router port (6 is west; 4 is east; EAS Table 11). The fourth column is the offset into the merged file. The first six are loaded through the memory controller at 0x00. Cores 6 and 7 are loaded from the memory controller on the right. But there are 12 cores for each memory controller. Cores c and d which are right above cores 0 and 1 are also loaded through memory controller 0x00. And you can see, the offse for core 0x0c is sequential after that for core core 0x05.

                 

                So by modifying linux.mt before you make the obj directory, you can control what linux obj goes to what core through what memory controller.

                • 5. Re: Booting custom kernels on the SCC
                  viveks

                  Hello Ted,

                   

                  Thanks for that detailed set of steps. I got stuck with some other experiments and haven't looked into the booting problem. I shall try out this set of steps and update the results I see here.

                  • 6. Re: Booting custom kernels on the SCC
                    tedk

                    Currently you cannot use sccBoot -l to load different versions  of Linux on different cores. This issue is being discussed in Bug 254

                    http://marcbug.scc-dc.com/bugzilla3/show_bug.cgi?id=254