8 Replies Latest reply on Sep 4, 2011 3:49 PM by vjain27

    About the booting sequence of SCCLinux




      I'm curious about the booting sequence of SCCLinux.

      For instance, I want to know about how the sccBoot command from the MCPC loads the Linux image onto the memory, and then how it passes control to the kernel code loaded on to each core's memory space, how the bootloader code is called and executed. Also, I would like to know which part of the loaded sccLinux kernel code is executed the first time it is booted on a core.


      I am  trying to look into the source code of the sccKit, especially the sccBoot code, and the kernel source code. However, I find it hard to figure out those things.


      Is there any kind of document that I can refer to?

      Or could someone kindly tell me about these things a little?


      Thank you.

        • 1. Re: About the booting sequence of SCCLinux

          There isn't any documentation about these details other than the source code. And I would do the same thing you are doing ... reading and parsing the source. I do have access to the people who wrote the code (they also read this forum) and can try to answer some specific questions.


          Basically, I think you are interested in what happens after the SCC Linux code gets into core memory, how the core actually comes up.

          • 2. Re: About the booting sequence of SCCLinux

            Hi, Ted.


            Yes, you're right. That's what I want to know.


            First of all for now, I would like to find out about these 2 specifics :


            1. What sccBoot code actually does.

                 I'm reckoning that sccBoot's basic job is to just load sccLinux from the MCPC's disk onto the SCC core's memory, but what does sccBoot do other than this?


            2. After the sccLinux is loaded onto a core's memory, which part in the kernel code gets executed first of all?

                 And who calls that code?

                 A normal kind of bootloader doesn't seem to be taking part in SCC's booting process, and so I'm guessing that sccBoot does the bootloader's job. However, I'm having a hard time finding out how it goes from the sccBoot code to the loaded sccLinux kernel code.


            Could the people you mentioned who wrote the code answer these questions for me?

            It would be very helpful!

            • 3. Re: About the booting sequence of SCCLinux

              Hi John,


              I didn't write that code but I might be able to tell you the answers to your questions.

              So let's see. After uploading the memory image and releasing the reset, a core

              begins execution at 0xfffffff0. Normally the bios would put a jump instruction

              there. Due to the lack of bios on SCC you need to do this by yourself.


              There is a folder called reset_vector. That's the code which is loaded

              to this address.


              #define LOAD_SEG 0x9000    
                   .globl     _start

              .org 0x0000

                      mov $LOAD_SEG, %ax
                      mov %ax, %ds
                      mov %ax, %es
                      mov %ax, %fs
                      mov %ax, %gs
                      mov %ax, %ss
                      mov $0x8ffe, %sp
                      ljmp $LOAD_SEG + 0x20, $0

              .org 0x0ff0
                   jmp     _realstart


              So, what happens is: the core executes jmp _realstart. Then the code disables interrupts and reloads the segment registers. After that it

              does a long jump to the linux initializiation code (which would be setup in arch/i386/boot/).

              1 of 1 people found this helpful
              • 4. Re: About the booting sequence of SCCLinux

                Hi John,

                I also didn't write the original code, but I think I understand it well enough to answer.


                The SCC does not have either a ROM BIOS or boot loader, so it also cannot use the traditional PC bootstrap. Instead, memory is preloaded with the Linux kernel while cores are in RESET state (i.e., halted). Once the RESET line is released, each core begins to execute at the platform-defined entry point (0xFFFFFFF0 for x86, 15 bytes below the end of the 32-bit physical address space).


                sccBoot prepares the cores for this process. There are two basic variations, depending on whether you use the default "pre-merged" Linux image delivered with sccKit or a custom one. They are quite similar, the "pre-merged" Linux just allows for some optimization. sccBoot does the following:


                1. For each core: assert RESET. This prevents all further code execution until it is released, so the core does not trash its memory while it is being initialized.
                2. For each core: enable or disable L2 cache, depending on the sccKit configuration setting.
                3. For each tile: clear Message-Passing Buffer.
                4. Load memory image into target core(s); more on that below
                5. For each core: set LUTs
                6. Clear shared memory regions for SCC<->MCPC communication (host network, SCC performance meter)
                7. Tell MCPC driver about the shared memory region (so the crbnet portion knows where to put its data)
                8. For each core: release RESET.


                The memory image that you input to sccBoot describes the contents seen by one core, so before it can be written to DRAM, it needs to be decided which addresses the cores are going to use (also taking the LUTs into account).

                For this, all sccKit software uses a straight-forward convention: core's lower physical address space is "local" memory, mapped to only this core, and resides on the "nearest" DRAM controller. So each controller supplies 12 cores, or from the perspective of the controller, needs 12 copies of the per-core memory image.

                Given the per-core image, sccBoot invokes sccMerge to create a per-memory-controller image. sccMerge needs to know the total amount of memory (attached to a single controller) and the number of cores that will use this; it outputs another memory image that can then be loaded as-is onto each of the four memory controllers. For this operation, sccBoot uses the broadcast feature, which sends write requests simultaneously to all controllers (look for calls in sccBoot.cpp that use the "BCMC" flag).


                Operation is similar with the "pre-merged" image, although duplication of per-core images is not performed by the MCPC then, but by the SCC itself. In this mode, a single copy of the "pre-merged" image is written, once, to DRAM; then, each core executes the "copy-cat" image under a special LUT mapping. The copycat image copies the image into local memory, then signals the MCPC via a flag in the MPBs. Afterwards, cores can be reset again with the Linux LUT mapping and execution continues normally.


                Once the RESET lines are finally released, cores begin execution at the normal entry point 0xFFFFFFF0, just like tpresche wrote. This range contains the BIOS ROM in normal PCs; for the SCC, it simply contains the trampoline code (in reset_vector/reset_vector.S) to jump directly to the loaded kernel.


                The kernel bootstrap itself consists of two phases: real-mode and protected-mode startup. The real-mode startup code is located in arch/i386/boot/setup.S for 2.6.17. This code expects that a "boot loader" has set up a specific data structure in memory. For the SCC, the kernel sources have manually been patched to contain the right settings in the right places; like, for example, the address of the loaded (protected-mode) kernel image.


                The real-mode setup code then performs basic hardware initialization, normally via software interrupts to the ROM BIOS. As these are not present in the SCC, the calls have been patched out completely. If they are to return certain values (like 0x15/0xe820, which returns usable ranges of physical memory), appropriate defaults have been written into the data area beforehand.


                If you want to see the amount of patches involved, I recommend you download the original 2.6.17 kernel and do a diff (especially on setup.S). The startup patches make it comparatively hard to run a newer Linux kernel on the SCC, mainly because the real-mode setup has undergone major changes in later revisions. To work around this, my group has written a very simple BIOS emulator for the SCC, which allows a newer kernel (we are currently at 2.6.37) to boot without any changes to the core real-mode setup. This will be presented at the upcomming MARC symposium, provided our paper gets accepted.

                1 of 1 people found this helpful
                • 5. Re: About the booting sequence of SCCLinux

                  tpresche and JanArneSobania, thank you very much.

                  You both were a great help

                  • 6. Re: About the booting sequence of SCCLinux

                    Hi tpresche,


                    Could you please let me know how does the code in reset_vector gets placed at  the

                    at the address 0xfffffff0 ?



                    • 7. Re: About the booting sequence of SCCLinux

                      Hi vjain27,


                      you can use the tool http://marcbug.scc-dc.com/svn/repository/trunk/linuxkernel/bin2obj/ to merge all your binary

                      files to an intel object file. Furthermore, you'll need a load.map file like this one http://marcbug.scc-dc.com/svn/repository/trunk/linuxkernel/images/load.map


                      So let's say you have the reset_vector and your program, test.bin. Then your load.map has to look like this:


                      0x00090200 test.bin
                      0xfffff000 reset_vector.bin

                      Kind regards

                      • 8. Re: About the booting sequence of SCCLinux

                        Thanks a lot!