About the booting sequence of SCCLinux

There isn't any documentation about these details other than the source code. And I would do the same thing you are doing ... reading and parsing the source. I do have access to the people who wrote the code (they also read this forum) and can try to answer some specific questions.

Basically, I think you are interested in what happens after the SCC Linux code gets into core memory, how the core actually comes up.

Hi John,

I didn't write that code but I might be able to tell you the answers to your questions.

So let's see. After uploading the memory image and releasing the reset, a core

begins execution at 0xfffffff0. Normally the bios would put a jump instruction

there. Due to the lack of bios on SCC you need to do this by yourself.

There is a folder called reset_vector. That's the code which is loaded

to this address.

#define LOAD_SEG 0x9000    
.code16     
.text
     .globl     _start

.org 0x0000

     
_realstart:
     cli
        cld
        mov $LOAD_SEG, %ax
        mov %ax, %ds
        mov %ax, %es
        mov %ax, %fs
        mov %ax, %gs
        mov %ax, %ss
        mov $0x8ffe, %sp
        ljmp $LOAD_SEG + 0x20, $0

.org 0x0ff0
_start:
     jmp     _realstart

So, what happens is: the core executes jmp _realstart. Then the code disables interrupts and reloads the segment registers. After that it

does a long jump to the linux initializiation code (which would be setup in arch/i386/boot/).

Hi John,

I also didn't write the original code, but I think I understand it well enough to answer.

The SCC does not have either a ROM BIOS or boot loader, so it also cannot use the traditional PC bootstrap. Instead, memory is preloaded with the Linux kernel while cores are in RESET state (i.e., halted). Once the RESET line is released, each core begins to execute at the platform-defined entry point (0xFFFFFFF0 for x86, 15 bytes below the end of the 32-bit physical address space).

sccBoot prepares the cores for this process. There are two basic variations, depending on whether you use the default "pre-merged" Linux image delivered with sccKit or a custom one. They are quite similar, the "pre-merged" Linux just allows for some optimization. sccBoot does the following:

1. For each core: assert RESET. This prevents all further code execution until it is released, so the core does not trash its memory while it is being initialized.
2. For each core: enable or disable L2 cache, depending on the sccKit configuration setting.
3. For each tile: clear Message-Passing Buffer.
4. Load memory image into target core(s); more on that below
5. For each core: set LUTs
6. Clear shared memory regions for SCC<->MCPC communication (host network, SCC performance meter)
7. Tell MCPC driver about the shared memory region (so the crbnet portion knows where to put its data)
8. For each core: release RESET.

The memory image that you input to sccBoot describes the contents seen by one core, so before it can be written to DRAM, it needs to be decided which addresses the cores are going to use (also taking the LUTs into account).

For this, all sccKit software uses a straight-forward convention: core's lower physical address space is "local" memory, mapped to only this core, and resides on the "nearest" DRAM controller. So each controller supplies 12 cores, or from the perspective of the controller, needs 12 copies of the per-core memory image.

Given the per-core image, sccBoot invokes sccMerge to create a per-memory-controller image. sccMerge needs to know the total amount of memory (attached to a single controller) and the number of cores that will use this; it outputs another memory image that can then be loaded as-is onto each of the four memory controllers. For this operation, sccBoot uses the broadcast feature, which sends write requests simultaneously to all controllers (look for calls in sccBoot.cpp that use the "BCMC" flag).

Operation is similar with the "pre-merged" image, although duplication of per-core images is not performed by the MCPC then, but by the SCC itself. In this mode, a single copy of the "pre-merged" image is written, once, to DRAM; then, each core executes the "copy-cat" image under a special LUT mapping. The copycat image copies the image into local memory, then signals the MCPC via a flag in the MPBs. Afterwards, cores can be reset again with the Linux LUT mapping and execution continues normally.

Once the RESET lines are finally released, cores begin execution at the normal entry point 0xFFFFFFF0, just like tpresche wrote. This range contains the BIOS ROM in normal PCs; for the SCC, it simply contains the trampoline code (in reset_vector/reset_vector.S) to jump directly to the loaded kernel.

The kernel bootstrap itself consists of two phases: real-mode and protected-mode startup. The real-mode startup code is located in arch/i386/boot/setup.S for 2.6.17. This code expects that a "boot loader" has set up a specific data structure in memory. For the SCC, the kernel sources have manually been patched to contain the right settings in the right places; like, for example, the address of the loaded (protected-mode) kernel image.

The real-mode setup code then performs basic hardware initialization, normally via software interrupts to the ROM BIOS. As these are not present in the SCC, the calls have been patched out completely. If they are to return certain values (like 0x15/0xe820, which returns usable ranges of physical memory), appropriate defaults have been written into the data area beforehand.

If you want to see the amount of patches involved, I recommend you download the original 2.6.17 kernel and do a diff (especially on setup.S). The startup patches make it comparatively hard to run a newer Linux kernel on the SCC, mainly because the real-mode setup has undergone major changes in later revisions. To work around this, my group has written a very simple BIOS emulator for the SCC, which allows a newer kernel (we are currently at 2.6.37) to boot without any changes to the core real-mode setup. This will be presented at the upcomming MARC symposium, provided our paper gets accepted.

Hi tpresche,

Could you please let me know how does the code in reset_vector gets placed at  the

at the address 0xfffffff0 ?

-Thanks

Hi vjain27,

you can use the tool http://marcbug.scc-dc.com/svn/repository/trunk/linuxkernel/bin2obj/ to merge all your binary

files to an intel object file. Furthermore, you'll need a load.map file like this one http://marcbug.scc-dc.com/svn/repository/trunk/linuxkernel/images/load.map

So let's say you have the reset_vector and your program, test.bin. Then your load.map has to look like this:

0x00090200 test.bin
0xfffff000 reset_vector.bin

Kind regards
Thomas

Thanks a lot!