10 Replies Latest reply on May 13, 2013 9:48 AM by Nil

    SCC as a shared memory system

    Nil

      Hi,

       

      is it possible to configure SCC in such a way that it behaves as it was shared memory system (linux running on cores). i.e creating some structure on core 0 and pass pointer to core 1  where core 1 can read or write to that structure?

       

      Thank you.

        • 1. Re: SCC as a shared memory system
          saibbot

          I am using the  SHMADD option for shared memory among cores. There is also the POPSHM which I haven't used.

           

          SHMADD partially provides what you need. You get a globally visible shared memory area, but the virtual address space is not common among cores. For example, core 0 may open the address space on 0x977bb000 while core 1 on 0x97869000.

           

          To solve this, you can use offsets from the beginning of the address space as your pointers. So, instead of storing a pointer p, you can store (p - shmem_start_addr) and then to get the pointer do (shmem_start_addr + offset).

           

          Vasilis.

          • 2. Re: SCC as a shared memory system
            Nil

            Hi,

             

            at the moment i am experiencing the issue you mentioned about "virtual address space is not common among cores".

             

            some questions:-

            are you using rcce?

            are using simple malloc or anything special for allocating memory?

            Is it possible for you to post a simple example?

             

            Thank you

            • 3. Re: SCC as a shared memory system
              saibbot

              I am using RCCE_shmalloc and RCCE_shfree.

               

              I initialize the variable shmem_start_address (see attachment)

              Then, I do the following:

              1. before storing to the shared memory (only for pointers), I offset the address (see attachment)
              2. when loading from the shared memory, if the content is a pointer, I recreate the address (see attachment)

               

              Doing so you can implement a linked list for example.

               

              Vasilis.

              • 4. Re: SCC as a shared memory system
                Nil

                Thanks for simple example, here is what I am trying to do:

                 

                Let’s assume I am only using core 0 and 1, at the start on both cores I map lut entries 14,15,16,17,18,19,1a,1b,1c,1d such that it points so same address. I have special malloc (similar to shmalloc from RCCE, let’s call it smalloc) that allocates space in this range.

                 

                Core 0

                Core 1

                Create link list llC0 using smalloc

                Create link list llC1 using smalloc

                Send pointer of llC0 to core 1

                Receive pointer to llC0

                 

                Read/write to llC0

                The problem is I can create link list and send the pointer to core1, but when I try to read ID and testString values I can only get value for ID (correctly prints 101). If I print value for pointers that points to llC0 and llC1 they are both same. And if I change ID value later on and read it again it still gives me 101.

                 

                I had a look at http://intelopenport.hosted.jivesoftware.com/thread/17159?start=0&tstart=0 where it says:

                Allocate a "physical remapping" area (i.e., a set of LUT slots) per core. Map in this range, on all cores, at the same virtual address. Configure the LUTs to point to the same system memory. Now, you can exchange virtual addresses freely across cores (as long as they refer to objects in this special shared section).

                 

                What is that I am doing wrong? Any suggestion?

                I have attached code for my lists as well.

                • 5. Re: SCC as a shared memory system
                  saibbot

                  Do you properly coordinate the allocations of smalloc?

                  What I mean is that if both allocators allocate from the same address space and both take the same steps, then the second will override the list of the first.

                   

                  If what I just said is the case, then you need for instance to put an offset to the allocators of each. So, core 0 could be allocating from address X, while core 1 starting from address (X+offset).

                   

                  I hope I didn't completely miss your point.

                   

                  Vasilis.

                  • 6. Re: SCC as a shared memory system
                    Nil

                    Vasileios Trigonakis wrote:

                     

                    Do you properly coordinate the allocations of smalloc?

                    What I mean is that if both allocators allocate from the same address space and both take the same steps, then the second will override the list of the first.

                    No you did not miss my point.

                     

                    That is happening at the moment they do take nearly same steps, but in that case how can reading ID from llC0 (on core 1) return 101 and not 102 if it was overwitten? Hence the confusion.

                     

                    Secondly:

                    So I actually have to create this smalloc so that it knows if some other core does/done smalloc and it should not overwrite existing blocks?

                    • 7. Re: SCC as a shared memory system
                      saibbot

                      Nil wrote:

                       

                      Vasileios Trigonakis wrote:

                       

                      Do you properly coordinate the allocations of smalloc?

                      What I mean is that if both allocators allocate from the same address space and both take the same steps, then the second will override the list of the first.

                      No you did not miss my point.

                       

                      That is happening at the moment they do take nearly same steps, but in that case how can reading ID from llC0 (on core 1) return 101 and not 102 if it was overwitten? Hence the confusion.

                      Well, are you sure which of the two writes happens first? Do you coordinate the steps with barriers (id0 create list; barrier; id1 create list; ...)?

                       

                      Nil wrote:

                      Secondly:

                      So I actually have to create this smalloc so that it knows if some other core does/done smalloc and it should not overwrite existing blocks?

                      The way I see this, you need to do one of the following. If both cores take the exact same steps and the allocator starts from the same address, then you will end up creating two identical lists with values of the second guy that wrote on each field. This is perfectly OK, as soon as it is what you want. For instance, you could have core 0 doing all the initialization of the shared structures.

                       

                      If you want to avoid this, you need to make something like the following to the allocator:

                      starting_addr_core += ID * offset;
                      

                        Then, each core will be allocating from a separate address space and since you have created a common virtual address space, you can still pass the pointers around over message passing.

                       

                      Otherwise, a more elaborate solution would be to create a common allocator in the shared memory, or the MPB, and coordinate using either  message passing (complex) or locks (the test-and-set register(s)).

                      • 8. Re: SCC as a shared memory system
                        Nil

                        Vasileios Trigonakis wrote:

                         

                        Well, are you sure which of the two writes happens first? Do you coordinate the steps with barriers (id0 create list; barrier; id1 create list; ...)?

                        At the moment i do not use barriers or anything like that, i just run the code on core0 first (waits untill i kil it)and then start on core1.

                         

                         

                        The way I see this, you need to do one of the following. If both cores take the exact same steps and the allocator starts from the same address, then you will end up creating two identical lists with values of the second guy that wrote on each field. This is perfectly OK, as soon as it is what you want. For instance, you could have core 0 doing all the initialization of the shared structures.

                         

                        If you want to avoid this, you need to make something like the following to the allocator:

                        starting_addr_core += ID * offset;
                        

                          Then, each core will be allocating from a separate address space and since you have created a common virtual address space, you can still pass the pointers around over message passing.

                         

                        Otherwise, a more elaborate solution would be to create a common allocator in the shared memory, or the MPB, and coordinate using either  message passing (complex) or locks (the test-and-set register(s)).

                        Since you working with RCCE, can you tell me how it is done there? Is the shmalloc has central structure (something that is visible or shared by all the cores taking part in execution) that is accesses by a core when it calls shmalloc?

                         

                        Thank you for your help so far it has provided me needed insight

                        • 9. Re: SCC as a shared memory system
                          saibbot

                          Nil wrote:

                           

                          Otherwise, a more elaborate solution would be to create a common allocator in the shared memory, or the MPB, and coordinate using either  message passing (complex) or locks (the test-and-set register(s)).

                          Since you working with RCCE, can you tell me how it is done there? Is the shmalloc has central structure (something that is visible or shared by all the cores taking part in execution) that is accesses by a core when it calls shmalloc?

                           

                          Well, it is not implemented in RCCE. If you need something like this, you have to implement it (unless there is an existing implementation that I am not aware of).

                           

                          What I am using is the following:

                          every core performs  a RCCE_shmalloc(ID * offset) as a part of initialization. The returned address is never used or freed during the execution. In this way, every next allocation of a core is "offseted"

                          • 10. Re: SCC as a shared memory system
                            Nil

                            On more question

                             

                            Vasileios Trigonakis wrote:

                             

                            Otherwise, a more elaborate solution would be to create a common allocator in the shared memory, or the MPB, and coordinate using either  message passing (complex) or locks (the test-and-set register(s)).

                             

                            I am re-writing lut entries which overwrites entries 0x80 to 0x84 (default shared) as well in that case there is no shared memory to keep common allocator,  but my question is am I interrupting anything by overwriting 0x80 to 0x84?


                            I am trying to run my prog using pssh and it stops just after I finish lut mapping, but if I run it by logging into core it works fine? 


                            Thank you