12 Replies Latest reply: Oct 31, 2011 8:15 AM by compres RSS

    LUT entries of POPSHM pages

    apapag

      Hello,

       

      I use a kernel that supports POPSHM pages and I have initialized the number of
      pages at each kernel to be 4. So the output of 'cat /proc/meminfo | grep POPSHM'
      is (core 0):
      POPSHM pages:        4
      POPSHM page size:    16384 kB
      POPSHM buffer size:  65536 kB
      POPSHM base address: 0x00000000
      If I understand correct in this case the kernel has allocated 4*16MB=64MB contiguous
      physical memory in 4 LUT entries. Which are those 4 LUT entries and these are
      the same for all cores?
      Thank you,
      Anastasios Papagiannis
        • 1. Re: LUT entries of POPSHM pages
          compres

          apapag wrote:

           

          Hello,

           

          I use a kernel that supports POPSHM pages and I have initialized the number of
          pages at each kernel to be 4. So the output of 'cat /proc/meminfo | grep POPSHM'
          is (core 0):
          POPSHM pages:        4
          POPSHM page size:    16384 kB
          POPSHM buffer size:  65536 kB
          POPSHM base address: 0x00000000
          If I understand correct in this case the kernel has allocated 4*16MB=64MB contiguous
          physical memory in 4 LUT entries. Which are those 4 LUT entries and these are
          the same for all cores?
          Thank you,
          Anastasios Papagiannis

           

          Hello Anastasios,

           

          the location of the 4 LUTs is determined at kernel initialization.  It is only necessary (for the initialization of the library) to know the first one, since as you correctly state, they are contiguous.

           

          You can see the first LUT for each core with the function (http://marcbug.scc-dc.com/svn/repository/trunk/rckmpi/src/mpid/ch3/channels/scc_common/src/popshm.c):

           

          unsigned int popshm_data(int value, int coreid);
          
          By passing POPSHMDATA_LUT as the first parameter and the core id in question as the second parameter.
          Another option is to call the function:
          void popshm_print(int to_print);
          
          

          with the POPSHMPRINT_TABLE option, and it will output information about the internal state of the POPSHM in table form.  The information includes the first LUT entry of each core.  If you pass it POPSHMPRINT_PAGE_LIST, it will print each POPSHM page together with their LUT entries; this option is very verbose.

           

          Another option is to look at the location in shared memoryh where the LUT data is located.  You can tell from the header that these are located in:

           

          #define POPSHM_PAGEDATA_BASE (SHM_X5_Y0 + 0x0800)
          
          where SHM_X5_Y0 is defined as: 
          #define SHM_X5_Y0 0x81000000 

           

          in the file: http://marcbug.scc-dc.com/svn/repository/trunk/rckmpi/src/mpid/ch3/channels/scc_common/include/scc_api.h

           

          - Isaías

          • 2. Re: LUT entries of POPSHM pages
            tedk

            Thanks, Isaias.  Bug 337 is somewhat related to this ... Anastasios has a number of comments on that bug.

            http://marcbug.scc-dc.com/bugzilla3/show_bug.cgi?id=337

             

            What has bothered me is that when I load the very same linux that Anastasios does, I see a different base address. He sees a base address of

            POPSHM pages:        4
            POPSHM page size:    16384 kB
            POPSHM buffer size:  65536 kB
            POPSHM base address: 0x00000000

            I see the following.  Is this OK? What does that POPSHM base address represent?  A different system, but really, I am loading the very same linux obj that he is.

            rck00:/root # echo 16 > /proc/sys/vm/nr_hugepages
            rck00:/root # cat /proc/meminfo |grep POP
            POPSHM pages:        4
            POPSHM page size:    16384 kB
            POPSHM buffer size:  65536 kB
            POPSHM base address: 0x22000000
            rck00:/root #
            • 3. Re: LUT entries of POPSHM pages
              compres

              Ted Kubaska wrote:

               

              Thanks, Isaias.  Bug 337 is somewhat related to this ... Anastasios has a number of comments on that bug.

              http://marcbug.scc-dc.com/bugzilla3/show_bug.cgi?id=337

               

              What has bothered me is that when I load the very same linux that Anastasios does, I see a different base address. He sees a base address of

              POPSHM pages:        4
              POPSHM page size:    16384 kB
              POPSHM buffer size:  65536 kB
              POPSHM base address: 0x00000000

              I see the following.  Is this OK? What does that POPSHM base address represent?  A different system, but really, I am loading the very same linux obj that he is.

              rck00:/root # echo 16 > /proc/sys/vm/nr_hugepages
              rck00:/root # cat /proc/meminfo |grep POP
              POPSHM pages:        4
              POPSHM page size:    16384 kB
              POPSHM buffer size:  65536 kB
              POPSHM base address: 0x22000000
              rck00:/root #

               

              Ohh good catch; I missed that.

               

              That is the physical address as seen by the kernel (therefore 32 bits), where the memory was allocated and pinned (byte 0 of the total  65536 kB).

               

              A value of 0x00000000 is incorrect.  So in your case Ted, it is OK, but incorrect in Anastasios output.

               

              Are these from the 2.6.16 or 2.6.38 kernel?

              • 4. Re: LUT entries of POPSHM pages
                apapag

                Hello Isaias,

                 

                in my case I have 2.6.38.3 kernel.

                • 5. Re: LUT entries of POPSHM pages
                  compres

                  Does the library work in spite of this output?

                   

                  Edit:

                  can you do a:

                   

                  dmesg | grep popshm

                   

                  and post the output here?

                   

                  - Isaías

                  • 6. Re: LUT entries of POPSHM pages
                    apapag

                    What do you mean library? The popshm test gives me a segmentation fault.

                     

                    Anastasios

                    • 7. Re: LUT entries of POPSHM pages
                      compres

                      apapag wrote:

                       

                      What do you mean library? The popshm test gives me a segmentation fault.

                       

                      Anastasios

                       

                      can you do a:

                       

                      dmesg | grep popshm

                       

                      and also:

                       

                      dmesg | grep POPSHM

                       

                      right after you boot and post the output here?

                       

                      - Isaías

                      • 8. Re: LUT entries of POPSHM pages
                        apapag

                        apapag@marc019:/shared/apapag/object$ sccBoot -l linux_912_rckmem_006-38.obj

                        [output of sccBoot]

                        apapag@marc019:/shared/apapag/object$ sleep 60

                        apapag@marc019:/shared/apapag/object$ ssh root@rck00

                        rck00:/root # cat /proc/meminfo  | grep POP

                        POPSHM pages:        0

                        POPSHM page size:    16384 kB

                        POPSHM buffer size:      0 kB

                        POPSHM base address: 0x00000000

                        rck00:/root # dmesg | grep popshm
                        Kernel command line: priv_core_space=638 video=rckgfx:base:638,size:2,x:640,y:480,bpp:16 popshmpages=0
                        --------------
                        I use the same kernel image with Ted from trunk. In Ted's case after boot he has POPSHM enabled.
                        Anastasios
                        • 9. Re: LUT entries of POPSHM pages
                          apapag

                          I missed your second grep. Here is the output:

                          rck00:/root # dmesg | grep POPSHM

                          rck00:/root #

                           

                          Anastasios

                          • 10. Re: LUT entries of POPSHM pages
                            compres

                            apapag wrote:

                             

                            apapag@marc019:/shared/apapag/object$ sccBoot -l linux_912_rckmem_006-38.obj

                            [output of sccBoot]

                            apapag@marc019:/shared/apapag/object$ sleep 60

                            apapag@marc019:/shared/apapag/object$ ssh root@rck00

                            rck00:/root # cat /proc/meminfo  | grep POP

                            POPSHM pages:        0

                            POPSHM page size:    16384 kB

                            POPSHM buffer size:      0 kB

                            POPSHM base address: 0x00000000

                            rck00:/root # dmesg | grep popshm
                            Kernel command line: priv_core_space=638 video=rckgfx:base:638,size:2,x:640,y:480,bpp:16 popshmpages=0
                            --------------
                            I use the same kernel image with Ted from trunk. In Ted's case after boot he has POPSHM enabled.
                            Anastasios

                             

                            That is the problem right there in this line:

                             

                            rck00:/root # dmesg | grep popshm
                            Kernel command line: priv_core_space=638 video=rckgfx:base:638,size:2,x:640,y:480,bpp:16 popshmpages=0

                             

                             

                            That should read at the end:

                            popshmpages=x

                            with x = 1,2,3,4 or 5

                             

                            Are you using the GUI?  There is an entry in the boot setings that allows you to specify the number of pages.  I forgot how the sccKit devs do it from the command line, but should be trivial to do.

                             

                            Ted: do you know how to do it from the command line? Michael can answer that one.

                             

                            - Isaías

                            • 11. Re: LUT entries of POPSHM pages
                              apapag

                              No, I am using comand line. If someone knows how to do this from command line I will be very helpful because I have some problems using the GUI. I don't know if the following helps. After the boot:

                              ----------------
                              rck00:/root # echo 4 > /proc/sys/vm/nr_hugepages
                              rck00:/root # cat /proc/meminfo | grep POP
                              POPSHM pages:        1
                              POPSHM page size:    16384 kB
                              POPSHM buffer size:  16384 kB
                              POPSHM base address: 0x00000000
                              ----------------
                              I will try to use GUI and I will inform you if I use this boot option.
                              
                              Anastasios
                              
                              
                              • 12. Re: LUT entries of POPSHM pages
                                compres

                                apapag wrote:

                                 

                                No, I am using comand line. If someone knows how to do this from command line I will be very helpful because I have some problems using the GUI. I don't know if the following helps. After the boot:

                                ----------------
                                rck00:/root # echo 4 > /proc/sys/vm/nr_hugepages
                                rck00:/root # cat /proc/meminfo | grep POP
                                POPSHM pages:        1
                                POPSHM page size:    16384 kB
                                POPSHM buffer size:  16384 kB
                                POPSHM base address: 0x00000000
                                ----------------
                                I will try to use GUI and I will inform you if I use this boot option.
                                
                                Anastasios
                                
                                

                                Ahh I see what your problem is.

                                 

                                This way of setting the pages at runtime is no longer supported.  That worked in the 2.6.16 kernel, but no longer in the 2.6.38 one.

                                 

                                The number of pages for the 2.6.38 kernel can only be set at boot time.  This was also added as an option to the GUI, but I am not aware how this was added to the comman line tools.  Let's wait for Ted or Michael on how to do this from the CLI.

                                 

                                - Isaías