12 Replies Latest reply: Oct 31, 2011 8:15 AM by compres RSS

LUT entries of POPSHM pages

apapag Community Member
Currently Being Moderated

Hello,

 

I use a kernel that supports POPSHM pages and I have initialized the number of
pages at each kernel to be 4. So the output of 'cat /proc/meminfo | grep POPSHM'
is (core 0):
POPSHM pages:        4
POPSHM page size:    16384 kB
POPSHM buffer size:  65536 kB
POPSHM base address: 0x00000000
If I understand correct in this case the kernel has allocated 4*16MB=64MB contiguous
physical memory in 4 LUT entries. Which are those 4 LUT entries and these are
the same for all cores?
Thank you,
Anastasios Papagiannis
  • 1. Re: LUT entries of POPSHM pages
    compres Community Member
    Currently Being Moderated

    apapag wrote:

     

    Hello,

     

    I use a kernel that supports POPSHM pages and I have initialized the number of
    pages at each kernel to be 4. So the output of 'cat /proc/meminfo | grep POPSHM'
    is (core 0):
    POPSHM pages:        4
    POPSHM page size:    16384 kB
    POPSHM buffer size:  65536 kB
    POPSHM base address: 0x00000000
    If I understand correct in this case the kernel has allocated 4*16MB=64MB contiguous
    physical memory in 4 LUT entries. Which are those 4 LUT entries and these are
    the same for all cores?
    Thank you,
    Anastasios Papagiannis

     

    Hello Anastasios,

     

    the location of the 4 LUTs is determined at kernel initialization.  It is only necessary (for the initialization of the library) to know the first one, since as you correctly state, they are contiguous.

     

    You can see the first LUT for each core with the function (http://marcbug.scc-dc.com/svn/repository/trunk/rckmpi/src/mpid/ch3/channels/scc_common/src/popshm.c):

     

    unsigned int popshm_data(int value, int coreid);
    
    By passing POPSHMDATA_LUT as the first parameter and the core id in question as the second parameter.
    Another option is to call the function:
    void popshm_print(int to_print);
    
    

    with the POPSHMPRINT_TABLE option, and it will output information about the internal state of the POPSHM in table form.  The information includes the first LUT entry of each core.  If you pass it POPSHMPRINT_PAGE_LIST, it will print each POPSHM page together with their LUT entries; this option is very verbose.

     

    Another option is to look at the location in shared memoryh where the LUT data is located.  You can tell from the header that these are located in:

     

    #define POPSHM_PAGEDATA_BASE (SHM_X5_Y0 + 0x0800)
    
    where SHM_X5_Y0 is defined as: 
    #define SHM_X5_Y0 0x81000000 

     

    in the file: http://marcbug.scc-dc.com/svn/repository/trunk/rckmpi/src/mpid/ch3/channels/scc_common/include/scc_api.h

     

    - Isaías

  • 2. Re: LUT entries of POPSHM pages
    tedk Community Member
    Currently Being Moderated

    Thanks, Isaias.  Bug 337 is somewhat related to this ... Anastasios has a number of comments on that bug.

    http://marcbug.scc-dc.com/bugzilla3/show_bug.cgi?id=337

     

    What has bothered me is that when I load the very same linux that Anastasios does, I see a different base address. He sees a base address of

    POPSHM pages:        4
    POPSHM page size:    16384 kB
    POPSHM buffer size:  65536 kB
    POPSHM base address: 0x00000000

    I see the following.  Is this OK? What does that POPSHM base address represent?  A different system, but really, I am loading the very same linux obj that he is.

    rck00:/root # echo 16 > /proc/sys/vm/nr_hugepages
    rck00:/root # cat /proc/meminfo |grep POP
    POPSHM pages:        4
    POPSHM page size:    16384 kB
    POPSHM buffer size:  65536 kB
    POPSHM base address: 0x22000000
    rck00:/root #
  • 3. Re: LUT entries of POPSHM pages
    compres Community Member
    Currently Being Moderated

    Ted Kubaska wrote:

     

    Thanks, Isaias.  Bug 337 is somewhat related to this ... Anastasios has a number of comments on that bug.

    http://marcbug.scc-dc.com/bugzilla3/show_bug.cgi?id=337

     

    What has bothered me is that when I load the very same linux that Anastasios does, I see a different base address. He sees a base address of

    POPSHM pages:        4
    POPSHM page size:    16384 kB
    POPSHM buffer size:  65536 kB
    POPSHM base address: 0x00000000

    I see the following.  Is this OK? What does that POPSHM base address represent?  A different system, but really, I am loading the very same linux obj that he is.

    rck00:/root # echo 16 > /proc/sys/vm/nr_hugepages
    rck00:/root # cat /proc/meminfo |grep POP
    POPSHM pages:        4
    POPSHM page size:    16384 kB
    POPSHM buffer size:  65536 kB
    POPSHM base address: 0x22000000
    rck00:/root #

     

    Ohh good catch; I missed that.

     

    That is the physical address as seen by the kernel (therefore 32 bits), where the memory was allocated and pinned (byte 0 of the total  65536 kB).

     

    A value of 0x00000000 is incorrect.  So in your case Ted, it is OK, but incorrect in Anastasios output.

     

    Are these from the 2.6.16 or 2.6.38 kernel?

  • 4. Re: LUT entries of POPSHM pages
    apapag Community Member
    Currently Being Moderated

    Hello Isaias,

     

    in my case I have 2.6.38.3 kernel.

  • 5. Re: LUT entries of POPSHM pages
    compres Community Member
    Currently Being Moderated

    Does the library work in spite of this output?

     

    Edit:

    can you do a:

     

    dmesg | grep popshm

     

    and post the output here?

     

    - Isaías

  • 6. Re: LUT entries of POPSHM pages
    apapag Community Member
    Currently Being Moderated

    What do you mean library? The popshm test gives me a segmentation fault.

     

    Anastasios

  • 7. Re: LUT entries of POPSHM pages
    compres Community Member
    Currently Being Moderated

    apapag wrote:

     

    What do you mean library? The popshm test gives me a segmentation fault.

     

    Anastasios

     

    can you do a:

     

    dmesg | grep popshm

     

    and also:

     

    dmesg | grep POPSHM

     

    right after you boot and post the output here?

     

    - Isaías

  • 8. Re: LUT entries of POPSHM pages
    apapag Community Member
    Currently Being Moderated

    apapag@marc019:/shared/apapag/object$ sccBoot -l linux_912_rckmem_006-38.obj

    [output of sccBoot]

    apapag@marc019:/shared/apapag/object$ sleep 60

    apapag@marc019:/shared/apapag/object$ ssh root@rck00

    rck00:/root # cat /proc/meminfo  | grep POP

    POPSHM pages:        0

    POPSHM page size:    16384 kB

    POPSHM buffer size:      0 kB

    POPSHM base address: 0x00000000

    rck00:/root # dmesg | grep popshm
    Kernel command line: priv_core_space=638 video=rckgfx:base:638,size:2,x:640,y:480,bpp:16 popshmpages=0
    --------------
    I use the same kernel image with Ted from trunk. In Ted's case after boot he has POPSHM enabled.
    Anastasios
  • 9. Re: LUT entries of POPSHM pages
    apapag Community Member
    Currently Being Moderated

    I missed your second grep. Here is the output:

    rck00:/root # dmesg | grep POPSHM

    rck00:/root #

     

    Anastasios

  • 10. Re: LUT entries of POPSHM pages
    compres Community Member
    Currently Being Moderated

    apapag wrote:

     

    apapag@marc019:/shared/apapag/object$ sccBoot -l linux_912_rckmem_006-38.obj

    [output of sccBoot]

    apapag@marc019:/shared/apapag/object$ sleep 60

    apapag@marc019:/shared/apapag/object$ ssh root@rck00

    rck00:/root # cat /proc/meminfo  | grep POP

    POPSHM pages:        0

    POPSHM page size:    16384 kB

    POPSHM buffer size:      0 kB

    POPSHM base address: 0x00000000

    rck00:/root # dmesg | grep popshm
    Kernel command line: priv_core_space=638 video=rckgfx:base:638,size:2,x:640,y:480,bpp:16 popshmpages=0
    --------------
    I use the same kernel image with Ted from trunk. In Ted's case after boot he has POPSHM enabled.
    Anastasios

     

    That is the problem right there in this line:

     

    rck00:/root # dmesg | grep popshm
    Kernel command line: priv_core_space=638 video=rckgfx:base:638,size:2,x:640,y:480,bpp:16 popshmpages=0

     

     

    That should read at the end:

    popshmpages=x

    with x = 1,2,3,4 or 5

     

    Are you using the GUI?  There is an entry in the boot setings that allows you to specify the number of pages.  I forgot how the sccKit devs do it from the command line, but should be trivial to do.

     

    Ted: do you know how to do it from the command line? Michael can answer that one.

     

    - Isaías

  • 11. Re: LUT entries of POPSHM pages
    apapag Community Member
    Currently Being Moderated

    No, I am using comand line. If someone knows how to do this from command line I will be very helpful because I have some problems using the GUI. I don't know if the following helps. After the boot:

    ----------------
    rck00:/root # echo 4 > /proc/sys/vm/nr_hugepages
    rck00:/root # cat /proc/meminfo | grep POP
    POPSHM pages:        1
    POPSHM page size:    16384 kB
    POPSHM buffer size:  16384 kB
    POPSHM base address: 0x00000000
    ----------------
    I will try to use GUI and I will inform you if I use this boot option.
    
    Anastasios
    
    
  • 12. Re: LUT entries of POPSHM pages
    compres Community Member
    Currently Being Moderated

    apapag wrote:

     

    No, I am using comand line. If someone knows how to do this from command line I will be very helpful because I have some problems using the GUI. I don't know if the following helps. After the boot:

    ----------------
    rck00:/root # echo 4 > /proc/sys/vm/nr_hugepages
    rck00:/root # cat /proc/meminfo | grep POP
    POPSHM pages:        1
    POPSHM page size:    16384 kB
    POPSHM buffer size:  16384 kB
    POPSHM base address: 0x00000000
    ----------------
    I will try to use GUI and I will inform you if I use this boot option.
    
    Anastasios
    
    

    Ahh I see what your problem is.

     

    This way of setting the pages at runtime is no longer supported.  That worked in the 2.6.16 kernel, but no longer in the 2.6.38 one.

     

    The number of pages for the 2.6.38 kernel can only be set at boot time.  This was also added as an option to the GUI, but I am not aware how this was added to the comman line tools.  Let's wait for Ted or Michael on how to do this from the CLI.

     

    - Isaías

More Like This

  • Retrieving data ...

Legend

  • Correct Answers - 4 points
  • Helpful Answers - 2 points