LUT entries of POPSHM pages

apapag wrote:

 

Hello,

 

I use a kernel that supports POPSHM pages and I have initialized the number of

pages at each kernel to be 4. So the output of 'cat /proc/meminfo | grep POPSHM'

is (core 0):

POPSHM pages:        4

POPSHM page size:    16384 kB

POPSHM buffer size:  65536 kB

POPSHM base address: 0x00000000

If I understand correct in this case the kernel has allocated 4*16MB=64MB contiguous

physical memory in 4 LUT entries. Which are those 4 LUT entries and these are

the same for all cores?

Thank you,

Anastasios Papagiannis

Hello Anastasios,

the location of the 4 LUTs is determined at kernel initialization.  It is only necessary (for the initialization of the library) to know the first one, since as you correctly state, they are contiguous.

You can see the first LUT for each core with the function (http://marcbug.scc-dc.com/svn/repository/trunk/rckmpi/src/mpid/ch3/channels/scc_common/src/popshm.c):

unsigned int popshm_data(int value, int coreid);

By passing POPSHMDATA_LUT as the first parameter and the core id in question as the second parameter.

Another option is to call the function:

void popshm_print(int to_print);

with the POPSHMPRINT_TABLE option, and it will output information about the internal state of the POPSHM in table form.  The information includes the first LUT entry of each core.  If you pass it POPSHMPRINT_PAGE_LIST, it will print each POPSHM page together with their LUT entries; this option is very verbose.

Another option is to look at the location in shared memoryh where the LUT data is located.  You can tell from the header that these are located in:

#define POPSHM_PAGEDATA_BASE (SHM_X5_Y0 + 0x0800)

where SHM_X5_Y0 is defined as: 
#define SHM_X5_Y0 0x81000000 

in the file: http://marcbug.scc-dc.com/svn/repository/trunk/rckmpi/src/mpid/ch3/channels/scc_common/include/scc_api.h

- Isaías

Thanks, Isaias.  Bug 337 is somewhat related to this ... Anastasios has a number of comments on that bug.

http://marcbug.scc-dc.com/bugzilla3/show_bug.cgi?id=337

What has bothered me is that when I load the very same linux that Anastasios does, I see a different base address. He sees a base address of

POPSHM pages:        4
POPSHM page size:    16384 kB
POPSHM buffer size:  65536 kB
POPSHM base address: 0x00000000

I see the following.  Is this OK? What does that POPSHM base address represent?  A different system, but really, I am loading the very same linux obj that he is.

rck00:/root # echo 16 > /proc/sys/vm/nr_hugepages
rck00:/root # cat /proc/meminfo |grep POP
POPSHM pages:        4
POPSHM page size:    16384 kB
POPSHM buffer size:  65536 kB
POPSHM base address: 0x22000000
rck00:/root #

Ted Kubaska wrote:

 

Thanks, Isaias.  Bug 337 is somewhat related to this ... Anastasios has a number of comments on that bug.

http://marcbug.scc-dc.com/bugzilla3/show_bug.cgi?id=337

 

What has bothered me is that when I load the very same linux that Anastasios does, I see a different base address. He sees a base address of

POPSHM pages:        4
POPSHM page size:    16384 kB
POPSHM buffer size:  65536 kB
POPSHM base address: 0x00000000

I see the following.  Is this OK? What does that POPSHM base address represent?  A different system, but really, I am loading the very same linux obj that he is.

rck00:/root # echo 16 > /proc/sys/vm/nr_hugepages
rck00:/root # cat /proc/meminfo |grep POP
POPSHM pages:        4
POPSHM page size:    16384 kB
POPSHM buffer size:  65536 kB
POPSHM base address: 0x22000000
rck00:/root #

Ohh good catch; I missed that.

That is the physical address as seen by the kernel (therefore 32 bits), where the memory was allocated and pinned (byte 0 of the total  65536 kB).

A value of 0x00000000 is incorrect.  So in your case Ted, it is OK, but incorrect in Anastasios output.

Are these from the 2.6.16 or 2.6.38 kernel?

Does the library work in spite of this output?

Edit:

can you do a:

dmesg | grep popshm

and post the output here?

- Isaías

apapag wrote:

 

What do you mean library? The popshm test gives me a segmentation fault.

 

Anastasios

can you do a:

dmesg | grep popshm

and also:

dmesg | grep POPSHM

right after you boot and post the output here?

- Isaías

apapag wrote:

 

apapag@marc019:/shared/apapag/object$ sccBoot -l linux_912_rckmem_006-38.obj

[output of sccBoot]

apapag@marc019:/shared/apapag/object$ sleep 60

apapag@marc019:/shared/apapag/object$ ssh root@rck00

rck00:/root # cat /proc/meminfo  | grep POP

POPSHM pages:        0

POPSHM page size:    16384 kB

POPSHM buffer size:      0 kB

POPSHM base address: 0x00000000

rck00:/root # dmesg | grep popshm

Kernel command line: priv_core_space=638 video=rckgfx:base:638,size:2,x:640,y:480,bpp:16 popshmpages=0

--------------

I use the same kernel image with Ted from trunk. In Ted's case after boot he has POPSHM enabled.

Anastasios

That is the problem right there in this line:

rck00:/root # dmesg | grep popshm

Kernel command line: priv_core_space=638 video=rckgfx:base:638,size:2,x:640,y:480,bpp:16 popshmpages=0

That should read at the end:

popshmpages=x

with x = 1,2,3,4 or 5

Are you using the GUI?  There is an entry in the boot setings that allows you to specify the number of pages.  I forgot how the sccKit devs do it from the command line, but should be trivial to do.

Ted: do you know how to do it from the command line? Michael can answer that one.

- Isaías

apapag wrote:

 

No, I am using comand line. If someone knows how to do this from command line I will be very helpful because I have some problems using the GUI. I don't know if the following helps. After the boot:

----------------
rck00:/root # echo 4 > /proc/sys/vm/nr_hugepages
rck00:/root # cat /proc/meminfo | grep POP
POPSHM pages:        1
POPSHM page size:    16384 kB
POPSHM buffer size:  16384 kB
POPSHM base address: 0x00000000
----------------
I will try to use GUI and I will inform you if I use this boot option.

Anastasios

Ahh I see what your problem is.

This way of setting the pages at runtime is no longer supported.  That worked in the 2.6.16 kernel, but no longer in the 2.6.38 one.

The number of pages for the 2.6.38 kernel can only be set at boot time.  This was also added as an option to the GUI, but I am not aware how this was added to the comman line tools.  Let's wait for Ted or Michael on how to do this from the CLI.

- Isaías