I am unable to understand how the symmetric memory model and the collective operation RCCE_malloc work
and how they are useful. Following are some of the queries that I have -
- When RCCE_malloc() is executed , the offset of memory area allocated on a core's MPB from the start of the MPB is same for all the cores.
Am I right ?
- Is it correct that the contents of the memory areas created by a single RCC_malloc() are not meant to be identical ?r
- How do RCC_put and RCC_get use this symmetric model ?
It would be great if an example is used to explain.
RCCE_malloc() and RCCE_free() are assumed to be called collectively by all cores, in the same program order, to facilitate message passing via RCCE_put() and RCCE_get().
Suppose you call RCCE_malloc() to allocate a buffer b in the MPB of core i. How can you access the buffer on, say, core j? RCCE maintains an array where j can look up the address at which i's portion of MPB memory starts, but that's about it. You would need to determine the buffer offset (in bytes) on i's side and communicate that offset to j, to be able to perform remote memory accesses. By relying on collective allocations, and thus a symmetric name space, you basically sidestep that communication. If both i and j call RCCE_malloc(), the buffer offset can be determined implicitly by just looking at j's MPB. No need to communicate information from i to j.
Internally, RCCE_put() and RCCE_get() include code like this:
int offset = (int)b - RCCE_comm_buffer[myID]; char *p = (char *)(RCCE_comm_buffer[otherID] + offset); // put/get using p
This works precisely because we assume collective allocations. So, yes, as long as you perform identical allocations on every core, you can be sure of identical buffer offsets and take advantage of it.
If you do it collectively, yes. If you want to save space and not replicate the buffer, that is, if you want to call RCCE_malloc() only once, you have to explicitly pass the buffer offset to every core that needs access (for example, by setting a shared variable in off-chip memory). But note that RCCE_put() and RCCE_get() assume collective allocations. If you break this assumption, you must write your own put and get functions.
Thanks! There is just one more thing that I need to be clear about. I was looking at the code for RCCE_malloc() and I saw that it uses standard malloc() for allocating memory.
b2 = (RCCE_BLOCK *) malloc(sizeof(RCCE_BLOCK));
So is the standard malloc() modified to allocate memory from MPB instead of private memory?
No, MPB memory is actually allocated (memory-mapped) at the beginning in RCCE_init(). Whenever you RCCE_malloc(), you mark a block of MPB memory as used (and return this block), and vice versa, whenever you RCCE_free(), you mark a block of MPB memory as free. The RCCE_BLOCKs that are allocated here are used to keep track of which portions of the MPB are currently in use. It's required for internal bookkeeping.