After completing some more experiments, I've discovered the problem;
Memory mapping from "/dev/mem" (for some unknown reason) does -NOT- give you L2 cached memory! This is a bit strange, as I thought this was supposed to be the 'legacy' mode of memory access, which was supposed to be L2 cached. This also makes me wonder if the memory assigned to normal programs is fully L2 cached or not.
After changing my code to use "/dev/rckdcm", I did see the behavior I was expecting.
I think the LUT becomes active right away. It's 32-bit indexed.It's not write through.
I've only done an mmap() on SCC using /dev/rckncm and /dev/rckdcm. I'm not sure what happens on the SCC if you mmap() from /dev/mem. I think you may be getting at the core's private memory.But I sent off some inquiries to our Linux developers.