3 Replies Latest reply on May 13, 2011 4:44 PM by mwvantol

    Reads from uncachable memory hitting in L1 cache

    mwvantol

      As contradictory as the title sounds, I recently discovered that uncachable memory accesses on the SCC (through /dev/rckncm) are not fully guaranteed to not hit the cache. This might be relevant for people that are using uncachable memory to have a coherent shared memory between cores.

       

      I have observed the behavior that when reading from memory locations (marked uncachable through /dev/rckncm ) resulted in hits in the L1 cache, if these same physical addresses have been recently accessed through cachable memory (/dev/rckdcm) - therefore they must have still been present in the L1 cache. This only seems to be the case for the L1 cache, the L2 cache seems to be fully bypassed.

       

      These observations drew me to the following conclusions;

      - The L1 cache is physically addressed, not virtually (I could not confirm this in the SCC documentation, but I think it was mentioned in the P54C specs)

      - Whenever a read of an address is done that is marked 'uncachable' by the PCD bit in the page table, this means that the L1 cache will not replace a line on a miss, however, on a hit it would still return the cached data. (I could not yet test what happens on a write hit, but my test revealed this happens on a read hit)

      - The L2 is always bypassed when PCD is set

       

      Can someone confirm this? Is this the expected behavior? Is this a bug?

       

      I wasn't sure what it was, so I didn't straight away wanted to open a new ticket on the Marcbug Bugzilla.

      Ted: I have test code available if you're interested.

        • 1. Re: Reads from uncachable memory hitting in L1 cache
          tedk

          L1 and L2 on the SCC chip are not the same as L1 and L2 on Pentium. Similar but not the same, and this has caused confusion for many people ... especially if you are very familiar with Pentium docs.

           

          For example, the SCC L1 is larger ... 16 KB code and 16 KB data. It does this by being 4-way rather than 2-way. There are the INVD and the WBINVD instructions for L1 ... but these instructions sadly do not affect L2.

           

          As you observed, the L1 cache is physically addressed. This is not a difference. This is true on Pentium as well.

           

          Refer to CR0 and the bits CD and NW. Yes, you can disable L1. To do that set both bits to 1. But when you do that, read hits still access the cache and write hits update the cache but not memory. You might argue that this is not really disabling, but it is if you invalidate and flush L1 first. Then, there won't be any read or write L1 hits. You can do this on a page basis with the PCD and PWT bits of the page table entry.

           

          I don't think the L2 is bypassed when PCD is set. The L2 is bypassed when MPBT is set. This is bit 7 in the page table entry. The MPBT bit does not exist on Pentium ... well, it does, but it is a reserved bit.

           

          If you take some memory that was cacheable and make it uncacheable, you can certainly flush L1 first and be OK. But since we don't have a 100% reliable L2 flush (we might have with the latest update; remains to be seen), the data can get stuck in L2.

           

          Bugzilla is OK to use. Not everything in Bugzilla is a "bug."  Sometimes we put very detailed (hardware, architectural) questions in Bugzilla. The Severity is flagged as how important an answer is to your research. The reason for using Bugzilla this way is that Bugzilla entries get more internal visibility. That's why that long discussion on L2 cache flushing is a "bug" rather than a forum entry.

           

          So yes, if you want to create a bug and attach your test code, one of us will try to look at it. I guess you're saying you think L2 is being affected when we say it isn't.

          • 2. Re: Reads from uncachable memory hitting in L1 cache
            tedk

            BTW, we say the SCC chip is based on the P54C. This is pre Pentium Pro. This is the Pentium from 1993. The original Pentium didn't even have an L2.

            • 3. Re: Reads from uncachable memory hitting in L1 cache
              mwvantol

              Ted Kubaska wrote:

               

              Refer to CR0 and the bits CD and NW. Yes, you can disable L1. To do that set both bits to 1. But when you do that, read hits still access the cache and write hits update the cache but not memory. You might argue that this is not really disabling, but it is if you invalidate and flush L1 first. Then, there won't be any read or write L1 hits. You can do this on a page basis with the PCD and PWT bits of the page table entry.

               

              I think this bit answers my question. I was indeed using the PCD bit to mark as uncachable, but was still getting data back from cache hits. As this only happened in a 16KB region of the whole 256KB region I was testing, my conclusion was that only L1 was returning data to me and not L2. Therefore my conclusion was that L2 was really bypassed, and L1 is well, let's call it "pseudo-bypassed"

               

              Anyway, I guess it might be benificial to document this somewhere, that a flush is required. I did not come accross this in the original P54C docs, but of course the original core does not have LUTs so you can remap memory beyond what the core can detect. As that is potentially where someone could encouter this on the SCC; remapping a block of 'real physical memory' to a 'core physical' range that was used before, then thinking they can to access it safely by mapping it as uncached, and then still reading stale data (kind of counter intuitive - that you have to flush your cache before being able to read uncached data, right? )

               

              I'll create a Bugzilla entry and share my test there.

               

              Thanks.