1 2 Previous Next 25 Replies Latest reply on Dec 5, 2011 6:15 PM by mwvantol

    WCB - When is it exactly flushed?

    mwvantol

      [edit]: This discussion was named "WCB - Physical or Virtual addressed?" previously, but was renamed to reflect where the discussion went. Original question starts below;

       


       

       

       

      Just a question that popped into my mind; is the cacheline sized Write Combine Buffer that is enabled for MPBT flagged data physically or virtually addressed? Because I was wondering what happens when the processor context switches while there is still data in the WCB that is not written back yet. If the other process then triggers the WCB's writeback, is it possible that the data ends up in the wrong address space?

       

       

       

      In fact, it would be helpful if there would be a diagram that represents the SCC's memory hierarchy. I also noticed while reading the community discussions here that this often causes confusion.

        • 1. Re: WCB - Physical or Virtual addressed?
          tedk

          I'm pretty sure it's physical, but I pinged one of our cache designers to be sure. The cache is physical. Why would the WCB not be?

           

          I attached a diagram that shows SCC cache policy. It's been floating around for a while but I dont't think it's ever been officially published. I put it together after talking with a number of people. If you see an error, please point it out.

          • 2. Re: WCB - Physical or Virtual addressed?
            junghyun

            Hi, Ted.

             

            I have a question about the diagram.

             

            Assume that a page is MPBT+WT type.

             

            If a write of the page hits L1 cache, then does it go to L2 cache??

             

            I think it needs another check routine before checking L2 hit.

            • 3. Re: WCB - Physical or Virtual addressed?
              rrotta

              This is a very helpful diagram. The non-cacheable memory type (NCM) is missing. It looks like it is checked similary to the MPBT type after the L1 check, not before and this is important when working with this type. What happens to an NCM write/read when the data is present in the L2 cache? That would be interesting when using NCM for random access to arrays (because caching does not necessarily help in such situations).

              • 4. Re: WCB - Physical or Virtual addressed?
                tedk

                Sorry, Junghyun ... I've been out-of-office for a couple of weeks. Yes, I do think there is need for a MPBT check after an L1 hit before checking L2. This is because MPBT typed data bypass the L2. I had thought that this was covered by just assuming that MPBT typed data would not be an L2 hit (would go down the no path). This is not true because such an assumption neglects the WCB.  I attached a new png.

                • 5. Re: WCB - Physical or Virtual addressed?
                  tedk

                  Randolf, I don't understand your comment. The diagram is intended to show cacheable writes. Why would an NCM write be checked for NCM after an L1 check? I think if it's NCM, it's NCM, and there isn't any cache checking going on.

                   

                  If the data are in L2, then you have to do a read first before the NCM write. Am I missing something here?

                   

                  One of (probably the biggest) difficulty with understanding the SCC cache is that you can't really write good test programs because even your test program is using the cache. So research turns into reading specs and talking with the cache designers. Hence, the forum is a good place to bring questions up, because Intel's cache designers are not going to let an erroneous statement remain uncorrected, although it may require someone (often me) to bring the post to their attention.

                   

                  One aspect that I find confusing in the fact that L1 and L2 are not strictly inclusive ... often inclusive but apparently not guarenteed to be.

                  • 6. Re: WCB - Physical or Virtual addressed?
                    junghyun

                    Thank you for updating the diagram.

                    I have some questions.
                    1) Does any write after an MPBT-type write flush WCB?
                    For example, consider this scenario.
                    - an MPBT-type write to address 0x1000
                    - a non-cacheable write to address 0x2000
                    Does the second write make WCB flushed?
                    2) If WCB is not filled fully, are they non-burst writes?
                    Assume all writes are 4-byte writes.
                    - an MPBT-type write to address 0x1000
                    - an MPBT-type write to address 0x1004
                    - an MPBT-type write to address 0x100c
                    - an MPBT-type write to address 0x1100
                    In this case, the three writes are non-burst?
                    • 7. Re: WCB - Physical or Virtual addressed?
                      tedk
                      >> 1) Does any write after an MPBT-type write flush WCB?
                      >>  For example, consider this scenario.
                      >>  - an MPBT-type write to address 0x1000
                      >>  - a non-cacheable write to address 0x2000
                      >> Does the second write make WCB flushed?
                      Yes. Definitely if the next write is non-contiguous. I don't know what happens if the next write is a contiguous non-cacheable memory location. I think (but not positive) that the answer in that case is also yes.
                      >> 2) If WCB is not filled fully, are they non-burst writes?
                      >> Assume all writes are 4-byte writes.
                      >> - an MPBT-type write to address 0x1000
                      >> - an MPBT-type write to address 0x1004
                      >> - an MPBT-type write to address 0x100c
                      >> - an MPBT-type write to address 0x1100
                      >> In this case, the three writes are non-burst?
                      We may be playing somewhat fast and loose with the terms burst and non-burst in the diagram. What happens in the case you describe is that after the first three writes the WCB looks like  | ? | x | x | x | where the x represents data to be flushed and the ? is undefined. The fourth write then flushes the WCB. All 32 bytes go on the bus with byte enables for the 24 valid bytes. Writes are 4-byte writes so there are actually 8 available byte enables.
                      When people say non-burst, they often mean that each 4-byte write is sent out separately, and that is not really what happens. Burst mode would be when all 32 bytes of the cacheline are written.
                      • 8. Re: WCB - Physical or Virtual addressed?
                        rrotta

                        Ted Kubaska wrote:

                        The diagram is intended to show cacheable writes. Why would an NCM write be checked for NCM after an L1 check? I think if it's NCM, it's NCM, and there isn't any cache checking going on.

                         

                        Hello, many thanks for the reply. I am just discovering the forum as a tool and still fear to ask the interesting questions

                         

                        I think there is some fundamental difficulty when talking about memory access types. Traditionally, the focus seems to be on the destination memory: For certain destinations one has to use non-cached access (e.g. SCC's CRBs and SIF) and in all other cases, cached access is the only accepted way. This point of view is, for example, reflected in the naming scheme for access modes (aka memory types) in the page table: "non-cacheable memory", "message passing buffer type", "cachable memory" and so on. However, from an application point of view it is more interesting to focus on how some data is accessed depending on the current access pattern: For a linear scan through a data array the cached access mode provides better performance than no caching, while for random access the caching does not improve performance (each access is a cache miss) and puts alot of unnecessary presure to the caches (pushes out other data). Thus, during random access phases, an application would like to use non-cached access to this data.

                         

                        This is the background why I asked about non-cached access (NCM) to physical memory that was previously accessed through cached access. Some other people already observed, that it is not possible to access memory through NCM access when the data is still present in the L1 cache. Somehow the L1 cache hit for the physical address overrides the access mode configuration for the virtual address and, on the SCC, even wrong results might be returned. In the past we circumvented this by mixing just MPBT and NCM access: MPBT data is never stored in the L2 cache and MPBT data in the L1 cache can be invalidated by SCC's special instruction, NCM data is never stored in any cache. Thus, there isn't this problem when switching from NCM to MPBT access, and before switching from MPBT to NCM it is sufficient to invalidate the MPBT lines from the L1 cache.

                         

                        I guess there are some undocumented/unintended interactions when mixing cached and non-cached access to the same physical memory. A L1 cache hit to a NCM access probably always is a problem on the SCC. This might be solved by flushing the L1 before accessing the data. But what happens if there is a NCM read access to data that still is present in the L2 cache? Is the cache ignored (ok), is the data read from the cache (good), or does it result in garbage (bad)? Likewise for write requests...

                         

                        Best regards, randolf

                        • 9. Re: WCB - Physical or Virtual addressed?
                          mwvantol

                          Ted Kubaska wrote:

                           

                          >> 1) Does any write after an MPBT-type write flush WCB?
                          >>  For example, consider this scenario.
                          >>  - an MPBT-type write to address 0x1000
                          >>  - a non-cacheable write to address 0x2000
                          >> Does the second write make WCB flushed?
                          Yes. Definitely if the next write is non-contiguous. I don't know what happens if the next write is a contiguous non-cacheable memory location. I think (but not positive) that the answer in that case is also yes.

                          Unfortunately, I do not believe this is the case. In our work on our SVP run-time on the SCC for which we submitted a paper to the MARC4 symposium we encountered this situation. As we covered in our previous MARC3 paper, and by other as well, copying memory is much faster using MPBT type writes. However, initially we forgot to insert a dummy write after the copy operation. And for some reason, the last cacheline written seemed to be updated at a much much later point in time - actually giving us the impression that the WCB is only flushed when -another- MPBT write is done which is a different cacheline. I mean, normal memory writes would happen often if you're using the stack, going in/out of functions etc. But we got many inconsistencies and errors receiving data, which were fixed by inserting a dummy write.

                          • 10. Re: WCB - Physical or Virtual addressed?
                            aprell

                            Michiel, I think you have a point there. I had a situation where one core was regularly updating part of an MPBT cache line, but other cores never saw a single update during the entire execution of the program. This was easily fixed by writing whole cache lines, but in other instances, I had additional MPBT writes in the program, which made it look like everything worked as intended...

                            • 11. Re: WCB - Physical or Virtual addressed?
                              tedk

                              Thanks, Michiel. Data do not lie. I got the information from talking with our cache designers. When we looked at the RTL, we interpreted it as showing that even a write to non MPBT data flushed the WCB. But if that write is non-cacheable. Hmmm ... We'll take another look.

                              • 12. Re: WCB - Physical or Virtual addressed?
                                tedk

                                Michiel, just a clarification ...

                                 

                                Case 1:

                                We write MPBT data to the WCB. The WCB is not filled.

                                The next write is to a non-cacheable memory address that is non-contiguous?

                                Is the WCB flushed? I'm not sure now (because of the non-cacheable aspect) and need to revisit.

                                 

                                Case 2:

                                We write MPBT data to the WCB. The WCB is not filled.

                                The next write is to a cacheable memory address (but not MPBT data) that is non-contiguous?

                                Is the WCB flushed? Do you have data that says that the WCB is not flushed in this case?

                                • 13. Re: WCB - Physical or Virtual addressed?
                                  mwvantol

                                  I have conducted some experiments today. I'm not completely done yet, but the interactions between MPBT, the L1 cache and the WCB are quite 'interesting' to say the least. Also weird stuff happens when you access the L1 (which is physically addressed) while using a cacheable and an MPBT mapping to the same physical addresses. MPBT writes for example hit on non-MPBT lines, but MPBT reads dont... and in one case even my modified data seemed to be lost completely.

                                   

                                  Anyway, to answer your question Ted;

                                  Ted Kubaska wrote:

                                   

                                  Michiel, just a clarification ...

                                   

                                  Case 1:

                                  We write MPBT data to the WCB. The WCB is not filled.

                                  The next write is to a non-cacheable memory address that is non-contiguous?

                                  Is the WCB flushed? I'm not sure now (because of the non-cacheable aspect) and need to revisit.

                                   

                                  At least for single byte writes this is not the case. Uncached single byte writes to the same word, same 'cacheline' or somewhere else do not seem to influence the WCB.

                                   

                                  Ted Kubaska wrote:

                                   

                                  Case 2:

                                  We write MPBT data to the WCB. The WCB is not filled.

                                  The next write is to a cacheable memory address (but not MPBT data) that is non-contiguous?

                                  Is the WCB flushed? Do you have data that says that the WCB is not flushed in this case?

                                  I am 100% sure now that the WCB is not flushed. I wrote a single byte MPBT so that it hangs in the WCB, then wrote 16 MB to cacheable memory, and after that the byte in the WCB has not been written back yet. Only after I do a dummy MPBT write it appears in memory. This is while none of the locations are present in the L1 or L2 caches (so from a clean state).

                                   

                                  Would you like to continue the discussion here or should we move to bugzilla? It's easier to share code there for example.

                                  • 14. Re: WCB - Physical or Virtual addressed?
                                    mwvantol

                                    I changed the title to reflect the current topic of the discussion.

                                    1 2 Previous Next