8 Replies Latest reply on Oct 17, 2010 8:05 PM by tedk

    modifying lut_init.vi

    junghyun

      According to the "How to Hijack Memory" document, I can shrink or enlarge the shared memory area.

       

      However, the file "/tmp/sccKit_yourname/obj/lut_init.vi" is changed automatically when I execute "sccBoot" with different argument.

       

      I'm investigating which is the best place to modify, but I think Ted can know better solution for this.

       

      Could you tell me where is the original part of lut_init.vi?

        • 1. Re: modifying lut_init.vi
          tedk

          The file lut_init.vi is not intended to be changed manually. You won't see any effect by editing it. We refer to it because it is a good record of what the default LUT settings are.

           

          It is true that sccBoot reloads LUTs but I have not seen it modify unused LUT entries. Please check this yourself to be sure. I think the best way to modify LUT entries is in code that you run on the cores. In RCCE, we do it inside RCCE_Init(). The updates persist when the RCCE program completes, but it doesn't hurt to set them again.You can check current LUT settings with sccDump or the sccGui memory widgit.

           

          If you cycle the power (which you cannot do as a data center user ... you can request that we do it if you hang), the modified LUT entries go away.

           

          Please also note that hijacking is a temproary solution intended only for use during the interim when we prepare a better solution. Currently, this unshared memory is uncached so do not expect great performance; it is intended for proof of concept of your algorithms. And hopefully the interim time for hijacking will be short. Also hijacking is in the RCCE trunk ... not tagged as a release. It is the latest ... not necessarily the greatest, so please be cautious.

          • 2. Re: modifying lut_init.vi
            junghyun

            This is what I found so far.

             

            lut_init.vi is made by sccMerge which is perl script file.

             

            I changed this file, so I can get different lut_init.vi file.

             

            "sccBoot" uses sccMerge internally, but it does not use PATH.

             

            I'm not sure, but the source code of sccBoot sets exact execution path of sccMerge.

             

            I'm looking into sccBoot source code in order to run the modified sccMerge file.

            • 3. Re: modifying lut_init.vi
              tedk

              What is it you want to do?  The easiest way to modify the default LUT values is to edit them after Linux boots with your own program. This is what I have done.

               

              Your question leads me to believe that you want to have a new set of LUTs at boot time before running one of your own programs. This very well may be what your research requires, but I'd first ask you to evaluate if modifying the LUTs after boot time would work for you.

               

              Modifying LUTs before booting is not something I've personally done, but it has been done by people listening on this forum. So please be prepared for a possible correction.

               

              I don't think you will achieve a new set of LUT values by editing lut_init.vi. The actual file used to load the LUTs is I believe lut_init.dat. Although this file is ASCII, exactly how to edit the bits to do what you want is not apparent. I think you need to be able to make a lut_init.dat from the the lut_init.vi. The file lut_init.vi is also ASCII; you can read lut_init.vi in a text editor and see how the LUTs are configured.

               

              The sccKit tools like sccMerge and sccBoot are of course all opensource and you can modify them for your needs.  When you do, you have the option of keeping your modifications private or sharing them with the community.

              • 4. Re: modifying lut_init.vi
                tedk

                I should clarify what I said about sharing code.

                 

                I would certainly encourage you to share your code with the community, but that is optional on your part. However, your agreement with Intel does require that you share your modifications to any Intel-provided code with Intel at the conclusion of your research.

                • 5. Re: modifying lut_init.vi
                  junghyun

                  Ok, I got your point. I can share my modifying code if somebody wants.

                   

                  sccMerge code is too complex to paste in this page.

                   

                  I don't know I can upload a file to share it, if it is possible please let me know.

                   

                   

                  Anyway, why I modify sccMerge is that you made a document "How to Hijack share memory with RCCE".

                   

                  What I got from the document is modifying "lut_init.vi" to make shared memory region larger.

                   

                  So, I was looking into how to modify lut_init.vi.

                   

                  However, it is generated by sccMerge, so I thought that modifying sccMerge is the best option.

                   

                  Of course, SCC linux uses only 320MB, above the memory I can make it shared.

                   

                  Due to the limited slot of LUT, I just hijack only two slots of each core.

                   

                  It makes shared memory of 2*48*16M = 1536MB which is enough for me.

                   

                   

                  I cannot tell you what I want to do because it is just beginning of my project and I don't know I will use what I asked to you.

                   

                  Anyway, if my project is successful, I will share my project contents.

                  • 6. Re: modifying lut_init.vi
                    tedk

                    I'm sorry if the "How to Hijack ..." document was confusing. But there is no intention in that document to suggest that you modify lut_init.vi. That file is shown only as an example of how the default LUT configuration looks. The document suggests that you modify the LUT entries with code that runs after you boot Linux. The mechanism for that modification is memory-mapped I/O.

                     

                    Each additional LUT entry gives you 16MB of memory. And you add them in groups of four, one for each memory controller. An additional four LUT entries gives you 64MB. This is shared memory so you don't get additional memory from each core. You  should not multiply by 48. The maximum amount of shared memory is about 1GB. As a maximum you can use 60 new LUT slots which is 60*16MB = 960MB. And you get this if you do not use the slots 0x80 thru 0x83. The slots 0x80 through 0x83 would give an additional 64 MB, but some of that memory is used by SCC Linux itself. And what we've tried out here is starting the shared memory at 0x84 to avoid the shared memory used by the system.

                     

                    I agree that the forum is an inappropriate place to post other than a code snippet. What we are discussing is the possbility of having a branch in our public SVN so that users can check in code there. But we're still discussing how users might make their code available  and haven't set up such a branch. The University of Texas does have its own SVN directory in our repository. This is not a branch but rather a brand new directory that contains their original contribution, not a modification of Intel code.

                     

                    You've had some excellent questions about the SCC cache behavior and as a result of your posts, we'll try to clarify the description of SCC cache behavior. Thank you for your input.

                    • 7. Re: modifying lut_init.vi
                      junghyun

                      Hmm... then, I got another question.

                       

                      Linux kernel should know how much available physical memory it has. If I just use the default LUT table and modify it in runtime, it is ok to the kernel? For example, the default 32GB LUT setup from EAS tells 0x00 through 0x40 is mapped for private data. I don't know how to do it, but somehow the kernel saves some information of this private memory area in its data structure. Then, if I hijack last entry 0x28 from each core and make it shared, is there no problem in the kernel? If I hijack one entry from each core, I can get 48*16MB= 768MB of shared memory space. please tell me whether this approach is appropriate or not.

                       

                      The empty hole between shared area and MPB is 60*16MB=960MB as you mentioned, but the empty hole between shared area and private area is 87*16MB=1392MB. I want to use large shared memory area as much as possible. Is it okay to use the second area?

                       

                      By the way, I just poped up another question. Why the shared MCH0-3 is 8MB? The remained area was 128MB per controller for 32GB of system memory. EAS says 128MB is left per controller and one slot i sued for shared memory on each controller. "So there is unused memory available". But it is not true, 128MB per controller is left, used only 16MB. Then, 112MB is left per controller!!

                       

                       

                      So, in summary I asked three questions.

                       

                      1) Is it ok to modify the default LUT mapping after boot?

                       

                      2) Can I use the hole between private area and shared area in LUT table?

                       

                      3) Why EAS says there is unused memory available and why shared MCH0-3 is 8MB?

                       

                       

                      You've had excellent and fast answers for every questions. Thank you very much for your response. It was very helpful to dig SCC system.

                      • 8. Re: modifying lut_init.vi
                        tedk

                        Yes, it is OK to modify the LUTs after boot. That is what we do ourselves.

                         

                        If you find an area of memory that is unused ...yes, you can use it for shared memory. The issue is not so much the unused LUT slot itself but the memory it points to. In the "How to Hijack ..." document we describe how to hijack memory using RCCE. This is not the same method you would use if you were not using RCCE.

                         

                        I should emphasize that this is a temporary method. A future version of SCC Linux will have a better way of increasing shared memory.

                         

                        The LUT tables in the EAS are just examples. The basis of the example was to give each core as much private memory as possible and then to assign what is left over to shared memory. However, with the current SCC Linux we are getting 16MB per shared LUT slot. And the default amount of shared memory is 64MB. But as mentioned before you must be wary of the areas used by the system.

                         

                        The way to find out what is really in your configuration is to look at lut_init.vi. Here you can see that each core uses 40 LUT slots, not the 85 that the EAS mentions. The EAS is just using a different example.

                         

                        Another way to see what LUT slots and private memory your system is using is with the sccDump command. If you type sccDump -c 00, you will get the LUT values for Tile #00.

                         

                        I attached a file called coreLUTs.pdf. You can see there that each core is using 40 LUT slots and that the memroy addresses are contiguous from core to core. As in ...

                         

                        core #0 of Tile x=0, y=0
                        {9'h000,3'b110,10'h000}; // 0x00 (000): Route to private space: Code and data section...
                        :
                        {9'h000,3'b110,10'h028}; // 0x28 (040): Route to private space: Code and data section...
                        core #1 of Tile x=0, y=0
                        {9'h000,3'b110,10'h029}; // 0x00 (000): Route to private space: Code and data section...
                        :
                        {9'h000,3'b110,10'h051}; // 0x28 (040): Route to private space: Code and data section...

                         

                        Here core 0 ends with a prepend of 0x28 and core 1 begins with a prepend of 0x29. Now we know that each core doesn't even need all the private memory that it is assigned. So yes, you can take private memory from each core and yes, get more that way. But it won't be contiguous memory that way.

                         

                        When we hijacked memory for RCCE, we wanted the memory to be contiguous, so we hijacked from core 1 of Tile x=5, y=3. We grabbed the prepend of 0x1eb, then 0x1ea, etc. And then we run out of LUT slots before we run out of memory that we can take.

                         

                        Now you could investigate using the unused private memory of other cores as well as the slots that were assigned to that memory. Your shared memory would not then be contiguous, but possibly that is OK for what you want.