5 Replies Latest reply on Oct 22, 2012 4:29 AM by JanArneSobania

    Execute a priviledged Instruction on SCC

    devendra.rai

      Hello,

       

      I am testing RCCE/iRCCE and I would like to disable all interrupts. The instruction that I use is:

       

      __asm__ volatile("cli");

      __asm__ volatile("sti");

       

      However, I get a segfault (Exit Code 139), when the code executes this instruction.

       

      My intent is to be able to stop all interrupts (or mask them) in a critical region of the code. I am runing pre-emptible SCC-Linux (3.1.4).

       

      Any ideas on how to achieve the objective? Has anyone tried it? I am attaching  the source code which does this, if that would help.

       

      Thanks a lot.

       

      Devendra Rai

        • 1. Re: Execute a priviledged Instruction on SCC
          JanArneSobania

          Hi,

           

          RCCE applications run in user mode, on ring 3, as regular SCC Linux tasks. Privileged operations are not available to applications; only kernel code on ring 0 is allowed to use them.

           

          As an alternative, you could try to write a kernel module. These run in ring 0 and can use privileged instructions as they like, but must then bear the consequences. If you forget to re-enable interrupts, the core will no longer schedule tasks (ultimately, it will hang); wrong memory accesses can corrupt data or result in a panic, and so on. And I don't know whether RCCE supports such an environment.

           

          Why do you want to disable interrupts? If you let us know what you want to achieve, we may be able to suggest other solutions. Your problem may be better suited for bare metal programming, maybe using BareMichael, but its hard to suggest anything without knowing what you want to do.

           

          Regards,

          Jan-Arne

          1 of 1 people found this helpful
          • 2. Re: Execute a priviledged Instruction on SCC
            devendra.rai

            Hello Jan-Arne,

             

            My experiment requires consistent read/write timings (or MPI_Send/MPI_Recv) timings. I was noticing that this is not the case with RCKMPI, so I changed to RCCE. However, I still see that at random instances, the RCCE_Send/RCCE_Recv timings vary by as much as 50x of the normal. Some folks at Intel told me (and I also suspect) that this is due to interrupts. And, hence, the attempt to disable interrupts.

             

            Well, I understood that "sti/cli" are privileged instructions, and hence need to be dispatched from a kernel module. Other than the fact that I do not know how to build a kernel module (any helpful example?), I also doubt whether this scheme would work with rccerun, since kernel modules would have to be loaded into the kernel (modprobe, or something like that) before execution.

             

            So, I am really confused, if the kernel module technique will actually work. I understand that my best bet is baremetal. So, I believe, in baremetal, issuing

            __asm__ volatile("cli");

            __asm__ volatile("sti");

             

            would not be a problem. Correct?

             

            Thanks a lot.

             

            Devendra

            • 3. Re: Execute a priviledged Instruction on SCC
              JanArneSobania

              Hi Devendra,

               

              most device drivers included in the Linux kernel can be compiled as kernel modules, so there are many example. I would suggest to start with a simple one, like rckmem from SCC Linux. Please note that it is not compiled (or even compilable) as a module, but that should not be an issue; there are no source code modifications required, just a corresponding entry in Kconfig.

               

              You are right about rccerun, it won't work with a kernel module. I would expect the same for the rest of (the SCC Linux version of) RCCE, though, as it requires some user-mode support APIs. Although it should be possible to implement a kernel version, I think it has not been done yet. So your best bet is probably baremetal. If I remember correctly, there is a baremetal version of RCCE. Technically, CLI/STI should work there (if the program runs in kernel mode), but I have never tried and unfortunately cannot comment on it.

               

              Why do you need consistent timings? Is it a requirement of your algorithm? If not, it may be easier to just do all measurements using the existing RCCE infrastructure, and post-process/filter afterwards.

               

              Regards,

              Jan-Arne

              1 of 1 people found this helpful
              • 4. Re: Execute a priviledged Instruction on SCC
                devendra.rai

                Hello Jan-Arne

                 

                Thanks for your comments. You said:

                 

                "So your best bet is probably baremetal. If I remember correctly, there is a baremetal version of RCCE. Technically, CLI/STI should work there (if the program runs in kernel mode), but I have never tried and unfortunately cannot comment on it."

                 

                Does that mean that even on baremetal, it may not be possible to execute "sti/cli"? Then, how does one do it?

                 

                You are right: my algorithm requires very precisely determined timings for each read/write operation, since we are running a real-time application on SCC. I managed to port the application to baremetal, and it uses 11 cores. I will need to repeat my experiments with a total of three non-trivial examples. But, my algorithm provides certain timing guarantees, and these are possible if the reads/write timings are deterministic.

                 

                I know that I am stretching the capabilities of SCC a bit, but I think it is exciting that SCC can be used for time-sensitive non-trivial experiments as well.

                 

                Best,

                 

                Devendra

                • 5. Re: Execute a priviledged Instruction on SCC
                  JanArneSobania

                  Hi Devendra,

                   

                  I'm sorry, but I have never used a baremetal environment on the SCC and thus cannot say what will or will not be supported. For my main project RockyVisor (a distributed hypervisor for running a traditional SMP OS), all work is done under Linux.

                  I know that I am stretching the capabilities of SCC a bit, but I think it is exciting that SCC can be used for time-sensitive non-trivial experiments as well.

                  Sounds interesting. I can only recommend to try out baremetal, I would expect that it allows turning interrupts off.

                   

                  Regards,

                  Jan-Arne