1 2 3 4 5 Previous Next 62 Replies Latest reply on Jun 1, 2012 3:14 AM by christofi.c Go to original post
      • 15. Re: MPI on the SCC
        compres

        It is based on MPICH2 version 1.2.1p1. 

         

        I would expect it to be just as compliant.  There is, however, no support for dynamic processes, just like the dcmfd for the blue genes, so I don't expect this to be a problem to most people.  I would like to hear if someone is interested in dynamic processes support, since adding this is not trivial.

         

        It would be nice if someone runs a compliance tests and post the results.

         

        - Isaías

         

        PS:  I just found out spawn/spawn_multiple fail silently.  I will patch this so you may want to do an SVN update later during the day.

        • 16. Re: MPI on the SCC
          jeff.science@gmail.com

          That is useful information.  Blue Gene/P is my primary system, so I've never known a world where MPI dynamic processes were critical :-)

           

          If it helps, the only MPI features I care about are collectives, passive-target RMA, and, to a lesser extent,  send/recv and datatypes.

           

          There is a fairly major bug in MPICH2 RMA that was fixed very recently by Bill Gropp related to the number of messages one can enqueue in a lock-unlock epoch.  It is important to backport this fix in order for RMA to work properly for my use case.

           

          Thanks for your help.


          Jeff

          • 17. Re: MPI on the SCC
            compres

            Hello Jeff Hammond,

             

            The RCKMPI library consists of 3 new SCC specific channels, and some minor patches to MPICH2.  It is easier to port these channels to a newer version of MPICH2, than to backport changes.  I checked the channel API and it remains unchanged.

             

            If you know the specific version where these bug fixes have been done, it would be nice to know.  I already have an experimental tree based on the 1.3.2p1 release, that may contain the fixes. 

             

            Perhaps you cound do a test run to see if you have any issues with regards to data types and your RMA cases.  I have not run into any issues while testing collectives and RMA communication.

             

            - Isaías

            • 18. Re: MPI on the SCC
              jeff.science@gmail.com

              Unfortunately, I do not have any SCC hardware, so I can't run any tests of this.  I would love for this situation to change.  I have an offline dialog with the right people about this already.

               

              I'll have to check on the MPICH2 RMA bug fix details.  I can't find it in the bug tracker.  I think it was fixed a few months ago though, so it might be in 1.3.2p1.  I'll check with the MPICH guys when I get to the office and report back with a revision number.

              • 19. Re: MPI on the SCC
                cloud2ground

                Ted, yes I would be interested in exploring MPI on SCC.  Best, John

                • 20. Re: MPI on the SCC
                  devendra.rai

                  So, the state of the art is still that we do not have full support for MPI.

                   

                  Now, there is another question: I see that Intel has some implementation of MPI, the so called RCKMPI. However, this is just the light weight version of MPI (OpenMPI-1.4.2).

                   

                  We have an application that we are running on our own clusters, using OpenMPI-1.4.2, and we would like to see the performance on Intel SCC.

                   

                  It turns out that doing this currently is an ugly mess, and we are not interested in backporting our application, since we believe that multiprocessing machines, under our control will give us what we need.

                   

                  In short, we use an (unholy) mix of threads, threaded-MPI calls, c++0x standard, containers.

                   

                  I thought that the cleanest way was to compile our own kernel to run on SCC.

                   

                  I do not think that I am the first person to come up with this idea. Does anyone have any experience in compiling kernels (that include support for the features that we need)? Or, would Intel graciously help ?

                   

                  Thanks a lot!

                   

                  Best,

                   

                  Devendra Rai

                  • 21. Re: MPI on the SCC
                    tedk

                    Lots of people have built custom kernels for the SCC. The very latest SCC Linux code is now on github. It works with sccKit 1.4.2. You can load it with 1.4.1.3 but if I do that I lose connectivity to the cores. It's been working fine with the 1.4.2 beta. What I've done is just build the default SCC Linux from its sources. And for a custom SCC Linux, I think that is the first step. If you can do that, then your build environment is set up correctly. Once you build the default SCC Linux, you can make a modification, and it then of course builds much faster.

                     

                    You don't have to have a github account to clone and build. The github organization for SCC Linux is hpi-scc. If you want to suggest that your modifications get incorporated into the default build, then you should make a github account and fork SCC Linux and then make a pull request. But you probably should discuss the proposed modification on the Forum or in Bugzilla first.

                     

                    There is a new version of RCK MPI that I have not yet seen it, but will shortly. The download package will include Isaias' Master Thesis. You can listen to his talk at HPI’s site for videos of the talks http://tele-task.de/archive/series/overview/877/ I had difficulty with this file under Windows but not Linux. I don't know if this is due to Windows or to the fact that my Windows system is inside the Intel firewall and my Linux system is not. In any case, there have been significant improvements in RCK MPI that we should all look at.

                    • 22. Re: MPI on the SCC
                      christgau
                      So, the state of the art is still that we do not have full support for MPI.

                      That's right. The current version of RCKMPI works with fine with Pt2Pt messages. We successfully ported some of our MPI in-house applications as well as real scientific (climate simulation) to the SCC without severe problems.

                       

                      RCKMPI lacks support for dynamic process creation, but Isaias fixed that in the latest version, as Ted already hinted.  Follow the link in his last post to watch Isaias talk from the last MARC symposium where he presented some deteils.

                       

                      Additionally we currently found out, that RCKMPI does not fully support one-sided communication (or remote memory access - RMA -  like it is called in the MPI implementation). You can check this by running the MPICH2 test cases.

                       

                      So there are some tasks left open to get more features of MPI to work on the SCC.

                       

                      I see that Intel has some implementation of MPI, the so called RCKMPI. However, this is just the light weight version of MPI (OpenMPI-1.4.2).

                      RCKMPI is based on MPICH2 1.2.1p, which was very probably the most recent version when developpement started.

                       

                      Regards, Steffen

                      • 23. Re: MPI on the SCC
                        tedk

                        There's a new version of RCKMPI that we just uploaded to the public SVN today.

                        http://marcbug.scc-dc.com/svn/repository/trunk/rckmpi2/

                         

                        We haven't had a chance to use it much yet. If you try it out, can you post your experiences here?

                        • 24. Re: MPI on the SCC
                          compres

                          christgau wrote:

                           

                          So, the state of the art is still that we do not have full support for MPI.

                          That's right. The current version of RCKMPI works with fine with Pt2Pt messages. We successfully ported some of our MPI in-house applications as well as real scientific (climate simulation) to the SCC without severe problems.

                           

                          RCKMPI lacks support for dynamic process creation, but Isaias fixed that in the latest version, as Ted already hinted.  Follow the link in his last post to watch Isaias talk from the last MARC symposium where he presented some deteils.

                           

                          Additionally we currently found out, that RCKMPI does not fully support one-sided communication (or remote memory access - RMA -  like it is called in the MPI implementation). You can check this by running the MPICH2 test cases.

                           

                          So there are some tasks left open to get more features of MPI to work on the SCC.

                           

                          I see that Intel has some implementation of MPI, the so called RCKMPI. However, this is just the light weight version of MPI (OpenMPI-1.4.2).

                          RCKMPI is based on MPICH2 1.2.1p, which was very probably the most recent version when developpement started.

                           

                          Regards, Steffen

                           

                          Hello Steffen,

                           

                          it should not fail One-sided tests.  Could you tell which test or tests failed from the test suite?

                           

                          The MPICH2 version is indeed old, but porting the channels to the newer versions is trivial since the channel interface has not changed since 1.2.1p1.

                           

                          - Isaías

                          • 25. Re: MPI on the SCC
                            compres

                            Ted Kubaska wrote:

                             

                            There's a new version of RCKMPI that we just uploaded to the public SVN today.

                            http://marcbug.scc-dc.com/svn/repository/trunk/rckmpi2/

                             

                            We haven't had a chance to use it much yet. If you try it out, can you post your experiences here?

                             

                            Hi Ted,

                             

                            to run it a patched rckmb for 2.6.38 is necessary, as well as the inter-core signaling driver (SIGMOD).  It would be nice to have a pre-built and patched image somewhere, together with the driver as a module, and then put some basic instructions (similarly to the quick guide for the first release).

                             

                            - Isaías

                            • 26. Re: MPI on the SCC
                              christgau

                              In addition to Isaias:

                               

                              The basic steps to build the new RCKMPI are much like in the original version: svn checkout, configure, compile, install

                               

                              Here's what I did (we have a common directory under /shared on our DC system for stuff that every user might use)

                               

                              export RCKMPI_SRC=$HOME/src/rckmpi/version2/

                              export RCKMPI_DST=/shared/common/rckmpi/version2

                               

                              svn co http://marcbug.scc-dc.com/svn/repository/trunk/rckmpi2 $RCKMPI_SRC

                              cd $RCKMPI_SRC

                              configure --prefix=$RCKMPI_DST --with-device=invasive --with-pm=invasive

                              make # or make -j 9 for faster build

                              make install

                               

                              after a successful build

                               

                              $RCKMPI_DST/bin/mpich2version should give you information about the fresh build including the ./configure command line.

                               

                              Some useful configure options are

                              --enable-g=dbg = enable debug compile option (nice, if you want to debug the mpi implementation)

                              --disable-fast = disable optimization (nice, if you want to debug the mpi implementation)

                              --enable-fc = enable Fortran support (required the cross compiler built with Fortran support)

                               

                              Note: --enable-g=all won't work, because the SCCMPB device uses malloc which seems to be overwritten by MPICH2 with this setting (you will get warnings like "character constant too long for its type" and error about unexpected tokens) Further --enable-g with "log" won't work as well because there's function missing. But these options are only required if you want to dive into the implementation and debug it.

                              • 27. Re: MPI on the SCC
                                christgau
                                it should not fail One-sided tests.  Could you tell which test or tests failed from the test suite?

                                Hi Isaias,

                                 

                                my experienced are based on the old RCKMPI, but the current SVN revision 301 (http://marcbug.scc-dc.com/svn/repository/trunk/rckmpi, latest change was in rev 211)

                                 

                                I went to test/mpi/rma, compiled all testcases and put them on the SCC. I wrote a little shell script to iterate over the lines in the testlist file and tried to ran every test with the given number of processes. The first two (winname and allocmen) succeed, but they are pretty simple and not any one-sided communication at all. But putfence is the first which not really fails but in fact does never return. I logged in to the participating cores and discovered that the test case consumes 100% CPU. Logging the MPI calls revealed that the progress engine is continously polling the MPB. That looks like MPICH is waiting for a message that never appears. Anyway I made some progress on this, I will open a bug and share what I've found out.

                                • 28. Re: MPI on the SCC
                                  compres

                                  christgau wrote:

                                   

                                  it should not fail One-sided tests.  Could you tell which test or tests failed from the test suite?

                                  Hi Isaias,

                                   

                                  my experienced are based on the old RCKMPI, but the current SVN revision 301 (http://marcbug.scc-dc.com/svn/repository/trunk/rckmpi, latest change was in rev 211)

                                   

                                  I went to test/mpi/rma, compiled all testcases and put them on the SCC. I wrote a little shell script to iterate over the lines in the testlist file and tried to ran every test with the given number of processes. The first two (winname and allocmen) succeed, but they are pretty simple and not any one-sided communication at all. But putfence is the first which not really fails but in fact does never return. I logged in to the participating cores and discovered that the test case consumes 100% CPU. Logging the MPI calls revealed that the progress engine is continously polling the MPB. That looks like MPICH is waiting for a message that never appears. Anyway I made some progress on this, I will open a bug and share what I've found out.

                                   

                                  Thanks.

                                   

                                  I think that part of the tree has not changed since that revision, so the bug must still be there.  The typical point of failure is in the metadata updates, so I think the first thing to look out for is the order in which the stores are done in the flag areas.  If that happened to you in separate SCC systems it is deffinetly a bug.

                                   

                                  - Isaías

                                  • 29. Re: MPI on the SCC
                                    christgau

                                    i created a bug considering the OSC/RMA issues with some details I discovered up to now:

                                     

                                    http://marcbug.scc-dc.com/bugzilla3/show_bug.cgi?id=386