13 Replies Latest reply on Oct 16, 2012 1:51 AM by JanArneSobania

    Problem using gdb and valgrind

    Nil

      Hi,

       

      I have created sccLinux image (for sccKit 1.4.2.2) with gdb and valgrind but when I try to use them (to debug multithreaded program) I am getting following errors.

       

       

      Valgrind complains about the line

      /* Flush MPBT from L1. */

      static inline void flush() { __asm__ volatile ( ".byte 0x0f; .byte 0x0a;\n" ); }

       

      GDB

      warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

       

      valgrind

      vex x86->IR: unhandled instruction bytes: 0xF 0xA 0x5D 0xC3

      ==415== valgrind: Unrecognised instruction at address 0x405c064.

       

      Any idea or solution for those errors?

       

      Thank you


        • 1. Re: Problem using gdb and valgrind
          JanArneSobania

          Hi Nil,

           

          the instruction valgrind complains about is SCC-specific, so it is no surprise that it doesn't know it. To get rid of the error, you would need to patch valgrind's instruction decoder to make it recognize the instruction (which, if I remember correctly, is officially called CL1INVMB).

           

          Unfortunately I have no experience with valgrind, so I cannot say how time-consuming this is. You may get away with just decoding the instruction as a NOP. However, if valgrind uses its IRs to generate new code, you will need to add more functionality to also record CL1INVMB in the IR and output the corresponding instruction (byte sequence 0x0F, 0x0A) when needed.

           

          Your gdb warning could be related to missing debug symbols, but I'm not sure. I found this bug report that sounds similar: https://bugs.meego.com/show_bug.cgi?id=3161. Can you try recompiling at least the C library with debug symbols, if they are not generated already?

           

          Regards,

          Jan-Arne

          • 2. Re: Problem using gdb and valgrind
            Nil

            JanArneSobania wrote:

             

            Your gdb warning could be related to missing debug symbols, but I'm not sure. I found this bug report that sounds similar: https://bugs.meego.com/show_bug.cgi?id=3161. Can you try recompiling at least the C library with debug symbols, if they are not generated already?

             

            Thanks for quick reply, i just used "make menuconfig" in sccLinux to add gdb so in that where do i find the C library that you mentioned. is there a way to make C library compile with debug symbols from "make menuconfig" or not?

             

            Thank you

            • 3. Re: Problem using gdb and valgrind
              JanArneSobania
              Thanks for quick reply, i just used "make menuconfig" in sccLinux to add gdb so in that where do i find the C library that you mentioned. is there a way to make C library compile with debug symbols from "make menuconfig" or not?

              You can try the following (untested):

              "make ctng-menuconfig", select "C-library", and set "extra target CFLAGS" to "-g" (without quotes). Then, go one screen back (to the main selection menu), choose "Target Options", and set "Target LDFLAGS" to "-g" (again, without quotes). Recompile. You may want to check with objdump afterwards whether the binaries contain debug information.

               

              Setting CFLAGS to "-g" should compile with debug options, but I think ld removes them unless you link with "-g" as well.

              • 4. Re: Problem using gdb and valgrind
                Nil

                Hi,

                 

                i am getting the error when i run "make ctng-menuconfig" but "make menuconfig" works perfectly any idea?

                 

                $ make ctng-menuconfig

                  CONF  config/config.in

                #

                # configuration saved

                #

                  CONF  config/config.in

                #

                # configuration saved

                #

                *** Unable to find the ncurses libraries or the

                *** required header files.

                *** 'make menuconfig' requires the ncurses libraries.

                ***

                *** Install ncurses (ncurses-devel) and try again.

                ***

                make: *** [ctng-menuconfig] Error 2

                 

                Thank you

                • 5. Re: Problem using gdb and valgrind
                  JanArneSobania

                  Hi,

                   

                  this error looks familiar. Can you please verify that you do not have the LD_PRELOAD environment variable set? If this does not help, please post the output of "set" (shows all environment variables).

                   

                  Regards,

                  Jan-Arne

                  • 6. Re: Problem using gdb and valgrind
                    Nil

                    Hi,

                    this error looks familiar. Can you please verify that you do not have the LD_PRELOAD environment variable set? If this does not help, please post the output of "set" (shows all environment variables).

                    this error was due to my crosscompiler being in my path which has been resolved now.

                     

                    "make ctng-menuconfig", select "C-library", and set "extra target CFLAGS" to "-g" (without quotes). Then, go one screen back (to the main selection menu), choose "Target Options", and set "Target LDFLAGS" to "-g" (again, without quotes). Recompile. You may want to check with objdump afterwards whether the binaries contain debug information.

                    I have provided -g to those options you mentioned but the gdb warning is still there.

                     

                    Setting CFLAGS to "-g" should compile with debug options, but I think ld removes them unless you link with "-g" as well

                    where do I set this?

                     

                    another thing is there a way to specify that i want latest versions of companion tool to be installed in crosstool-ng? (i.e. make, libtool ...)

                     

                     

                    Thank you

                    • 7. Re: Problem using gdb and valgrind
                      Nil

                      this error looks familiar. Can you please verify that you do not have the LD_PRELOAD environment variable set? If this does not help, please post the output of "set" (shows all environment variables).

                      The error about ncurses have returned and it seems that solution in my previous post was not correct.

                      I can confirm that i do not have LD_PRELOAD set. I have attached the out put of "set".

                       

                      Thank you.

                      • 8. Re: Problem using gdb and valgrind
                        devendra.rai

                        Hello Nil,

                         

                        I have been through the frustration that you are feeling. I put in lot of effort to build a custom kernel with gdb and valgrind and what not.


                        However, even with debugger tools available (tested gdb), debugging is not easy on SCC, since the RCKMPI will probably mix messages from various sources, and your gdb prompt will probably be garbled.

                         

                        So, I chose to do all development on a desktop, where I have more control, and assume RCKMPI is version equivalent to OpenMPI version <1.0. That means, I use only the most basic MPI calls, MPI_Send, MPI_Recv, and things have worked out fine.

                         

                        I am edgy about using anything more.

                         

                        I had a few questions about RCKMPI's logging facilities, which remain unanswered. However, I conclude that even the logging facilities of RCKMPI are rudimentary.

                         

                        I certainly do not want to discourage you from doing what you plan to do, but you should be prepared for the problems that come next.

                         

                        Drop me a line on my email if you'd like to talk.

                         

                        Devendra

                        • 9. Re: Problem using gdb and valgrind
                          Nil

                          Hi Devendra,


                          So have you managed to build the kernel with GDB? If yes it would be very helpful if you can provide me the details of sort of stapes that you took to build it. In my case I am not using MPI. I have had no problem building kernel with GDB but I face problems as it seems it have problems with some libraries that is needed to do multi-threaded debugging.


                          I will continue trying to get GDB working with support for multi-threaded debugging.

                           

                          Thank you.

                          • 10. Re: Problem using gdb and valgrind
                            mwaughex

                            Hi Nil,

                            Did you also have a look at bug 436?

                            http://marcbug.scc-dc.com/bugzilla3/show_bug.cgi?id=436

                            -augie

                            • 11. Re: Problem using gdb and valgrind
                              devendra.rai

                              Hello,

                               

                              As far as I remember, this should have been resolved. I allowed Nil an access to my SCCLinux build, which contains gdb and valgrind. I had copied Mark too.

                               

                              Devendra

                              • 12. Re: Problem using gdb and valgrind
                                Nil

                                Hi,

                                mwaughex wrote:

                                 

                                Hi Nil,

                                Did you also have a look at bug 436?

                                http://marcbug.scc-dc.com/bugzilla3/show_bug.cgi?id=436

                                -augie

                                I have looked at all the relavent bugs that i can find including 436.

                                 

                                Devendra Rai wrote:

                                 

                                Hello,

                                 

                                As far as I remember, this should have been resolved. I allowed Nil an access to my SCCLinux build, which contains gdb and valgrind. I had copied Mark too.

                                 

                                Devendra

                                Sorry Devendra I was away for some time and i did not get chance to get access to your copy of sccLinux.

                                 

                                 

                                Overall, The problem i had was not that i can not build sccLinux with GDB but i was unable to use gdb to debug multithreaded program. After some diagnosis i have found a solution. The problem is that libthread_db is does not get copied on to sccLinux. Accrding to http://comments.gmane.org/gmane.comp.lib.uclibc.buildroot/46287 we need to apply patch that enbles copying of libthread_db onto sccLinux  in order to enables thread debugging with full gdb on sccLinux.i can confirm that after patch libthread_db gets copied to sccLinux. I have attached the patch with the post. Now the only problem remains is how to exclude libpthread from getting stripped. As if libpthread is stripped then it is no use for debugging.

                                 

                                It may take some time as i not too familiar with buildroot. I have found out that buildroot 2012.08 has the option to exculde certain libraries from getting stripped. @JanArneSobania is it worth to move to buildroot 2012.08? In current buildroot 2011.11 the only option i can see is to stripped all or none, with none option sccLinux image is around 90MB and GDB works fine. Is it buried inside some menu am i missing?

                                 

                                I will keep posted if i can find any way to exclude certain lib from being tripped (libpthread), any insight or pointer will be helpfull.

                                 

                                Nil

                                • 13. Re: Problem using gdb and valgrind
                                  JanArneSobania

                                  Hi Nil,

                                   

                                  I am not the owner of the SCC buildroot, so I cannot say for sure. However, my feeling is that, given the current state of both the toolchain and SCC Linux, it would be hard to convince anyone to spend the effort of moving to a newer version. That is, unless there were serious problems with the old one, which doesn't seem to be the case. I'm sorry, but in my opinion, a patch to the current toolchain would be more suitable. So if you find a way to exclude certain files from being stripped, please let us know.

                                   

                                  The CTNG option you mentioned for controlling stripping maps to the CT_STRIP_ALL_TOOLCHAIN_EXECUTABLES configuration variable, which seems to be used only in the file ./output/build/host-crosstool-ng-1.13.2/scripts/build/internal.sh. I think the actual (crosscompiled) strip utility is being invoked by the line "CT_DoExecLog ALL $(CT_HOST)-strip $(strip_args) "${_t}"". So as an experiment, I would start by trying to not perform this command if ${_t} contains one of the file names to be excluded.

                                   

                                  Regards,

                                  Jan-Arne