10 Replies Latest reply on Aug 5, 2015 1:18 PM by AlexT_Intel

    Image built from IOTDK source won't launch gdbserver properly

    xthunderheartx

      Using the instructions in Brendan's blog here: Creating a Yocto image for the Intel® Galileo board using split layers | Intel® Developer Zone, I successfully built and deployed an image.  However, for some reason, the Eclipse DSF automatic launcher hangs while trying to initialize a debug session.  It's a bit weird because the executable gets downloaded via sftp fine, but then the process hangs and never manages to execute the chmod 755 to make the executable actually runnable by linux.  Normally you'll see this output in the eclipse remote shell console:

       

      echo $PWD'>'~# echo $PWD'>'

      /home/root>

      root@galileo:~# chmod 755 /tmp/mdmctrld;gdbserver :2345 /tmp/mdmctrld;exit

      Process /tmp/mdmctrld created; pid = 265

      Listening on port 2345

      Remote debugging from host 192.168.0.222

       

      However, on the broken image it never even gets to the point where it does "echo $PWD'>'".  You just get the "root@galileo" shell prompt, the cursor return left (no linefeed) and blink.  Nothing further.  Crickets

       

      I looked at the processes running on the target when this happens and noticed two ssh login sessions with 2 corresponding shell instances, all apparently suspended:

       

        210 root      5380 S    sshd: root@pts/0   <-------------------------------------------This is the session from my dev system

        217 root      3424 S    -sh   <-------------------------------------------------------------- Ditto

        234 root      5760 S    sshd: root@pts/1,pts/2,pts/3  <----------------------------- Eclipse ?? Gotta be

        236 root      2284 S    /usr/lib/openssh/sftp-server

        237 root      2284 S    /usr/lib/openssh/sftp-server

        238 root      3408 S    -sh   <-------------------------------------------------------------  ditto

        243 root      3408 S    -sh   <-------------------------------------------------------------

        252 root         0 SW   [kworker/0:1]

        255 root      3408 S    -sh   <-------------------------------------------------------------

        257 root      2764 R    ps

       

       

      Man I'm in the weeds.  Any suggestions?

        • 1. Re: Image built from IOTDK source won't launch gdbserver properly
          PabloM_Intel

          Hi xthunderheartx,


          Let me investigate this a little bit more... Could you please tell me which version of Eclipse are you using? Also, is your code running successfully? Is this happening with every code that you upload to the board?

           

          Regards,

          PabloM_Intel

          • 2. Re: Image built from IOTDK source won't launch gdbserver properly
            xthunderheartx

            Hey PabloM_Intel,

             

            I'm using Eclipse from the most recent release of the IoTDK which is luna.  I think it is just stock right out of the iotdk archive, but I can send you the list of installed software from the "About Eclipse Platform" menu item if that would be helpful.  I don't remember installing any additional plugins or whatnot.

             

            My code loads and runs fine if I use the stock iotdk kernel image in the target.  And yes it happens with every thing I try.  Actually, I didn't notice the problem for about a week because I was working on adding stuff to the kernel that we will need for our release this fall.  So I wasn't actually building/testing code.  When I deployed the kernel to other developers, they instantly started complaining.  At first I thought it was something I had broken with updated packages, but I went back to a pristine load, right out of git, no changes, and I had the same issue.  But again, if I deploy the stock iotdk image all is in order.

             

            I retained each image that I built so I could share any or all of those with you as well. Anything I can do to help I'll do.  We really, really appreciate it!

             

            Dallas

            • 3. Re: Image built from IOTDK source won't launch gdbserver properly
              xthunderheartx

              Pablo I forgot to mention that I run Ubuntu 14.04 (specifically kubuntu) on my development box (where eclipse lives) and I bitbake the images in a Ubuntu 12.10 Virtualbox VM running on the same machine.  I doubt that matters though 12.10 isn't an approved version according to Yocto.  Anyway just wanted to be as precise as possible.  Thx again - Dallas

              • 4. Re: Image built from IOTDK source won't launch gdbserver properly
                xthunderheartx

                I repeated the experiment using a Kunbuntu 14.04 machine running natively on the hardware and got the same result.  Can't blame this on the build environment I don't think.  Maybe Brendan left out something?  I do get a couple of warning in the 14.04 environment about something needing bash and no RDEPENDS or something of that nature.  Also the standard fetch failures that are recovered from mirrors.  Couple of checksum failures. But they all seem to recover all the tasks complete with no errors.

                 

                Any word on this PabloM_Intel?

                • 6. Re: Image built from IOTDK source won't launch gdbserver properly
                  AlexT_Intel

                  I'm finally getting to this, the build has started in some five hours should be done, so realistically hopefully tomorrow in the evening I should be able to test it. I'll post here when I have something tangible.

                  • 8. Re: Image built from IOTDK source won't launch gdbserver properly
                    AlexT_Intel

                    In short - it works for me just fine. Both on Windows (7 x64 on a physical host) and Linux (Ubuntu 12.04 x86_64 on a VM).

                     

                    I've built the image following instructions here: https://software.intel.com/en-us/blogs/2015/03/04/creating-a-yocto-image-for-the-intel-galileo-board-using-split-layers

                    Installed Eclipse using instructions here: https://software.intel.com/en-us/articles/install-eclipse-ide-on-intel-iot-platforms

                     

                    Created a test Blink C project, ran it (entering the connection properties - I used the IP of my board and empty password as set by default). Blinks fine, then I ran the Debug configuration - it switched to a debugger prospective without any trouble on both OSes and has shown me debug contexts I expected.

                     

                    The remote shell console has this for me (no "echo $PWD" you mentioned):

                    root@galileo:~# chmod 755 /tmp/testprojlin;gdbserver :2345 /tmp/testprojlin;exit root@galileo:~# chmod 755 /tmp/testprojlin;gdbserver :2345 /tmp/testprojlin;exit

                    Process /tmp/testprojlin created; pid = 426

                    Listening on port 2345

                    Remote debugging from host 192.168.1.5

                     

                    So whatever it is, it's unlikely the image is the reason here.

                     

                    Did you see the "remote host authenticity <blah>" SSH window right after initiating the connection for the first time (step #10 here)? If it's somehow hidden then it might looks exactly like you describe - seemingly stalled SSH connection and no communication with the target (because the connection was not yet fully initiated). Overall I'd suggest you to rather dig into the potential connection problems between your Eclipse host and target (like SSH settings, firewall, etc) - that looks like the most plausible reason to be right now.

                    • 9. Re: Image built from IOTDK source won't launch gdbserver properly
                      xthunderheartx

                      Did you see the "remote host authenticity <blah>" SSH window right after initiating the connection for the first time (step #10 here)? If it's somehow hidden then it might looks exactly like you describe - seemingly stalled SSH connection and no communication with the target (because the connection was not yet fully initiated). Overall I'd suggest you to rather dig into the potential connection problems between your Eclipse host and target (like SSH settings, firewall, etc) - that looks like the most plausible reason to be right now.

                       

                      I'll try one of the images again but I don't remember the RSA dialog possibly because it is so ubiquitous.  I regularly debug on multiple remote targets scattered across two physically and logically separate networks.  Anytime the signature changes (which is fairly often) the dialog you mentioned comes up and I simply click yes without thought, as I know and trust the network (I built it after all). The thing is, the same hardware should behaved identically with the both images since nothing other than the image actually changes.  The IP is always the same because my router (pfsense) statically maps the IP to the board's MAC.  I never (well almost never) set a password for the root account on the target so that isn't an issue.  So I should be able to pop a default image (downloaded from intel) into the card. then debug blinky or whatever.  Then, pop that one out, pop in an image built using Brendan's instructions, and debug the same program without changing anything, though of course I should get the RSA Signature Changed <blah> the first time.  Other than that it should just work.  I should see the exact same dialog on the remote shell window.  I of course tried the image on multiple targets, including one located at the other end of an IPsec tunnel in another city with the exact same behavior before posting originally.  Also, images built using the standard build guide functioned as expected


                      I do appreciate you taking the time to try to duplicate it for me Alex.  Not sure how I should proceed though at this point.  I suppose I can rule out build environment problems ... it has to be some obvious pilot error that I'm just overlooking.  No idea what though, especially given the defunct ssh processes that get created and that the failure is completely silent.  No gonna mark this answered just yet ...

                       

                      BTW: How did you make the physical image after building the soft image?  I'm guessing DD on the Ubuntu build and some "I-don't-know-what-utility" on the Windoze build.  That can't be an issue I'm sure just brainstorming what I could be doing differently.

                      • 10. Re: Image built from IOTDK source won't launch gdbserver properly
                        AlexT_Intel

                        Yeah, I agree it looks strange. If nothing else you can try bumping up SSh server verbosity or attaching to the daemon with strace and attempting to connect - strace log should provide a lot of interesting information for understanding what's happening.

                         

                        And yes, let's try the image I've built as you've proposed in the PM (I was going to propose the same ) - it's posted here: repo.opkg.net/galileo/images/ (iot-devkit*)

                        I used win32diskimager to prepare the card as recommended by this article: https://software.intel.com/en-us/programming-blank-sd-card-with-yocto-linux-image-windows/