12 Replies Latest reply on Mar 17, 2016 8:30 PM by vpalacios

    Edison becomes unresponsive - too much data from MCU?

    nemrod

      Hi,

      I'm sending some measurement data from the MCU to Linux userspace using host_send(). It's always 20 bytes at a fairly constant rate of roughly 50Hz, so about 1kB/s. Problem is the Edison seems to freeze after a few minutes up to maybe ten when doing this; regardless of what's going on in Yocto (i.e. I don't have to actually read the data, I can just boot into it and after a while it'll freeze and I need to pull the plug and restart and uninstall while it's still responsive). When the MCU isn't sending any data it all works fine. Does this make sense? Are there any other approaches I can take to this that doesn't result in this behaviour? I've included the MCU code below.

       

      Thanks,

      Anders

       

      Optional context (i.e. me venting frustrations): This has all come about because of me trying to read some PWM signals. The Edison on the Yocto end is handicapped by embarassingly only having software interrupts on GPIOs (which makes the timing too imprecise to measure even 50Hz PWM signals), so I've had to resort to constantly polling the GPIOs using the MCU (polling because, again, the firmware is handicapped by not being able to trigger on both edges and I'm not going to resolder the circuit board to connect all of the PWM signals to two GPIO ports, assuming that'd work). Since I now have no choice but to run the measurements on the MCU I have to somehow pass the data between it and the main software running in Yocto, hence host_send() at about 50Hz.


      #include "mcu_api.h"
      #include "mcu_errno.h"
      void mcu_main() {
          int pin[4] = {81, 80, 79, 78};
          unsigned long pulseWidth[4] = {1500, 1500, 1000, 1500};
          unsigned long startTime[4] = {0, 0, 0, 0}; 
          int previousState[4] = {0, 0, 0, 0}; 
          int i;
          for (i = 0; i < 4; i++) {
              gpio_setup(pin[i], 0); 
          }   
          unsigned long printTime;
          unsigned char buf[20] = "0000000000000000000\n";
          debug_print(DBG_INFO, "Going into main loop.\n");
          while (1) {
              for (i = 0; i < 4; i++) {
                  if (gpio_read(pin[i])) {
                      if (!previousState[i]) {
                          startTime[i] = time_us();
                          previousState[i] = 1;
                      }   
                  } else {
                      if (previousState[i]) {
                          pulseWidth[i] = time_us() - startTime[i];
                          if (pulseWidth[i] < 1000) {
                              pulseWidth[i] = 1000;
                          } else if (pulseWidth[i] > 2000) {
                              pulseWidth[i] = 2000;
                          }   
                          previousState[i] = 0;
                      }   
                  }   
              }   
              if (time_us() - printTime > 20000) {
                  printTime = time_us();
                  mcu_snprintf(buf, 20, "%d;%d;%d;%d\n", pulseWidth[0], pulseWidth[1], pulseWidth[2], pulseWidth[3]);
                  host_send((unsigned char*)buf, 20);
              }   
          }   
      }
      
        • 1. Re: Edison becomes unresponsive - too much data from MCU?
          CMata_Intel

          Hi Anders,

           

          Which image do you have in your board?

          Have you tried different rates?

          When the project doesn't get stuck, are you able to test the signals? Is the PWM working as it should?

          There is another thread where Cooper is communicating the MCU with the Host. Take a look at it maybe it could be helpful for you.

           

          Regards,

          Charlie

          • 2. Re: Edison becomes unresponsive - too much data from MCU?
            nemrod

            Hi CMata_Intel, thanks for the reply!

            I'm running 2.0 (because I²C is broken on 2.1, see I²C bus freezes randomly).

             

            I've tried 100 and 50Hz and both fail, though 50 is subjectively slightly better (still unresponsive within 10 minutes). Running any slower will make it useless for my purposes but I will try lowering it further to see if there's a strong correlation between the amount of data/frequency of data transmissions and the time it takes to become unresponsive.

             

            Until the system becomes unresponsive it all works just fine; I can read /dev/ttymcu0 to see the data I'm sending. It seems almost gradual; after a while it takes a second to react to keyboard input and pretty soon after the session dies (at which point the Eclipse MCU functions don't work either). I haven't confirmed that behaviour conclusively though, could be some temporary network quirk.

             

            The PWM signals are working fine, confirmed with an oscilloscope (and by being able to read the data), but even if I turn the signals off (i.e. it just sends the same default values over and over, see code) there's no change in behaviour.

             

            Thanks for the link, unfortunately didn't help with my issue.

             

            Cheers,

            Anders

            • 3. Re: Edison becomes unresponsive - too much data from MCU?
              CMata_Intel

              Hi,

               

              I have just tested your code above using a clean image release-2.0 and I was able to run it and received the message (cat /dev/ttymcu0) in the Terminal Console for about 20 minutes without problems, are you running something else in your board? Or in the MCU sdk?

               

              Regards,

              Charlie

              • 4. Re: Edison becomes unresponsive - too much data from MCU?
                nemrod

                CMata_Intel, that's strange. I'm running completely stock 2.0 and nothing else, I can do a cold boot and not touch anything and if I try to contact the Edison after ten minutes it'll be frozen. When the MCU program is not running the Edison runs indefinitely (several days at least). I lowered the frequency to 10Hz and it seemed to run for much longer; at least half an hour.

                 

                edit: tested again and only got kicked out after a whopping 1.5h. That time I was running cat /dev/ttymcu0. Strangely I was able to connect to it again, only when I checked uptime I was greeted by 0 minutes; the Edison had rebooted without me doing a thing. After about 5 minutes from that self-induced reboot it's now completely unreachable again, in line with previous experience. This time without running cat /dev/ttymcu0. Coincidence, or does reading the tty stop the data from clogging the system as fast?

                 

                anders@nemtop ~ % ssh root@192.168.1.94
                root@192.168.1.94's password:
                root@nedison:~# for i in $(seq 1 50); do uptime; sleep 60; done;
                01:51:23 up 0 min,  load average: 1.16, 0.27, 0.09
                01:52:23 up 1 min,  load average: 1.48, 0.53, 0.19
                01:53:23 up 2 min,  load average: 1.44, 0.69, 0.27
                01:54:23 up 3 min,  load average: 1.41, 0.82, 0.34
                01:55:23 up 4 min,  load average: 1.34, 0.91, 0.40
                01:56:23 up 5 min,  load average: 1.45, 1.02, 0.47
                packet_write_wait: Connection to 192.168.1.94: Broken pipe
                anders@nemtop ~ % ssh root@192.168.1.94
                ssh: connect to host 192.168.1.94 port 22: No route to host
                
                
                

                 

                You'll notice that the load is 1-1.5 despite me not doing anything except running that single command (that should induce no load considering sleep isn't busy-wait). When not running the MCU program the load is 0.0x, as expected.

                • 5. Re: Edison becomes unresponsive - too much data from MCU?
                  CMata_Intel

                  Hi nemrod,

                   

                  As you may know the MCU feature came with the release 2.1  not with 2.0, this could be the root of the behavior you are having, you can keep trying to use successfully the MCU but you may have problems because of this. I suggest you to try the same tests with Release 2.1 and let us know. I know you are concerned about the performance of I2C but let's try to run your tests and check if the issue is the release version.

                   

                  Regards,

                  Charlie

                  • 6. Re: Edison becomes unresponsive - too much data from MCU?
                    nemrod

                    CMata_Intel, it's as you say, but I don't dare upgrade to 2.1 for fear of I²C not working despite downgrading next time. Quite frankly I don't trust the Edison at this point, after the numerous issues I've head. If it stops working and I can't get it to work again I won't be able to finish my thesis which is based around it, so for now I won't upgrade. Could you perhaps be so kind as to check if the load goes up as it does for me---on 2.0 and 2.1? If it doesn't on 2.0 then clearly my Edison is behaving differently from yours and it isn't necessarily an issue with 2.0. If it does on 2.0 but not on 2.1, then that's that and we know that's most likely the issue.

                     

                    Thanks,

                    Anders

                    • 7. Re: Edison becomes unresponsive - too much data from MCU?
                      CMata_Intel

                      Hi nemrod,

                       

                      I ran the same command with release 2.0 and had the same results. I am about to run more tests with 2.1 and I will let you know the outcome.

                       

                      Regards,

                      Charlie

                      • 8. Re: Edison becomes unresponsive - too much data from MCU?
                        CMata_Intel

                        Hi Anders,

                         

                        The MCU is not fully implemented on Release 2.0 which might be the root cause of this behaviour, we highly suggest to upgrade to Release 2.1. One way to alleviate on Release 2.0 is to constantly read the data coming from the MCU through device ttymcu0. The load is caused by the mcu function host_send() which sends data to the Host CPU, when this data is not managed it overwhelms

                        the Host CPU. The MCU program running itself has no impact over the Host CPU load.

                        • 9. Re: Edison becomes unresponsive - too much data from MCU?
                          nemrod

                          CMata_Intel, good to know that's indeed the cause as suspected. If it works on 2.1 that's great, but as explained earlier I can't upgrade because I²C is broken on that release. I look forward to release 2.2 where hopefully both work. Since my program won't necessarily be running at all times it's hard to ensure continuous reading of the MCU data; perhaps a script that runs at boot and just reads the data would work, but I'm not sure of the conditions of reading that ensures no freezing (for my program, since it's a bit more complicated than just `cat`)... I'll have to investigate I suppose.

                           

                          Thanks for the help testing, Charlie!

                          • 10. Re: Edison becomes unresponsive - too much data from MCU?
                            cobra2326

                            I'm having a similar issue to nemrod, except I'm on a newer version (I believe 2.2):

                             

                            root@edison_1:~# configure_edison --version

                            159.devkit


                            I'm sending messages over /dev/ttymcu1 ever 30 seconds or so after reading a DHT22 temperature sensor.  Eventually WiFi goes down and the only way to connect is via direct serial connection.  Once I got in here's what I see:


                            Screen Shot 2016-01-26 at 9.43.30 AM.png


                            irq/47-intel_ps is pegging the CPU.  If I then run 'cat /dev/ttymcu1', the irq pegging goes away and the system is responsive again. WiFi is still down (I'm working on that.


                            What's causing this?  I thought this was fixed in the latest version?  Also, could you explain which IRQ intel_ps is?

                            • 11. Re: Edison becomes unresponsive - too much data from MCU?
                              CMata_Intel

                              Hi cobra2326,

                               

                              The irq/47 is related to WiFi (/proc/irq/47), are you running something else on the board, you can run ps and check all the processes you have.

                              I’m sorry to hear that you are having problems with the MCU while using the 159.devkit image, I would like to help you with this but I will need more information regarding this.

                               

                              Remember the MCU SDK is intended to be used with a stable connection with the board in order to not lose the interactions between Eclipse and the board, I suggest you to look for network issues that may be causing the disconnection problem and this also could be affecting the problems in your application.

                               

                              Are you uninstalling the applications successfully?

                              On the Eclipse IDE, are you able to see errors on the Console?

                              Is this happening with the example Communicating between the host and the MCU or only with your project?

                              What are you writing to the ttymcu1?

                               

                              Regards,

                              Charlie

                              • 12. Re: Edison becomes unresponsive - too much data from MCU?
                                vpalacios

                                Hi.. im having the same issue as cobra2326 also with 159.devkit:

                                 

                                I send accelerometer data read from MCU using host_send()... I can see the data coming using cat /dev/ttymcu0, but if I stop the cat process to let the data pile up, after a while the serial terminal becomes sluggish and irq/47-intel_ps goes to the roof.

                                 

                                If I do cat /dev/ttymcu0 again I see most backlogged data and the irq/47-intel_ps process goes back to 0%, but at this point I've already lost wifi connectivity. Also no new data is arriving to /dev/ttymcu0.

                                 

                                I don't see any errors in the mcu sdk. Any ideas to track this down?

                                 

                                Regards,