13 Replies Latest reply on Jul 6, 2015 6:33 PM by LeonY

    Galileo Gen 2 SPI via MRAA in NodeJS

    LeonY

      I have a NodeJS app that runs a large LED Strip system via SPI.

      Everything works perfectly, however i have noticed that the app is using a HUGE amount RAM after few cycles.

      The app sends 7kb frames of data via SPI using the mraa library....im pretty certain its this library thats causing the runaway memory leaks.

      This is my SPI send function

      var m = require('mraa');
      x = new m.Spi(0);
      var frame = new Buffer(7000);
      function sendFrame()
      {
          rcv = x.write(frame);
      }
      
      

      Assume that the variable frame has a bunch of bytes already in it.

      If i comment it out, my app sits at around 32mb in TOP

      If i let it run, every couple of seconds the app adds on another 2 Mb. After 5 minutes its over 200mb and forces the system to crash the app....

      Any idea why the mraa library would be leaking memory? and any way i can fix this?

        • 1. Re: Galileo Gen 2 SPI via MRAA in NodeJS
          Intel_Alvarado

          Hi,

           

          What image are you using? Is there any other way you are handling the data, are there any other processes running that may be affecting the performance? When using tops, does the process go up in priority and continues to increase the %MEM field? This issue may be caused because of a full buffer; I believe you need to clear the buffer so that it never gets to full so you can use it continuously. Take a look at this thread: Re: Question about Write/Read SPI functions for Edison (Mraa)

           

          Sergio

          • 2. Re: Galileo Gen 2 SPI via MRAA in NodeJS
            LeonY

            Hi Sergio

            >>What image are you using?

            Latest IOT image from the Intel web site


            >>Is there any other way you are handling the data

            No,as soon as i comment out

            rcv = x.write(frame);

            The memory usage is stable at around 32mb


            >>Are there any other processes running that may be affecting the performance?

            None, Tops is clear, the biggest process and RAM user is the "nodeapp slot" process, which is my app


            >>When using tops, does the process go up in priority and continues to increase the %MEM field?

            Yes the process is at the top of the list, and sits at 54% CPU usage. the RAM usage goes up every time TOP refreshes, it NEVER goes down


            >>This issue may be caused because of a full buffer; I believe you need to clear the buffer so that it never gets to full so you can use it continuously

            This is probably the case, HOW DO I "CLEAR THE BUFFER" ?? The javascript documentation for the 'mraa' library is virtually non existent. there is 1 example of how to use SPI and its wrong/doesn't actual work as described on the Galileo. Can you please point me in the right direction, I am getting increasingly frustrated.


            >>Take a look at this thread: Re: Question about Write/Read SPI functions for Edison (Mraa)

            That thread is irrelevant, its pointed at C/C++ code and has nothing to do with my issue. It talk

            • 3. Re: Galileo Gen 2 SPI via MRAA in NodeJS
              LeonY

              Dear Intel support, can you please respond, this seems to be a BUG in the mraa libarry...

              I have been doing a whole bunch of testing to eliminate what the problem might be and I'm 100% certain it has to do with the mraa SPI implementation.

               

              SPI Limits i have found:

              • The 1st thing i noticed was that there is limit of about 4000 bytes in the .write(data) function. anything larger does not write at all!
              • If I send out a packet of data that is 500 bytes or less with a gap of 100ms the Memory usage for the NodeJS App does not rise, it stays the same
              • If I send out a bunch of 500 byte packets with no gap in between, memory usage raises as per my original post
              • I then tried to test what is the smallest packet block i could send without memory leaks, even sending a bunch 200 bytes packets without a gap leaks memory

               

              This will show a memory leak, it sends a packet of 200 bytes 10 times via SPI

              var m = require('mraa'); 
              x = new m.Spi(0);
              var frame = new Buffer(200)
              for(var counter=0; counter<10; counter++)
              { 
                  rcv = x.write(frame); 
              }
              
              

               

              There is obviously a problem with the way memory is assigned in the SPI send function, is anyone able to tell me how I can work around this?

              • 4. Re: Galileo Gen 2 SPI via MRAA in NodeJS
                LeonY

                BUMP please. Does anyone have any ideas?

                • 5. Re: Galileo Gen 2 SPI via MRAA in NodeJS
                  Intel_Alvarado

                  Hi,

                   

                  We are currently working on your case. We will post a suggestion for you soon.

                   

                  Sergio

                  • 6. Re: Galileo Gen 2 SPI via MRAA in NodeJS
                    arfoll

                    Hey LeonY, thanks for the bug report! I've fixed this in git master of mraa, you're entirely right there is a memleak when the node::Buffer gets created it never frees the memory allocated for the C array. You can see the commit here: mraajs.i: Fix memory leak on javascript Spi::write · intel-iot-devkit/mraa@4698d18 · GitHub.

                     

                    You can either compile a new version of mraa from git or use the intelgalactic-dev repo. Either change /etc/opkg/mraa-upm.conf on your board or download the package manually and use opkg install *.ipk. Direct link will be here it'll be mraa-0.7.20 or above, just grab the latest - intelgalactic-dev/. I just pushed so it should be there in 30mins or so.

                    • 7. Re: Galileo Gen 2 SPI via MRAA in NodeJS
                      LeonY

                      Thank you mate

                      ill try this out today

                      • 8. Re: Galileo Gen 2 SPI via MRAA in NodeJS
                        LeonY

                        Hey arfoll

                        This did not fix the issue:

                         

                        1. I downloaded the latest *.ipk as you sugested V7.2.9 and save dit to the SD card
                        2. I then loged in via the terminal as root
                        3. ran the following command

                         

                        root@galileo:/media/card# ls
                        LOST.DIR  boot  bzimage  firmware  libmraa0_0.7.2.9_i586.ipk  win-driver
                        root@galileo:/media/card# opkg install libmraa0_0.7.2.9_i586.ipk
                        Upgrading libmraa0 on root from 0.6.0-r0 to 0.7.2.9...
                        Removing package libmraa-dev from root...
                        Removing package libmraa-doc from root...
                        Removing obsolete file /usr/lib/libmraa.so.0.6.0.
                        Configuring libmraa0.
                        

                         

                        1. Rebooted the board
                        2. Checked the MRAA version via a nodejs script:
                        Intel XDK - Message Received: run                                                                                                                                           
                        MRAA Version:                                                                                                                                                               
                        v0.7.2-9-g41d71e6                                                                                                                                                           
                        

                         

                        I then re uploaded my test case as in my original post, and saw that memory is still leaking....

                        Can you please let me know if i need to do anything else?

                        • 9. Re: Galileo Gen 2 SPI via MRAA in NodeJS
                          arfoll

                          That's the wrong package, my versioning info was a bit wrong. You want >=0.7.2.20. Here's the direct link to 0.7.2.21 which contains the fix:

                          http://iotdk.intel.com/repos/1.5/intelgalactic-dev/libmraa0_0.7.2.21_i586.ipk

                          • 10. Re: Galileo Gen 2 SPI via MRAA in NodeJS
                            LeonY

                            OK

                            So i downloaded the one you just pointed me to and installed it:

                             

                            root@galileo:/media/card# ls
                            LOST.DIR  boot  bzimage  firmware  libmraa0_0.7.2.21_i586.ipk  win-driver
                            root@galileo:/media/card# opkg install libmraa0_0.7.2.21_i586.ipk
                            Upgrading libmraa0 on root from 0.7.2.9 to 0.7.2.21...
                            Configuring libmraa0.
                            

                             

                            Ran a test nodejs script to see that its definitely updated
                            
                            Intel XDK - Message Received: run                                                                                                                                           
                            MRAA Version:                                                                                                                                                               
                            -----------------------                                                                                                                                                     
                            v0.7.2-21-gc5b0d7a                                                                                                                                                          
                            ------------------
                            

                             

                            However the Memory is still leaking, all be it slower.... Im running this as a test:

                             

                            var m = require('mraa');
                            x = new m.Spi(0);
                            var testBuffer = new Buffer(500);
                            while(true)
                             {
                              x.write(testBuffer);   
                             }
                            

                             

                            And you can see the RAM usage go up to 40mb, it says like that for 10 seconds then starts climbing again.

                            • 11. Re: Galileo Gen 2 SPI via MRAA in NodeJS
                              LeonY

                              Hey arfoll

                              SO i have been running my test code fro about 10-15 minutes now..its definitely better but there is still a leak.

                              RAM usage climbs about 2 meg every minute or so and they stays still, not a single byte more or less.

                              Then it climbs another 2 meg or so etc....

                               

                              If you can not resolve this, Im going to just put my application on a time contraint loop and then kill it and restart it when the RAM usage gets over 100mb(which is about 15min run time) I would rather not do this, its pretty hacky..

                               

                              Im constantly getting blocked by this board inability to do what i want it to, pretty close to just binning it and going with the Edisson or maybe even a totally different device.

                              • 12. Re: Galileo Gen 2 SPI via MRAA in NodeJS
                                arfoll

                                Sorry @LeonY it took me so long to get back, I wanted to spend the time to get you a decent answer. How are you measuring memory usage?

                                 

                                I've had a look with valgrind on a minnowboardmax (galileo is too slow to do this) and I can't see an obvious leak that would be causing this much damage, it's a bit hard to know because node does some very weird stuff with alocation so it's very hard to know 100%. But overall I can see that the node process isn't growing in used memory but can see that galileo is.

                                 

                                So here comes my guess - by doing a while(true) loop you're pegging the CPU with the node.js process, it gets so busy that it stops bothering doing garbage collection effectively leaking all your node::Buffer objects. By using node --expose-gc and then calling global.gc() I manage to keep VmSize down on the node process and I can see total system memory stops going down after a few minutes. I left it running for 10 minutes and MemFree (in free) went from 143564kB to 143572kB so essentially hasn't moved.

                                 

                                The reason for the crash is probably that OOM just decides to kill the app before node even realises it really needs to garbage collect :/ Note I did all this using IOTDK 1.5 release so you may also want to update your gailelo image to the latest, it's using yocto 1.7 which may help (or not :/). Good luck!

                                • 13. Re: Galileo Gen 2 SPI via MRAA in NodeJS
                                  LeonY

                                  Hey mate thanks for your hard work


                                  >>I've had a look with valgrind on a minnowboardmax (galileo is too slow to do this) and I can't see an obvious leak that would be causing this much damage, it's a bit hard to know because node does >>some very weird stuff with alocation so it's very hard to know 100%. But overall I can see that the node process isn't growing in used memory but can see that galileo is.


                                  I was afraid this would be the issue. I was very much hoping that it was a memory leak in the SPI library..

                                   

                                  >>by doing a while(true) loop you're pegging the CPU with the node.js process, it gets so busy that it stops bothering doing garbage collection effectively leaking all your node::Buffer objects

                                  The example code i posted is just an example to prove that its not my code, and is indeed the mraa spi implementation in node. My actual code does a whole bunch of stuff before calling the x.write() function.

                                   

                                  >>you may also want to update your gailelo image to the latest, it's using yocto 1.7


                                  I'm hesitant to update the board, every time i have done so i have lost the ability to do something, and gained the ability to do another...


                                  >>calling global.gc()

                                  I'm going to try and do this in my own code and see if this helps. I tried it befor elogging this bug, to this if it would help but didnt. Now that you have fixed the mraa memory leak, ill try it agian and see.

                                   

                                  All up, i can live with this as i can make a workaround...I have ordered an Edison kit for this project, and will probably ditch the Galileo entirely. I have been working on the Gen1 and Gen2 boards for this particular project for about a year now and every-time i think "Yei! It's working" the damn board throws something like this at me. Thanks for your time