13 Replies Latest reply on Oct 24, 2016 1:17 PM by Intel Corporation

    Running scripts in background invoked from service?

    zettez

      Greetings.

       

      I followed this guide how to connect to a bluetooth device from a service. The service will start a script that in turn will unblock the bluetooth and connect to the device. I want to run 2 python scripts in the background afterwards, but I am unsure if it is working.

       

      This is my current script:

       

       

      #!/bin/bash
      
      # Just to make sure everything works
      export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/
      
      # Unblock, nothing fancy
      rfkill unblock bluetooth
      
      sleep 2
      
      # Connect to my specific device
      bluetoothctl << EOF  
      
      connect XX:XX:XX:XX:XX:XX
      
      EOF  
      
      sleep 4
      
      # Set the audio to my bluetooth speaker
      pactl set-default-sink bluez_sink.XX_XX_XX_XX_XX_X
      
      sleep 1
      
      /home/root/scripts/script1.py anArgumentNeeded &
      
      sleep 10
      
      /home/root/scripts/script2.py anArgumentNeeded &
      

       

      Is this the correct way to run these 2 python scripts? They both have a shebang and are executable. They work if I invoke them manually from the command line. When I run them from the command line, I can find them by typing ps and grep the script. This seems not to work if I run it from the background. Should they be visible with ps and grep if they are executed from the service? Will they have different names or will the service "hide" or overlap these 2 python scripts?

       

      This is my service:

       

      #!/bin/sh
      
      [Unit]
      Description=Bootstrap my script
      
      [Service]  
      ExecStart=/home/root/scripts/start_script.sh
      Type=idle
      
      [Install]
      WantedBy=basic.target
      
        • 1. Re: Running scripts in background invoked from service?
          Intel Corporation
          This message was posted on behalf of Intel Corporation

          Hello zettez,

          I will try to help you with this. When trying to start a Python script from a system service you have to use a slightly different configuration on your service. Since I don't have access to the scripts you are using I will show you how to create a system service that starts a blink script written on Python.

          This is the script I used (it's called blink.py):

          #!/usr/bin/python
          import mraa
          import time

          x = mraa.Gpio(13)
          x.dir(mraa.DIR_OUT)

          while True:
              x.write(1)
              time.sleep(0.2)
              x.write(0)
              time.sleep(0.2)

          This is the system service:

          #!/bin/sh
          [Unit]
          Description=Edison Arduino board LED Blinker
          [Service]
          WorkingDirectory=/home/root/
          ExecStart=/home/root/blink.py
          Type=simple
          [Install]
          WantedBy=basic.target

          I believe the difference is made by "WorkingDirectory", I suggest you to try this configuration. 

          Let us know how it goes.
          -Peter.

          • 2. Re: Running scripts in background invoked from service?
            zettez

            Hey!

             

            That did not work, my scripts never started for some reason.

             

            WorkingDirectory should be pointing to what you mentioned right? /home/root

            • 3. Re: Running scripts in background invoked from service?
              Intel Corporation
              This message was posted on behalf of Intel Corporation

              The "WorkingDirectory" should be the place where your scripts are stored, in my case it was /home/root/ in your case this may vary. For example, according to your first reply it should be /home/root/scripts/.

              Try it and let me know how it goes.
              -Peter.

              • 4. Re: Running scripts in background invoked from service?
                zettez

                Ah, I thought it was the user's home that should be specified. I changed it to /home/root/scripts/ but it still does not work. It took a while longer to boot/log into the Edison, so maybe something did happen, but the bluetooth was still off and no scripts were running.

                • 5. Re: Running scripts in background invoked from service?
                  Intel Corporation
                  This message was posted on behalf of Intel Corporation

                  I understand, that is unexpected. Is it possible for you to share your scripts? I would like to see if I can replicate the behavior and to see if I can find a way to prevent this to happen.
                   
                  -Peter.

                  • 6. Re: Running scripts in background invoked from service?
                    zettez

                    The scripts connect to a remote socket and send data to our backend endpoint, so it will not be able for you to reproduce the exact same thing, but I think you can comment these lines out.

                     

                    My bootstrap_script.service:

                     

                    #!/bin/sh
                    
                    
                    [Unit]
                    Description=Bootstrap Script
                    
                    
                    [Service] 
                    WorkingDirectory=/home/root/scripts
                    ExecStart=/home/root/scripts/start_script.sh
                    Type=simple
                    
                    
                    [Install]
                    WantedBy=basic.target
                    

                     

                     

                    Now my start_script.sh:

                     

                    #!/bin/bash
                    
                    
                    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/
                    
                    
                    rfkill unblock bluetooth
                    
                    
                    sleep 2
                    
                    
                    hciconfig hci0 down
                    
                    
                    sleep 1
                    
                    
                    hciconfig hci0 up
                    
                    
                    sleep 2
                    
                    
                    bluetoothctl << EOF
                    
                    
                    connect XX:XX:XX:XX:XX:XX
                    
                    
                    EOF
                    
                    
                    sleep 4
                    
                    
                    pactl set-default-sink bluez_sink.XX_XX_XX_XX_XX_XX
                    
                    
                    sleep 1
                    
                    
                    /home/root/scripts/socketCommunication.py ws://our.socket.endpoint &
                    
                    
                    sleep 4
                    
                    
                    /home/root/scripts/discoverBeacons.py https://our.report.endpoint &
                    

                     

                    Now my socketCommunication.py:

                     

                    #!/usr/bin/python
                    
                    
                    import os
                    import time
                    import json
                    import boto3
                    import sys
                    import time
                    from websocket import create_connection
                    
                    
                    lastActionCommand = ""
                    lastActionTimestamp = 0
                    
                    
                    
                    
                    def handleAction(action, args):
                      global lastActionCommand
                      global lastActionTimestamp
                    
                    
                      print(action)
                    
                    
                      print("lastActionCommand")
                      print(lastActionCommand)
                    
                    
                      print("lastActionTimestamp")
                      print(lastActionTimestamp)
                    
                    
                      tempActionCommand = lastActionCommand
                      tempActionTimestamp = lastActionTimestamp
                    
                    
                      lastActionCommand = action
                    
                    
                      now = time.time()
                    
                    
                      print("now")
                      print(now)
                    
                    
                      if tempActionTimestamp == 0:
                      tempActionTimestamp = now
                    
                    
                      print("tempActionCommand")
                      print(tempActionCommand)
                    
                    
                      print("tempActionTimestamp")
                      print(tempActionTimestamp)
                    
                      if action == tempActionCommand and tempActionTimestamp > now - 8:
                      return
                    
                      lastActionTimestamp = now
                    
                    
                      if action == "play":
                      audioFile = args[0]
                    
                    
                      print("playing audio file")
                      print(audioFile)
                    
                    
                      os.system(u"gst-launch-1.0 filesrc location= /home/root/sound/" + audioFile + u" ! wavparse ! pulsesink &")
                    
                    
                      elif action == "download":
                      name = args[0]
                    
                    
                      print(name)
                    
                    
                      s3_client = boto3.client("s3")
                    
                    
                      print("downloading file")
                    
                    
                      s3_client.download_file(u"edison", u"sound/" + name, u"/home/root/sound/" + name)
                    
                    
                      print("download complete")
                    
                    
                      elif action == "delete":
                      name = args[0]
                    
                      print("removing")
                      print(name)
                    
                      os.remove(u"/home/root/sound/" + name)
                    
                    
                      print("removing complete")
                    
                    
                    url = sys.argv[1]
                    
                    
                    while True:
                      try:
                      print("create connection to socket")
                    
                    
                      socket = create_connection(url)
                    
                    
                      try:
                      while True:
                      print("reading data..")
                    
                    
                      result = socket.recv()
                    
                      print("got data: ")
                      print(result)
                      print(type(result))
                    
                    
                      data = json.loads(result)
                    
                    
                      action = data["action"]
                      args = data["args"]
                    
                      handleAction(action, args)
                    
                    
                      except Exception as error:  
                      print("socket receive error")
                      print(error)
                    
                    
                      socket.close()
                    
                    
                      time.sleep(8)
                    
                    
                      except Exception as error:  
                      print("socket error")
                      print(error)
                    
                    
                      if socket != None:
                      socket.close()
                    
                    
                      time.sleep(8)
                    

                     

                    Now my discoverBeacons.py:

                     

                    #!/usr/bin/python
                    
                    
                    import pycurl, json, sys
                    
                    
                    from gattlib import DiscoveryService
                    
                    
                    url = sys.argv[1]
                    
                    
                    service = DiscoveryService("hci0")
                    
                    
                    
                    
                    while True:
                      print("discovering..")
                    
                      devices = service.discover(4)
                    
                    
                      data = []
                    
                    
                      for address, name in devices.items():
                      print("Found device, sending data (name: {}, address: {})".format(name, address))
                    
                    
                      data.append({"address":address})
                    
                    
                      listData = json.dumps(data)
                    
                    
                      c = pycurl.Curl()
                      c.setopt(pycurl.URL, url)
                      c.setopt(pycurl.HTTPHEADER, ["content-type: application/json"])
                      c.setopt(pycurl.POST, 1)
                      c.setopt(pycurl.POSTFIELDS, listData)
                      c.perform()
                    

                     

                    Note that the indentation got screwed up for some reason, you might have to add a tab or two at some places.

                     

                    As mentioned previously, executing start_script.sh manually works fine. Is there any logs I can look at for the upstart? Maybe I try to do something that has not been loaded yet, resulting in an exception and then the script stops?

                     

                    All my scripts are located in /home/root/scripts.

                    • 7. Re: Running scripts in background invoked from service?
                      Intel Corporation
                      This message was posted on behalf of Intel Corporation

                      Thank you for sharing this information. I will try to see what I can find out, meanwhile, I would like to know if there is any dependency I need to install. If so, could you please let me know what they are and how you installed them?

                      Also, when you run start_script.sh what happens? Are you able to connect correctly to the Bluetooth device and the Python scripts start as expected? If so, have you considered creating separated services for each script? I mean, one to connect the Edison to the Bluetooth device, one for the first Python script and one for the other Python script. You could make them start in the order you need, would that work for you?

                      Let me know.
                      -Peter.

                      • 8. Re: Running scripts in background invoked from service?
                        zettez

                        First of all, you need to install boost. I installed boost 1.58.0. It is needed for one of the libraries (gattlib).

                         

                        Libraries:

                         

                        * gattlib

                        * boto3

                        * aws-cli

                        * websocket-client

                        * pycurl

                         

                        Just use pip install library.

                         

                        And as mentioned earlier, the script will not work correctly as you do not have our socket or endpoint url (I cannot share them unfortunately). These libraries will make the script start though.

                         

                        When I run start_script.sh, everything works flawlessly. It connects to my bluetooth device, it listens to our socket and reports to our report endpoint.

                         

                        If I make them all services, how will I know they chain correctly? Both my scripts depend on having bluetooth connected. Maybe I can do something like this?

                         

                        [Install]

                        WantedBy=bluetooth.target

                         

                        Which means that my 2 scripts starts whenever I have bluetooth. But I need them to start after I have paired to my specific device, that is why I ran them in sequence in one script.

                         

                        The best scenario would just be to run the start_script.sh that starts everything. It is tested and works for me manually when everything has booted.

                         

                        Is there any file I can log to or something at startup? Maybe the script quits because an error came up? For instance, I try to access the internet before wifi has been engaged. That would crash the script. Or I try to start bluetooth because it has finished booting up. Maybe there are other things. Would be great if I could read the startup file to get any clue of what is going on.

                        • 9. Re: Running scripts in background invoked from service?
                          Intel Corporation
                          This message was posted on behalf of Intel Corporation

                          I understand, as you mentioned I won't be able to replicate your environment as I don't have the socket or endpoint URL. So, what I would like to try now is to check if we can create three services that start one after the other. This might cause different results and hopefully it is of help. You can do this with the option "After", for example, I created two services and scripts that started one after the other successfully (I verified it with the file they write to called serv.log), these are my scripts and services:

                           

                          blink.py

                           

                          #!/usr/bin/python
                          import mraa
                          import time
                          import os
                          
                          x = mraa.Gpio(13)
                          x.dir(mraa.DIR_OUT)
                          time.sleep(1)
                          os.system("echo blink_started_at: >> /home/root/serv.log")
                          os.system("date >> serv.log")
                          
                          while True:
                              x.write(1)
                              time.sleep(0.2)
                              x.wrrite(0)
                              time.sleep(0.2)
                          

                           

                          pyblink.service

                           

                          #!/bin/sh
                          [Unit]
                          Description=Edison Arduino board LED Blinker
                          [Service]
                          WorkingDirectory=/home/root/
                          ExecStart=/home/root/blink.py
                          Type=simple
                          [Install]
                          WantedBy=basic.target
                          

                           

                          sec.py

                           

                          #!/usr/bin/python
                          import time
                          import os
                          
                          time.sleep(1)
                          os.system("echo sec_started_at: >> /home/root/serv.log")
                          os.system("date >> serv.log")
                          
                          while True:
                              os.system("echo working >> sec.log")
                              os.system("date >> sec.log")
                              time.sleep(5)
                          

                           

                          sec.service

                           

                          #!/bin/sh
                          [Unit]
                          Description=Second service that starts after pyblink
                          After=pyblink.service
                          [Service]
                          WorkingDirectory=/home/root/
                          ExecStart=/home/root/sec.py
                          Type=simple
                          [Install]
                          WantedBy=basic.target
                          

                           

                          In this specific scenario sec.service will start right after pyblink.service thanks to the "After" option. Please try this configuration and let us know if it helps.
                          -Peter.

                          • 10. Re: Running scripts in background invoked from service?
                            zettez

                            Hey.

                             

                            I actually copied your idea of logging to a file to verify. Whenever I run the start_script.sh manually it prints out to a file. When I reboot, then the start_script.sh actually is run! It starts my first script, and it logs to the file. It never starts the second script though.

                             

                            My conclusion is that when the service starts the start_script.sh, it is run in a process. When the script finishes, it kills the process. My two python scripts are run with the & sign, spawning new processes, but I think they are actually sub processes of the other process, and it gets killed as well.

                             

                            I tried using nohup to skip the hangup signal, but it did not work either. It works flawlessly manually still, but not if my script is run as a service.

                             

                            I would prefer not to run my other scripts as services, as it increases complexity. I just want my script to set stuff up, and start two other scripts. Maybe I could put in a loop in the end of the start_script.sh so it does not get killed. Not sure if that would block other execution though.

                             

                            Is there any way to detach a sub process and let it be run as a stand alone process (much like nohup but more robust)?

                            • 11. Re: Running scripts in background invoked from service?
                              Intel Corporation
                              This message was posted on behalf of Intel Corporation

                              I understand why you wouldn't want to complicate your project adding three separate services. Nevertheless, it might be the best way to correct this behavior.
                               
                              However, there is something I would like to test. What happens if you remove the "&" sign from start_script.sh? Does your Edison have the same behavior? Or, is there something different?
                               
                              Let me know.
                              -Peter.

                              • 12. Re: Running scripts in background invoked from service?
                                zettez

                                Hey.

                                 

                                I removed the & sign and as expected, the second script never started. This is because the first script runs in a while true loop, blocking that thread or process or whatever it now is from executing the next script. I tried to solve it using the & sign to create a new thread/sub-process that would run these two scripts simultaneously.

                                 

                                I tried the service approach and they did start but then crashed. The first script had a socket error for some unknown reason and just stopped working, and the other script could not find the libraries needed (I am very certain it was the missing export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/ line, creating a new service does not let the current thread/process have access to the variables set in the previous service apparently).

                                 

                                I will not look into this further for now as I have other things with higher priority, I might come back to this later.

                                 

                                So in short: the service chaining worked, but my scripts crashed because some missing libraries/variables and what I think a too early startup before WiFi had finished setting up. Running the all scripts from one script manually did also work, but it did not work when booting. I guess I have to dig deeper with services and see when they run. It might be that the services has to be set after wifi has loaded to fix some issues. Will maybe look into it later.

                                 

                                Anyway, thanks for all the help along the way!

                                • 13. Re: Running scripts in background invoked from service?
                                  Intel Corporation
                                  This message was posted on behalf of Intel Corporation

                                  Hi zettez,

                                  I understand your situation. Whenever you are able to continue with this project, please feel free to post your updates in here. We'll try to help you in any way we can.

                                  -Peter.