1 2 Previous Next 17 Replies Latest reply on Jan 20, 2017 11:19 AM by supermomo90 Branched from an earlier discussion.

    Re: Ball controlled from hand gestures via SR300


      Hi MartyG,


      Basically I am trying to control a Sphero SPRK+ robot with my Realsense camera. I am having difficulty connecting the sphero to my pc as the manufacturers of Sphero have informed that Sphero SPRK+ does not support Windows PCs. Hence I was thinking about using Android or iOS devices as  a middle man to control the Sphero SPRK+.


      Hence in my other post ( Adding hand control in Unity application using SR300  ), I was trying to build a software version of the game. But I cannot control the ball nicely. I want to control it's direction and speed just by using gestures detected.


      Moving hand forward = faster

      Close fingers= stop

      Spread fingers = start moving


      and so on.


      Thank you.

        • 1. Re: Ball controlled from hand gestures via SR300

          What you are describing should be quite simple to achieve with a Unity TrackingAction.  What you need to do is define Start and Stop "rules" for the TrackingAction, instead of using the default Hand Detected activation setting.


          (a) Right-click on the 'Hand Detected' setting under the Start Event label to bring up its menu and select the 'Remove' option to remove it from the TrackingAction.


          (b) Left-click on the 'Add' button beside the Start Event label  and select the 'Gesture Detected' option.


          (c) In the new set of options that has appeared, expand open the Gesture Detected option using the small arrow beside it and leave the setting on the default 'Spread Fingers' setting.  With this setting, the TrackingAction will only activate when the camera sees the Spread Fingers gesture.


          (d) Go to the 'Stop Event' option in the middle of the TrackingAction.  Usually, this is set to 'Hand Lost' by default.  Right-click on the 'Hand Lost' option and remove it.  Then left-click on the Add button beside the Stop Event label and set it to 'Hand Closed'.  Now the TrackingAction will stop tracking and cease your object's movement when the camera sees the Hand Closed gesture.


          If you need to reduce the movement speed of your object, go to the 'Smoothing Factor' of the TrackingAction.  Ensure that it is set to the 'Weighted' type of smoothing, and give if a higher value than the default of '0'.  '5' is a good test value for a quick but stable movement, '10' for a slow movement, and a maximum of '19' for very slow movement (objects tend to stop moving once their weighting is 20 or more on the Weighted setting).

          1 of 1 people found this helpful
          • 2. Re: Ball controlled from hand gestures via SR300

            Hi MartyG,


            I can now control it smoothly. Thank you. However there is a small issue. When the hand is closed, the ball should stop moving completely (freeze at it's last position). But it continues to roll even though the hands are closed.


            May I know how I can make the ball freeze at the position the hands closed action is activated and continue from that position when the gesture detected action is activated?


            Thank you.

            • 3. Re: Ball controlled from hand gestures via SR300

              My guess would be that when the TrackingAction stops tracking when the hand is closed, the Rigidbody physics component is taking over and making the ball roll.  This is similar to a setup I had in my project, with a TrackingAction that lifted a block up in the air and a gravity-enabled Rigidbody.  Whenever tracking was lost, the gravity influence of the Rigidbody - which is usually disabled during TrackingAction use - kicked in and made the block fall down.


              If you have gravity ticked on your Rigidbody, I would un-tick it.  Then when tracking ceases when you close the hand, the ball shouldn't roll.  This is because without gravity, the Rigidbody physics can't accelerate the ball and it should stay fixed on whatever spot you left it on when the hand was closed.

              1 of 1 people found this helpful
              • 4. Re: Ball controlled from hand gestures via SR300

                Hi MartyG,


                Thank you very much. That solved the problem. But I still have 1 small issue. Let's say the ball stops completely once I have initiated the stop action and the ball is said to be at position X. Then when I move the hand to another position and resume the gameplay, the ball does not continue from position X. It starts from a different position. I have tried to experiment but cannot seem to solve the issue.


                Thank you.

                • 5. Re: Ball controlled from hand gestures via SR300

                  I spent about 2 years trying continuously without luck to solve that trait of TrackingAction with experiments and editing of its code.  It is built into its code to update the object's position to the current position of the hand when tracking of the hand is re-acquired.  I ended up writing my own custom control system last year that takes the angle of the object and converts it into a movement animation generated by an animation clip.  By doing it this way, you can control an animation in real-time by using the TrackingAction to wind backwards and forwards through an animation sequence.


                  So if you create, for example, an animation of an arm lifting from fully dropped to fully raised above the head, moving the real life hand upward winds forward through the clip's timeline towards its end, lifting the arm object, whilst lowering the arm rewinds the clip back towards its beginning, making the arm object drop down.


                  It's an extremely precise, smooth and stable system.  It also solves a couple of other drawbacks of TrackingAction (objects moving when you turn the wrist even if you haven't set wrist detection, and the object flipping-out if you do too extreme a movement in a particular direction).  Unfortunately I haven't got around to writing a how-to guide to the technique yet.

                  • 6. Re: Ball controlled from hand gestures via SR300

                    Hi MartyG,


                    Thank you for sharing your expertise in Unity and Realsense.

                    • 7. Re: Ball controlled from hand gestures via SR300

                      You're welcome!


                      I was just answering your post about Unity volume but the post got deleted, so I'll put the reply here!


                      You should not need an Audio Listener in your canvas.  You only need one listener in each Unity scene.  There is one inside the main camera by default.  So unless you un-tick the Audio Listener option in the camera's settings in the Inspector then you will get the message in the debug console that "There are 2 audio listeners in this scene".


                      I do not see a problem with putting your Audio Source inside the canvas if your canvas is permanently active.  Otherwise, if the Canvas is deactivated, the Audio Source inside it should become deactivated too.


                      I think I see what the problem in your code may be.  When you create a float, its value is '0' by default unless you give it another value, or connect it to something that will change the value.  So what the above code is saying is that the float 'controller' will have the value '0'.  And because 'Audiolistenervolume' uses the 'controller' float for its volume value, then the volume will be '0' (i.e silent) when the function public void volumeControl is activated.

                      1 of 1 people found this helpful
                      • 8. Re: Ball controlled from hand gestures via SR300

                        Hi MartyG,


                        Yes I deleted it because I forgot this forum is supposed to be for Intel Realsense.


                        I tried adjusting the script. I was able to reduce the volume to a specific value because I hardcoded the value. I think a for loop needs to be created for the variable newVolume.


                        float newVolume = 0.4f;


                        public void volumeControl(float controller=0.0f)


                                AudioListener.volume = controller;

                                AudioListener.volume = newVolume;


                        • 9. Re: Ball controlled from hand gestures via SR300

                          Yes, you need to use a For or a Foreach.  I use Foreach in my own project's volume analysis system.  It's not exactly what you are trying to do - in my system, it finds out what the volume is instead of trying to change the volume - but it might give you some useful ideas about how to proceed.


                          STEP 1


                          As well as your newVolume float, you need to put a definition for an Array type variable in the header of your script.  Array type variables have a pair of square brackets (not the squiggly ones) after the variable name.


                          private int SampleArrayLength = 2;

                          private float Data[];


                          STEP 2


                          In your Start() section:


                          Data = new float[SampleArrayLength];


                          STEP 3


                          In the Update() section: (this script is going to assume that the activation code for your AudioSource is in the canvas at the same location as your volume script).


                          GetComponent<AudioSource>().clip.GetData (Data, GetComponent<AudioSource>().timeSamples);


                          foreach (var sample in Data) {


                          newVolume = math.Abs (sample);




                          STEP 4


                          The above foreach code will look at the volume of the audio clip currently playing in the Audio Source  and constantly update newVolume to a volume value between 0 and 1.


                          In my project, I use the volume to move the mouth of the player character.  If the audio input is silent then volume = 0 and the mouth is closed.  The louder the volume, the wider the mouth opens.  In the silent space between words, the mouth automatically closes as volume nears 0.  In this way, the mouth lip-syncs to the audio in real-time without having to be pre-programmed.


                          "My Father's Face" Tech Trailer 4 - Lip Synching And Facial Expression From An MP3 - YouTube

                          • 10. Re: Ball controlled from hand gestures via SR300

                            Hi MartyG,


                            My button controller script has no start and update section. It's all public void declarations.

                            • 11. Re: Ball controlled from hand gestures via SR300

                              You can define Start and update sections in the following way:


                              public void Start() {

                              public void Update() {


                              public void Start() should be the first function in your script (below the Public Class instruction), and public void Update() below it.


                              Remember to define the Start and update names as written, with capital S and U, or it will not work.


                              This is probably a good opportunity to give readers a basic lesson about functions.  The Start() function is a "fire once" type of function.  It typically contains setup instructions for your script, including the location of objects so that the script can find them in your project.  Once it has completed processing the list of instructions inside it, it does not run that section again.


                              Update() functions are infinitely looping functions that run the list of instructions inside them over and over until the function is stopped.  They are good for processing information that is constantly changing and so needs to be re-checked every second, such as the current volume of an audio source or the position of an object.


                              If you create a function with your own user-defined name then these are treated as fire-once functions like Start.  You can run a fire-once function again though at any time though by putting a reference to its name in another part of the script, and the script then jumps to that function when it sees the name.  For example, if you had a function called MyVolume(), then somewhere else in the script, you would use this line to make the script jump to MyVolume:




                              So basically, the function name followed by ();

                              • 12. Re: Ball controlled from hand gestures via SR300

                                Hi MartyG,


                                Thanks for the important definition about Start and Update functions. I have declared some variables according to the steps you mentioned, however I am getting 2 errors. I have tried to rectify them. Thank you.



                                • 13. Re: Ball controlled from hand gestures via SR300

                                  You have two exact copies of the Update() block in your script.  You can only have one each of a particular function name, so you should delete one of the Update() blocks. 


                                  In the 'Data' error, the word 'float' is missing from it.  It should be:


                                  Data = new float[SampleArrayLength];


                                  Also, where you have put math.abs, it should be:


                                  Mathf.Abs (sample);


                                  That's Math with a capital M, and a small f on the end.

                                  • 14. Re: Ball controlled from hand gestures via SR300

                                    Hi MartyG,


                                    Thank you. It all works now. Thank you very much.

                                    1 2 Previous Next