12 Replies Latest reply on May 21, 2017 1:55 AM by jflipts

    GStreamer depth data

    jflipts

      Hi

       

      I'm trying to capture depth data from the intel realsense on the rtf drone in order to do some processing on a workstation. First I setup a SSH connection and then I'm able to set up a stream but for the moment it's pretty useless. On the workstation I execute following command:

      gst-launch-1.0 -e udpsrc port=5013 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, a-framerate=(string)30.000000, a-framesize=(string)640-480" ! rtph264depay ! avdec_h264 ! autovideosink

       

      And on the drone:

      gst-launch-1.0 v4l2src  device=/dev/video12 do-timestamp=true  ! autovideoconvert ! vaapih264enc ! rtph264pay !  udpsink host=192.168.1.3 port=5013

       

      With these commands a window will open on the workstation and a distorted green capture will be displayed. See image below.

      Screenshot from 2017-05-01 18-03-44.png

      I was wondering how to adjust the command in order to have proper dimensions and not have multiple layers above each other. Second I'm actually only interested in the integer depth values so I'd like to know how to write to a file (I assume filesink) and what file format to use.

       

      Kind regards,

      Jeroen Flipts

        • 1. Re: GStreamer depth data
          Intel Corporation
          This message was posted on behalf of Intel Corporation

          Hello Jeroen,

           

          Thank you for your interest in the Intel® Aero Compute Board.

           

          I saw that you have been participating in this other thread https://communities.intel.com/thread/113443, as community member lstrano suggested, you should update the drone (https://github.com/intel-aero/meta-intel-aero/wiki/02-Initial-Setup) before we can attempt any troubleshooting, since the issue you are experiencing may be related to a previous image version.
          I’ll be waiting for your response.

           

          Have a nice day.

           

          Regards,
          Andres V.

          • 2. Re: GStreamer depth data
            Intel Corporation
            This message was posted on behalf of Intel Corporation

            Hello Jerorn,

            I saw in this thread https://communities.intel.com/thread/113443 that you ordered a USB OTG cable.

            Please get back to us if you keep experiencing the GStreamer issue after flashing the board with the latest image.

            If you have any other question or update, don’t hesitate to contact us.

            Regards,
            Andres V.

            • 3. Re: GStreamer depth data
              jflipts

              Dear Andres

               

              Over the weekend I updated my drone to the latest version as well as the flight controller. If I execute the commands given above I still get the same green stream. Is it possible to look further into this. Or do you know which commands should work?

               

              Kind regards

              Jeroen Flipts

              • 4. Re: GStreamer depth data
                Intel Corporation
                This message was posted on behalf of Intel Corporation

                Hello Jeroen,

                I’m glad to hear that you could update your drone.

                Regarding the GStreamer issue you are experiencing, I would like to try and reproduce it, could you please provide the specifications of your host PC/laptop (particularly the OS you are using)?

                I assume you are following this guide https://github.com/intel-aero/meta-intel-aero/wiki/WiFi-Streaming, have you tried WiFi streaming using QGroundControl?  

                I’ll be waiting for your response.

                Have a nice day.

                Regards,
                Andres V.

                • 5. Re: GStreamer depth data
                  jflipts

                  Hi Andres

                   

                  Thanks for the reply and the support I'm getting on the forums.

                   

                  I'm working on Ubuntu 16.04.

                  For this question I started from the GStreamer commands stated on this page of the wiki: WiFi Streaming · intel-aero/meta-intel-aero Wiki · GitHub . I adapted the commands for wifi streaming until I got a working stream. My best guess for whats going wrong is that I'm missing the video format and the width an height parameters from the outgoing stream on the yocto.

                   

                  On this page of the wiki: RealSense · intel-aero/meta-intel-aero Wiki · GitHub there is a camera preview command for the depth data. When I try to execute this command on the yocto, I get the following output:

                  root@intel-aero:~# gst-launch-1.0 v4l2src device=/dev/video11 ! xvimagesink

                  Setting pipeline to PAUSED ...

                  ERROR: Pipeline doesn't want to pause.

                  ERROR: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: Could not initialise Xv output

                  Additional debug info:

                  ../../../gst-plugins-base-1.8.2/sys/xvimage/xvimagesink.c(1760): gst_xv_image_sink_open (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:

                  Could not open display (null)

                  Setting pipeline to NULL ...

                  Freeing pipeline ...

                   

                  Kind regards

                  Jeroen Flipts

                  • 6. Re: GStreamer depth data
                    Intel Corporation
                    This message was posted on behalf of Intel Corporation

                    Hello Jeroen,

                    Thank you for the information provided.

                    I’ll check the guide and try to reproduce the issue.

                    As soon as I have relevant information I’ll contact you through this post.

                    In meantime, if you have any update regarding the issue you are experience, don’t hesitate to share it.

                    Have a nice day.

                    Regards,
                    Andres V.

                    • 7. Re: GStreamer depth data
                      jflipts

                      Hi

                       

                      I looked a bit further into it and came to the following assumptions. When I run the command from in the picture I get the supported formats from the depth camera device.Video12 formats.png

                      Gstreamer will default to the first format, but I was wondering why this has size 314x938. In my eyes this doesnt make sense, but it could explain why the green Image from the original question looks like it has multiple layers on top of eachother. I'd guess 4 because this would get an aspect ratio of 4/3. So basically the captured image is split divided into 4 bands an all are placed on top of eachother. This is only a theory though, but if true would there be a way to split the layers of the image so the original could be created.

                       

                      A second thing I can deduct form the command image above is that the format Z16 is supported by the camera. I have no idea how i can possibly get the stream into Z16 format because I haven't found any reference that it is supported by Gstreamer. The only documentation I find about the Z16 format is that it is an unsigned 16-bit integer per pixel mapped linearly to depth. Actually this is exactly what I'd like to pull from the camera. I assumed this would be a basic feature because what would otherwise be the point of putting a depth camera on the drone. If you buy the standalone r200 you can use the sdk to acces the point cloud but with the aero on yocto I haven't found a way.

                       

                      Any suggestions would be appreciated.

                       

                      Kind regards

                      Jeroen Flipts

                      • 8. Re: GStreamer depth data
                        Intel Corporation
                        This message was posted on behalf of Intel Corporation

                        Hello Jeroen,

                        I have an update regarding this topic.

                        I ran the following commands:
                        sh-4.3# gst-launch-1.0 v4l2src device=/dev/video11 ! xvimagesink

                        And didn’t get any error message, the depth camera output was as expected (green stream display, 320x962)

                        Could you please provide the output to the following command?
                        aero-get-version.py

                        I’ll be waiting for your response.

                        Have a nice day.

                        Regards,
                        Andres V.

                        • 9. Re: GStreamer depth data
                          jflipts

                          Hi Anders

                           

                          So i ran the following command gst-launch-1.0 v4l2src device=/dev/video11 ! xvimagesink and got this error:

                          root@intel-aero:~# gst-launch-1.0 v4l2src device=/dev/video11 ! xvimagesink Setting pipeline to PAUSED ... ERROR: Pipeline doesn't want to pause. ERROR: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: Could not initialise Xv output Additional debug info: ../../../gst-plugins-base-1.8.2/sys/xvimage/xvimagesink.c(1760): gst_xv_image_sink_open (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: Could not open display (null) Setting pipeline to NULL ... Freeing pipeline ...

                           

                          When I run aero-get-version.py I get this:

                          root@intel-aero:~# aero-get-version.py BIOS_VERSION = Aero-01.00.12_Prod OS_VERSION = Poky Aero (Intel Aero Linux Distro) 1.3.1 (krogoth)" AIRMAP_VERSION = 1.8 FPGA_VERSION = 0xff

                           

                           

                           

                           

                          • 10. Re: GStreamer depth data
                            Intel Corporation
                            This message was posted on behalf of Intel Corporation

                            Hello Jeroen, 

                            Thank you for the information provided.

                            I’ll proceed to analyze the error message that you got.

                            As soon as I find useful information I’ll post it here.

                            Thank you for your patience.

                            Regards,
                            Andres V.

                            • 11. Re: GStreamer depth data
                              Intel Corporation
                              This message was posted on behalf of Intel Corporation

                              Hello Jeroen,

                               

                              I have been trying to replicate the issue you experienced, but the output I get is the same as the one I mentioned in the previous post. So I informed about this to the RealSense engineering team, to see if they know the reason behind this, to which they mentioned that GStreamer is not supported by RealSense.

                               

                              With the previous in mind, the alternative is to use librealsense to access the camera. I also found an example that creates a GStreamer node that gets data from librealsense, I haven’t tested it, but if you are interested in giving it a try, please follow this link: https://github.com/01org/camera-streaming-daemon/blob/master/samples/stream_realsense.cpp.

                               

                              Have a nice day.

                               

                              Regards,
                              Andres V.

                              • 12. Re: GStreamer depth data
                                jflipts

                                Hello Anders

                                 

                                Our project with the Intel Aero drone has been terminated. Though we were unable to succesfully stream depth data from the drone to a workstation, I'd like to thank you for your help and time. You can mark the thread as completed if you wish to.

                                 

                                Kind regards

                                Jeroen Flipts