7 Replies Latest reply on May 8, 2017 4:28 PM by Intel Corporation

    Problem with landmark points

    André_Filipe

      Greetings

       

      I'm currently working on a project for the University and I'm using the F200 realsense camera with the r3 2016, soon upgrading the camera to the R300, working on visual studio 2013 and programming in C#.

       

      My project consists in using the camera with the face tracking software to detect a yawn but we are having trouble using the software. Visual studio is giving us an error with the Intel namespace not being found.
      We found a forum (https://communities.intel.com/thread/113038) where it was said the namespace error was currently being worked on but we didn't find any updates since then (7th of April).

       


      We also have problems while trying to access to the face's points since the documentation says to do

      "LandmarksData ldata=face.Landmarks; //where face comes from Intel.RealSense.Face.Face face
      FaceData.LandmarkPoint[] points = ldata.Points;"
      but there's no variable called Landmarks on the face object. After some debugging, we found we were using
      "PXCMFaceData.Face = faceOutput.QueryFaceByIndex(0);"
      instead of
      "Intel.realsense.Face.Face variableFace =faceOutput.QueryFaceByIndex(0);"

       

      Is there a specific version/sdk that uses the "Intel" lib? If so how can we have access to it?

       

      Best regards

        • 1. Re: Problem with landmark points
          Intel Corporation
          This message was posted on behalf of Intel Corporation

          Hello André_Filipe,

          Thank you for your interest in Intel® RealSense™ Technology.

          The issue related with the namespace error is currently under analysis, that’s why there hasn’t been an official announcement.

          Could you please share further information regarding your project? What is the documentation you have been using as reference?

          I’ll be waiting for your response.

          Regards,
          Andres V.

          • 2. Re: Problem with landmark points
            André_Filipe

            Hello Andres,

             

            First of all, thank you for the quick reply.

             

            Our project is based on detecting if a person (driver of a car) is sleepy in a way that it can interfere with his/her driving. We also want to see where the person is looking at (for a certain amout of time) and determine if it's related with driving or not. If not, we will alert him/her to pay attention to the road.

            We were thinking about doing the detections based on the landmarks on the face of the driver.

             

            We are using the R3 2016 documentation (Intel® RealSense™ SDK 2016 R3 Documentation).

             

            We program in JAVA and when we started the project, we looked into the R2 2016 SDK, however, we couldn't find any links to download it. So, we decided to use the R3 in C# (although we have never programmed in it before). With this problem of the "Intel" namespace, we might start to use the R2 2016 SDK, where it seems to be a different way of getting the landmark points, with the "face" being from the object "PXCMFaceData".

            If you could provide a link for download the R2 2016 SDK, we would be very thankful.

             

            Best regards,

            André C.

            • 3. Re: Problem with landmark points
              MartyG

              I have some knowledge of the kind of tiredness analysis system that you are developing.  It is gaining traction particularly in trucking and in construction vehicle operation. 

               

              You can download the 2016 R2 from the following link, which launches the 1.8 gb download in your browser immediately.

               

              http://registrationcenter-download.intel.com/akdlm/irc_nas/vcp/9078/intel_rs_sdk_offline_package_10.0.26.0396.exe

               

              Please be aware that web functionality is being removed from RealSense SDKs due to potential security vulnerabilities.  As far as I am aware, it should still function in the R2 though for as long as that SDK is available.  If you are likely to need that SDK and Java functionality for the long term, I recommend burning a copy of the above install file to a backup dvd.  You could download the file to your computer without launching it by right clicking on the above link and selecting Save Target As / Save Link As.  Continued use of the R2 SDK with the known web vulnerability potential will of course be at your own risk.

               

              Intel® RealSense™ Web Component is Discontinued

              • 4. Re: Problem with landmark points
                Intel Corporation
                This message was posted on behalf of Intel Corporation

                Hello André_Filipe,

                 

                Please check the following documentation that explains the current state of the Face tracking and landmark detection features: http://registrationcenter-download.intel.com/akdlm/irc_nas/9560/release_notes_RealSense_SDK_2016_Face_Detection_v11.0.pdf.

                F200.png


                “Alpha Features” imply that the feature is in certain maturity level where the reliability is not guaranteed. There are known issues, but the corresponding fix may not be planned.

                 

                Have a nice day.

                 

                Regards,
                Andres V.

                • 5. Re: Problem with landmark points
                  André_Filipe

                  Hi,

                   

                  I downloaded the R2 2016 SDK software you provided.

                  Now I'm having trouble with the face detection.

                  From what I've been testing, I gathered that it's not possible to have the face tracking module and the stream of the camera on the same sense manager instance.

                  Here's some part of the code I'm using:

                   

                             public void SimplePipeline()

                             {

                              PXCMSenseManager sm = PXCMSenseManager.CreateInstance();

                          

                              if (sm == null) throw new Exception("Failed to create sense manager");

                              if (sm.captureManager == null) throw new Exception("Capture manager does not exist");

                              bool instantMode = false;

                              bool configureStream = true;

                              disconnected = false;

                   

                              /* Set Source & Profile Index */        

                              if (this.form.GetRecordState()) sm.captureManager.SetFileName(this.form.GetFileName(), true);

                            

                              else if (this.form.GetPlaybackState())

                              {

                                  sm.captureManager.SetFileName(this.form.GetFileName(), false);

                                  configureStream = false;

                              }

                   

                              if (configureStream) sm.captureManager.FilterByDeviceInfo(form.GetCheckedDeviceInfo());

                   

                              /* Set Module */

                              sm.EnableTracker(form.GetCheckedModule());

                   

                              /* Initialization */

                              form.UpdateStatus("Init Started");

                   

                              sm.EnableStream(PXCMCapture.StreamType.STREAM_TYPE_COLOR, 640, 480);

                   

                              //Enable the face module

                              sm.EnableFace();

                              faceModule = sm.QueryFace();

                              faceConfig = faceModule.CreateActiveConfiguration();

                   

                              if (sm.Init() >= pxcmStatus.PXCM_STATUS_NO_ERROR)

                              {

                                  form.UpdateStatus("Streaming");

                                  form.timer = new FPSTimer(form);

                                  PXCMCaptureManager captureManager = sm.QueryCaptureManager();

                                  if (captureManager == null) throw new Exception("Failed to query capture manager");

                                  PXCMCapture.Device device = captureManager.QueryDevice();

                   

                                  if (device != null && !this.form.GetPlaybackState()) device.SetDepthConfidenceThreshold(7);

                   

                                  if (sm.captureManager == null)  throw new Exception("Failed to query capture manager");

                   

                                  if (sm.captureManager.QueryDevice() == null) throw new Exception("Invalid device");

                                                 

                                  // Configure for 3D face tracking

                                  faceConfig.SetTrackingMode(PXCMFaceConfiguration.TrackingModeType.FACE_MODE_COLOR_PLUS_DEPTH);

                   

                                  // Known issue: Pose isEnabled must be set to false for R200 face tracking to work correctly

                                  faceConfig.pose.isEnabled = true;

                                 

                                  // Track faces based on their appearance in the scene

                                  faceConfig.strategy = PXCMFaceConfiguration.TrackingStrategyType.STRATEGY_APPEARANCE_TIME;

                   

                                  // Set the module to track four faces in this example

                                  faceConfig.detection.maxTrackedFaces = 50;

                   

                                  // Enable alert monitoring and subscribe to alert event hander

                                  faceConfig.EnableAllAlerts();

                                  faceConfig.SubscribeAlert(FaceAlertHandler);

                   

                                  // Apply changes and initialize

                                  faceConfig.ApplyChanges();

                                  faceData = faceModule.CreateOutput();

                   

                                  while (!form.stop)

                                  {

                                      pxcmStatus acquireStatus = sm.AcquireFrame(true);

                   

                                      if (acquireStatus < pxcmStatus.PXCM_STATUS_NO_ERROR) break;

                   

                                      if (!DisplayDeviceConnection(!sm.IsConnected()))

                                      {

                                          /* Display Results */

                                          PXCMCapture.Sample sample = sm.QuerySample();

                                          Bitmap colorBitmap;

                                          PXCMImage.ImageData colorData;

                                          sample.color.AcquireAccess(PXCMImage.Access.ACCESS_READ, PXCMImage.PixelFormat.PIXEL_FORMAT_RGB32, out colorData);

                                          colorBitmap = colorData.ToBitmap(0, sample.color.info.width, sample.color.info.height);

                   

                                          if (sample == null) continue;

                                          if (sample.color == null) continue;

                   

                                          DisplayPicture(sample.color);

                   

                                          // Acquire face data

                                          if (faceData != null)

                                          {

                                              faceData.Update();

                                            

                                              facesDetected = faceData.QueryNumberOfDetectedFaces(); // Always 0...

                   

                                              if (facesDetected > 0)

                                              {

                                                  // Get the first face detected (index 0)

                                                  PXCMFaceData.Face face = faceData.QueryFaceByIndex(0);

                   

                                                  // Retrieve face location data

                                                  PXCMFaceData.DetectionData faceDetectionData = face.QueryDetection();

                                              }

                                          }

                                      }

                                      sm.ReleaseFrame();                   

                                  }

                                  faceConfig.Dispose();

                                  faceModule.Dispose();

                              }

                              else  form.UpdateStatus("Init Failed");

                             

                              sm.Close();

                              sm.Dispose();

                          }

                   

                   

                  When I try to run both the face tracking and the camera stream on the same "sm" instance, the sm.Init() gives the following state: "PXCM_STATUS_ITEM_UNAVAILABLE".

                  How can we have both modules working at the same time? I did try to run in different Sense Manager instances and it worked, however, it wasn't able to detect any faces that way.

                  How can we detect the faces as well?

                   

                  Thank you,

                  André

                  • 6. Re: Problem with landmark points
                    André_Filipe

                    Hello,

                     

                    Here are some updates.

                    I did some more research and I finally solved the problems I described here before (having the face tracking module and the stream of the camera working at the same time and the landmark points).

                     

                    Thank you for your time and help,

                    André

                    • 7. Re: Problem with landmark points
                      Intel Corporation
                      This message was posted on behalf of Intel Corporation

                      Hello André,

                      I’m glad to hear that you could solve the issue.

                      Thank you for sharing your experience with the community, we really appreciate it.

                      Have a nice day.

                      Regards,
                      Andres V.