6 Replies Latest reply on Jun 20, 2017 4:59 PM by Intel Corporation

    SR300 - crash while facetracking multiple people


      Hi everyone,


      I'm trying to make a face tracking app for my Bachelor degree.

      The app will run in an artistic gallery, which means running 16 hours straight, with people experiencing the installation at anytime during these 16 hours, without supervision of any kind. Any crash of the app will completely disrupt the experience of the user, so we can't afford this.


      Thing is, if you read the title you already know that the app is still crashing at the moment.

      So here's a description of what I need, the conditions in which the bug happens, the conditions in which the bug seems not to trigger, the info I could get on that bug and what I tried / will try to work around it.


      Note : I'm working with a SR300, the essentials and face SDKs, and I'm using C++ with Qt on Windows 10.



      What I need


      I need an app that can retrieve the following features of one's face :

      - position x and y

      - width and height

      In short, the bounding box of the face. Pretty basic stuff, here's an example to illustrate this :




      One and only one person should be tracked at anytime, meaning that even though the app must support multiple people in the camera's FOV, I only need one face bounding box at a time.

      And here's the problem : multiple people can be in the FOV at the same time, they'll be detected, but I always track the first face in the list of all faces detected. But sometimes, the app crashes.



      Conditions in which the bug happens


      First things first, we need to make the difference between detected and tracked.

      The camera can detect multiple faces (meaning recognize "oh hey see there are 3 faces in here") and track only one (meaning retrieve the infos of position and size of only one face).


      Here are the conditions in which I could trigger the bug every time.

      When 3 faces are detected, if the tracked one leaves the FOV, the app crashes. Here's an illustration :




      Here you can see the video stream at the time of the crash, right when I leave the FOV of the camera. I could simulate the multiple people with this image of me and my girlfriend (only thing I had, but it worked alright so don't judge ).

      As you can see on the right-hand side of the image, "Nb tracked people" equals 3. This should be named "Nb detected people" to be consistent with the definition above btw. This is directly the value returned by SomeFaceData->QueryNumberOfDetectedFaces() at the moment of the crash.

      This only happens if it's the currently tracked face that leaves the FOV, but any of the detected but untracked faces cane leav the FOV as they wish, it does not trigger a crash.


      I could recreate this crash (or at least make crash, maybe for a totally unrelated reason) with the C++ face tracking exmaple, found in the samples of the SDK.



      Conditions in which the bug seems not to trigger


      This same bug has been triggered with only 2 people detected (and maybe even one), but it's rare. I could not find a setup to trigger the bug consistently without at least 3 faces detected.

      This bug doesn't seem to trigger more often when doing kissing-like gestures. I'm saying this because I read this thread : Re: Crash when two people are doing kissing-like gesture and coming close to each other  but it doesn't seem to be related. Can't be sure though.



      Infos I could get on the bug


      Here are all the infos I could gather on this bug.


      First of all, the screen I get in QtCreator when the app is launched in Debug mode :




      Coming with its full cryptic stacktrace :




      We can't see it clearly in here, but every single one of these calls appears in assembly only when selected. There is no decently readable code to go to and say "There's a pointer access that you didn't check for nullptr !", unless you debug ASM code for breakfast, in which case I welcome your input on the matter.


      Finally, my bug was strangely close to this one : SR300 + R3 SDK + Unity 3D 5.6.0 Crash !!! , even though I'm not using Unity at all. I can't tell if they're related at all though.



      What I tried / will try to work around the bug


      I tried to limit the number of detected faces to one.

      I made sure to only track one face in the config of the camera like this :


          pxcI32 nbFaces(1);

          m_config->detection.isEnabled = true;

          m_config->detection.maxTrackedFaces = nbFaces;




      I thought that this would limit the number of detected faces, but instead limits the number of faces I can track (apparently, this is not stated in the documentation, it's just my personnal observation).


      Later in my code, I verify it like this :


          int nbFaces = SomeFaceData->QueryNumberOfDetectedFaces();                  // nbFaces = 0, 1, 2 or 3...

          FaceData::Face* trackedFace = SomeFaceData->QueryFaceByIndex(0);        // Always take the first face in the list.


      If I did :


          FaceData::Face* trackedFace = SomeFaceData->QueryFaceByIndex(1);


      for example, this would trigger an exception when trying to access trackedFace, or its properties.


      This doesn't change much, because of the difference between a detected face and a tracked face. It's when, after writing this configuration code, the behavior of my app didn't change that I understood the difference.


      So this is as far as I could go into forcing my app to track one and only one person, but even with these precautions, it still crashes with multiple people in the FOV.


      What I will try finally is to detect when the tracked face is coming close to an edge and force switching to another face when this happens. This is not wonderful because it reduces the FOV for tracking only, and if someone moves faster than expected, depending on the framerate too, it might still break .



      To recap


      I have a bug which seems not to depend much on my code since nothing readable appears in the stack trace and I could reproduce it with the default SDK face tracking sample.

      The bug makes the app crash, and triggers consistently when 3 faces are detected and the tracked one leaves the FOV of the camera.


      Any idea what triggers this bug, and how to avoid it ? Is it an open issue (I couldn't find one related on github) ?




      Thank you in advance, and sorry if my english is approximative sometimes.

      It's a long post, but I wanted it as complete as possible.