9 Replies Latest reply on Aug 29, 2018 2:42 AM by jb455

    Problem on face tracking from a video

    Spied

      Hello everyone,

      I'm trying to use the face tracking solution from the SDK for tracking expression and pulse from a video. I use openCV to get each image in the video, then convert them to PXCImage and feed them to face module to process. But I meet a problem that when I use 2D mode, and the "Pulse" module is selected, the first frame is OK but from the next frame, it can't find any face though the picture is the same as the first frame. Everything is OK when the mode is "3D IR" or the "Pulse" module is not selected.

       

      Is there anyone who have met the same problem?

      Thanks.

        • 1. Re: Problem on face tracking from a video
          MartyG

          Could you tell me which RealSense SDK you are using please?   Is it '2016 R2' or '2016 R3', or is it the current RealSense SDK 2.0.  Thanks!

          • 2. Re: Problem on face tracking from a video
            Spied

            Hello,

            The SDK I'm using is "2016 R2".

            • 3. Re: Problem on face tracking from a video
              MartyG

              Thank you very much for the confirmation.

               

              Is the pulse module that you are using the Pulse Estimation module that comes with the R2 SDK?  Or is it an OpenCV pulse module?

               

              Intel® RealSense™ SDK 2016 R2 Documentation

              • 4. Re: Problem on face tracking from a video
                Spied

                I'm sorry what I said is unclear. I think I'm using the Pulse Estimation module that comes with the R2 SDK. Actually I select the "Pulse" check box on the interface of the face tracking sample solution.

                Then I set "Profile" to "2D", and set "Mode" to "playback". I choose a playback file and click "Start" button. While it is processing, I create a picture and convert it to PXCImage. Then I feed it to a sample and process it with

                ProcessSample(PXCCapture::Sample *sample)

                Thank you for being so patient.

                • 5. Re: Problem on face tracking from a video
                  MartyG

                  I tried replicating your test with the software that I have.  I ran the Face Tracking (C++) program with my SR300 and used it to record an RSSDK file in 2D mode with pulse selected.  Then I loaded the file into Playback mode with the same settings and it worked fine.

                   

                  Since I am not using your PXC processing routine, it is likely that the problem is with that aspect of the project.

                   

                  Edit: I have to go out of office for an hour.   I will reply to any new comments as soon as I'm back.  Thanks!

                  • 6. Re: Problem on face tracking from a video
                    Spied

                    What I do to replace sample is:

                    PXCCapture::Sample* sample = senseManager->QueryFaceSample();

                    This is the sample code in the solution. Replace it to:

                    PXCCapture::Sample sample = {};

                    PXCImage::ImageInfo iinfo = {};

                    memset(&iinfo, 0, sizeof(iinfo));

                    iinfo.width = 1280;

                    iinfo.height = 720;

                     

                    //Read raw file and convert it to PXCImage

                    PXCImage::PixelFormat format = PXCImage::PIXEL_FORMAT_Y8_IR_RELATIVE;

                    iinfo.format = format;

                    PXCImage::ImageData data = {};

                    data.format = format;

                    FILE* pFile;

                    pFile = fopen("e:\\ir_result.dat", "rb");//this is my raw file

                    long len = 1280*720*1;

                    pxcBYTE* planes0 = new pxcBYTE[len];

                    memset(planes0, 0, len);

                    fread(planes0, 1, len, pFile);

                    fclose(pFile);

                    data.planes[0] = planes0;

                    data.pitches[0] = (iinfo.width * 1);

                    PXCImage * pxcImagec = senseManager->QuerySession()->CreateImage(&iinfo, &data);

                     

                    //feed sample and process it

                    PXCImage::ImageInfo info = pxcImagec->QueryInfo();

                    sample.color = session->CreateImage(&info);

                    sample.color->CopyImage(pxcImagec);

                    pxcImagec->Release();

                     

                     

                    vm->ProcessSample(&sample);

                    Then in DrawBitmap function, replace "sample" to "&sample"

                     

                    Can you help me find what is the problem? I find several days but I can't find the problem.

                    Thanks.

                    • 7. Re: Problem on face tracking from a video
                      MartyG

                      I do not have the ability to test scripts on my PC, unfortunately.  I will link RealSense stream programming expert jb455 into this discussion for his input, as his scripting knowledge goes back to that earlier SDK's language.

                      • 8. Re: Problem on face tracking from a video
                        Spied

                        Thank you very much!

                        • 9. Re: Problem on face tracking from a video
                          jb455

                          I never used face tracking so can't help, sorry. Best suggestion I can make is to avoid using the old SDK altogether as it's no longer supported. Better to use the new SDK for obtaining the camera streams then third party libraries like OpenCV for anything else you need.