Problem on face tracking from a video

[ZWang41]

Hello everyone,

I'm trying to use the face tracking solution from the SDK for tracking expression and pulse from a video. I use openCV to get each image in the video, then convert them to PXCImage and feed them to face module to process. But I meet a problem that when I use 2D mode, and the "Pulse" module is selected, the first frame is OK but from the next frame, it can't find any face though the picture is the same as the first frame. Everything is OK when the mode is "3D IR" or the "Pulse" module is not selected.

Is there anyone who have met the same problem?

Thanks.

[MartyG]

Could you tell me which RealSense SDK you are using please? Is it '2016 R2' or '2016 R3', or is it the current RealSense SDK 2.0. Thanks!

[ZWang41]

Hello,

The SDK I'm using is "2016 R2".

[MartyG]

Thank you very much for the confirmation.

Is the pulse module that you are using the Pulse Estimation module that comes with the R2 SDK? Or is it an OpenCV pulse module?

https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?doc_face_pulse_estimation_data.html Intel® RealSense™ SDK 2016 R2 Documentation

[ZWang41]

I'm sorry what I said is unclear. I think I'm using the Pulse Estimation module that comes with the R2 SDK. Actually I select the "Pulse" check box on the interface of the face tracking sample solution.

Then I set "Profile" to "2D", and set "Mode" to "playback". I choose a playback file and click "Start" button. While it is processing, I create a picture and convert it to PXCImage. Then I feed it to a sample and process it with

ProcessSample(PXCCapture::Sample *sample)

Thank you for being so patient.

[MartyG]

I tried replicating your test with the software that I have. I ran the Face Tracking (C++) program with my SR300 and used it to record an RSSDK file in 2D mode with pulse selected. Then I loaded the file into Playback mode with the same settings and it worked fine.

Since I am not using your PXC processing routine, it is likely that the problem is with that aspect of the project.

Edit: I have to go out of office for an hour. I will reply to any new comments as soon as I'm back. Thanks!

[ZWang41]

What I do to replace sample is:

PXCCapture::Sample* sample = senseManager->QueryFaceSample();

This is the sample code in the solution. Replace it to:

PXCCapture::Sample sample = {};

PXCImage::ImageInfo iinfo = {};

memset(&iinfo, 0, sizeof(iinfo));

iinfo.width = 1280;

iinfo.height = 720;

//Read raw file and convert it to PXCImage

PXCImage::PixelFormat format = PXCImage::PIXEL_FORMAT_Y8_IR_RELATIVE;

iinfo.format = format;

PXCImage::ImageData data = {};

data.format = format;

FILE* pFile;

pFile = fopen("e:\\ir_result.dat", "rb");//this is my raw file

long len = 1280*720*1;

pxcBYTE* planes0 = new pxcBYTE[len];

memset(planes0, 0, len);

fread(planes0, 1, len, pFile);

fclose(pFile);

data.planes[0] = planes0;

data.pitches[0] = (iinfo.width * 1);

PXCImage * pxcImagec = senseManager->QuerySession()->CreateImage(&iinfo, &data);

//feed sample and process it

PXCImage::ImageInfo info = pxcImagec->QueryInfo();

sample.color = session->CreateImage(&info);

sample.color->CopyImage(pxcImagec);

pxcImagec->Release();

vm->ProcessSample(&sample);

Then in DrawBitmap function, replace "sample" to "&sample"

Can you help me find what is the problem? I find several days but I can't find the problem.

Thanks.

[MartyG]

I do not have the ability to test scripts on my PC, unfortunately. I will link RealSense stream programming expert jb455 into this discussion for his input, as his scripting knowledge goes back to that earlier SDK's language.

[ZWang41]

Thank you very much!

[jb455]

I never used face tracking so can't help, sorry. Best suggestion I can make is to avoid using the old SDK altogether as it's no longer supported. Better to use the new SDK for obtaining the camera streams then third party libraries like OpenCV for anything else you need.