3 Replies Latest reply on Sep 12, 2017 12:44 AM by MartyG

    SR300 precision & latency for face landmark detection

    pvgyshi

      Based on SR300 & SDK 2016 R2, I developed an application which adjust some display property in real-time according to view-point of end user, i,e., change display property according to current eye position got from SR300 & SDK landmark detection module.

       

      Since viewer would be very sensitive to those display property, we want to get high precision position value with low latency as possible. However, I didn't find those parameters in SR300 product DataSheet (Document: 334531-001), and I also search intel Forums & Community. I got below thread about precision but can't answer my question. https://software.intel.com/en-us/forums/realsense/topic/675499

       

      I also dumped the real data of my eye position when I move my head from right to left (x value goes increases). It shows that value has noise obviously. And I have to use Kalman filter to improve the result. below figure is the comparison of original data & after Kalman-filtered, the left sub-figure is the original one.  I only plot x (horizontal) & z (depth) value.

      I think the random of left sub-figure arise from the measure error of face landmark.

      WeChat Image_20170911213427.jpg

       

       

      Another important thing is the latency, I didn't find answer to discuss what is the max or how to calculate the latency for face landmark detected. I think it would be helpful to estimate the more exact value of current eye position if we can get the real latency. 

      There is another thread in community to talk about such latency of hand detection while also can't answer my question. RealSense SR300 Latency Measuring Method

       

       

       

      All the best.

      yshi

        • 1. Re: SR300 precision & latency for face landmark detection
          MartyG

          It is probable that if there was further data available about latency and precision for the SR300 camera then it would have emerged by now.  The information that is available now is likely the limit of what can be known.

           

          There is a new RealSense SDK called 'SDK 2.0' becoming available when the new super-powerful D415 and D435 cameras launch this month and is said by Intel to be SR300-compatible too.  It is cross-platform but no specific information on OS support for it is available yet.  It is possible that it may require Linux to operate rather than Windows though.  If Linux is an option for you, there may be performance gains in that new SDK though over the previous Windows SDKs.

           

          Intel® RealSense™ Depth Cameras D400-Series | Intel® Software

          • 2. Re: SR300 precision & latency for face landmark detection
            pvgyshi

            Hi Marty,

             

            Thanks for your information!

             

            Regarding to calculating the latency for face landmark detected, I've been wanting to find a way to convert the time value got by QueryFrameTimestamp(), i.e., the time domain of RealSense I guess, to the time value got from Windows.  The QueryFrameTimestamp() would be the time when the latest data (frame) sampled.

            I think if we know how to convert to OS time, then we can know how fresh of the position value when the time we use the position value to control something.  I also write such part code to get timestamp of RealSense sample & Windows system time like below. However I can't find way to convert one the the others.   It would be appreciated if you can tell some tip.

             

             

                class CSystemClock {

                public:

                    double m_dobDff;

               

                    CSystemClock() {

                        LARGE_INTEGER largeInt;

                        QueryPerformanceFrequency(&largeInt);

                        m_dobDff = largeInt.QuadPart;

                    }

               

                    int64_t getCurUs() {

                        LARGE_INTEGER largeInt;

                        QueryPerformanceCounter(&largeInt);

                        int64_t cur = largeInt.QuadPart;

                        cur = cur * 1000 * 1000 / m_dobDff;

                        return cur;

                    }

                };

             

                PXCFaceData *m_pFaceData;

                m_pFaceData->Update();

                //got time stamp of the sample from RealSense

                int64_t qryTime = m_pFaceData->QueryFrameTimestamp();

                qryTime = qryTime / 10000;  //QueryFrameTimestamp unit is 100ns, convert to ms.

             

             

            Best.

            yshi

             

            • 3. Re: SR300 precision & latency for face landmark detection
              MartyG

              Coincidentally, I was doing game programming with the system clock yesterday in the Unity game creation engine.  I don't know if the equation I used will be of any use, but I'll share it anyway.

               

              -  So you take the  24 hour PC clock time,: there should be 86400 seconds in 24 hours.

               

              -  Do %60 on this value to get the number of minutes in a day.

               

              86400 % 60 = 1440

               

              - Then do %60 on the minutes value to get the hours of the day

               

              1440 % 60 = 24

               

              The process works in the opposite direction to convert hours into seconds too on the 24 hour clock, using multiplication instead of %. 

               

              Hours x 60 = minutes, and minutes x 60 = seconds 

               

              ********

              So, if a QueryFrameTimestamp() second is the same length as a PC clock second, maybe a conversion to 24 hour clock could be done that way?

               

              In my game project, I used these equations to construct a timer in the 24 hour clock which counted up from 0, which was 0:00:00 (midnight).  So when it was running, you could see how many seconds had passed in 24 hour clock time  So if 3 seconds passed, the 24 hour clock time in 24 hours was 00:00:03.  If you did a printout of the QueryFrameTimestamp() value  in 24 hour time and the PC clock's seconds count in 24 hour time, you could see how closely the two were aligned, perhaps.

               

              My apologies if this thinking isn't of any use to you!