Person tracking showing center of mass of worlds points x,y,z =0,0,0

[NShai3]

Hi All,

I am using ZR300 on Ubuntu 16.04. I have written a driver for a system which gives images(colors, depth, params) from driver and converts to my format. And in the system I am using Person tracking module to track the person. Below is the procedure I am following.

1. Read Image from device and convert images (RGB, Depth) and params into my format.
2. Convert images and params back to realsense format to use the SDK for persontracking module.

I am able to read rgb, depth images well but the problem is when I use them with person tracking module. Below code might make my problem clear.

Intel::RealSense::PersonTracking::PersonTrackingData::PointCombined centerMass = personTrackingData->QueryCenterMass();after this call "image" coordinates are correct but "world" coordinates are always zero.struct PointCombined{PointInfo image; // These points valid and correctPointInfo world; // Always getting zeroint32_t reserved[10];};

...

[MartyG]

Are you the only person being tracked, or are there multiple people in the camera's vieew? The 'Person Detection' mode in the RealSense SDK For Linux can track all people, but the 'Person Tracking' mode can only track one person. Also, the person should be at least 0.5 m away from the camera, up to a maximum distance of 5 m. Bear in mind too that Center of Mass will stop working beyond 3 m distance or when less than 0.5 m from the camera.

https://software.intel.com/sites/products/realsense/person/developer_guide.html# 2.2_Person_Tracking_ Intel® RealSense™ for Linux - Person Library: Developer's Guide

[NShai3]

Hi MartyG,

Thanks for quick reply. I am tracking all the persons at the moment. I have run the examples(librealsense)and its working fine. But problem is with my system where I am converting realsense to my system back and forth the images and params. I guess some info is missed which is required for the algorithm. I have checked intrinsic and extrinsic params and they are fine. Can you suggest why the problem is only with world coordinates of center of mass. Thanks.

[MartyG]

I wonder if the problem is in the conversion. When converting RealSense to MATLAB .mat format, for example, the image is commonly converted from RealSense's native 16-bit resolution to 8-bit resolution. The user then finds that there are differences in color and scale between the original and converted images, and cannot do a reverse conversion from .mat to RealSense. Does your system convert image data to 8-bit?

[NShai3]

I am using 8 bit unsigned char as RGB format. I will check my changing it to 16 bit format. Thank you.

[MartyG]

Please let us know what happens. Good luck!

[NShai3]

For 8-bit I am using rgb8 format. What is the format for 16 bit rgb for realsense ?. Also I have found that in examples of ROS is also RGB8 format is used. So I guess problem is not with 8 bit.

[MartyG]

According to the table linked to below, the format for 16-bit grayscale is Y16.

https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?pixelformat_pxcimage.html Intel® RealSense™ SDK 2016 R2 Documentation

[NShai3]

I guess problem is not due to RGB8 format because same is used in ros application. Y16 is for Grayscale and person tracking uses color and depth image. Can you suggest which formats are 16 bit colour. Below are the aviable formats.

/// \brief Formats: defines how each stream can be encoded./// \c rs_format specifies how a frame is represented in memory (similar to the V4L pixel format). enum class format : int32_t { any , /**<</span> When passed to enable stream, librealsense will try to provide best suited format */ z16 , /**<</span> 16 bit linear depth values. The depth is meters is equal to depth scale * pixel <span style="col...

[MartyG]

I deeply researched this issue but was only able to find support for an RGB8 mode.

One of the leaders of Librealsense had this advice about getting depth from a 16 bit image:

https://github.com/IntelRealSense/librealsense/issues/509 How to get the real distance from 16-bit depth image pixel value? · Issue # 509 · IntelRealSense/librealsense · GitHub

[NShai3]

I am able to read rgb, depth images well but the problem is when I use them with person tracking module. Below code might make my problem clear.

Intel::RealSense::PersonTracking::PersonTrackingData::PointCombined centerMass = personTrackingData->QueryCenterMass();after this call "image" coordinates are correct but "world" coordinates are always zero.struct PointCombined{PointInfo image; // These points valid and correctPointInfo world; // Always getting zeroint32_t reserved[10];};

[MartyG]

The code that you quoted seems to be correct.

I note that in the Person Tracking module's code, the above struct is preceded earlier in the script by an enum instruction.

enum TrackingState

{

TRACKING_STATE_TRACKING = 0,

TRACKING_STATE_DETECTING

};

https://software.intel.com/sites/products/realsense/person/PersonTrackingData_8h_source.html Intel® RealSense™ for Linux - Person Library: /Users/mcurfman/ws/Linux_SDK/realsense_documentation/realsense_person/sdk/…

I apologize that my knowledge about programming Person Tracking in the Intel RealSense SDK For Linux is limited.

[NShai3]

No problem, I have added some debug info about stream param, it might be useful.

[MartyG]

Thanks a lot. This level of depth on extrinsics is outside of my field of knowledge, but hopefully someone else on this forum who has worked with these values can add some useful commentary.

[NShai3]

Finally I solved the problem. Problem is due to converting the depth image as array of floats and making cv::Mat with CV_32FC1. By using uint16_t and CV_16UC1 and evrerything is working fine.

[MartyG]

That's great news! Thank you very much for sharing the knowledge with the rest of the community.