Items with no label
3335 Discussions

convert depth image to point cloud

fwang22
Beginner
7,970 Views

How to convert depth frame to point cloud using Matlab? I know one way is if we know the camera parameters,we can obtain point cloud .But what are the parameters ?or Is there any other way to do it in Matlab?

0 Kudos
51 Replies
MartyG
Honored Contributor III
2,944 Views

If you are using Windows, in the most recent 2.16.0 version of the RealSense SDK, a MATLAB wrapper was introduced. This contains point cloud support..

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/matlab/pointcloud.m librealsense/pointcloud.m at master · IntelRealSense/librealsense · GitHub

You can find the front page of the wrapper here:

https://github.com/IntelRealSense/librealsense/tree/master/wrappers/matlab librealsense/wrappers/matlab at master · IntelRealSense/librealsense · GitHub

If you are not using Windows, the discussion below looks at the process from the opposite direction (converting a point cloud to depth) and may provide some useful insights.

https://github.com/IntelRealSense/librealsense/issues/2204 Convert pointcloud to Depth image (3d to 2d) · Issue # 2204 · IntelRealSense/librealsense · GitHub

0 Kudos
fwang22
Beginner
2,944 Views

Hi,MartyG,

I triesd the above ways,but I'm so sorry that I still don't know how to convert depth frame to point cloud ,the 2.16.0 version does not appear a program about converting depth frame into point clouds.

Best Regards!

0 Kudos
MartyG
Honored Contributor III
2,944 Views

I have not used the MATLAB wrapper myself. If it works like the SDK's other wrappers though then it would give access via MATLAB to functions in the core SDK such as point cloud creation. The problem tends to be that until example programs or script snippets for a particular function are published for a wrapper then it is hard to know how to write a script for the wrapper that produces the desired result. Such samples emerge over time, whether published by Intel or a member of the community. I understand that this does not help you today though.

I did some further research. On the page linked to below, there is a MATLAB function that can be downloaded that converts a depth image to a point cloud in MATLAB. Scroll down the page to the heading titled 'Depth Image to Point Cloud (MATLAB)' and click on the 'depthToCloud.m' link to launch its download in your browser.

https://rgbd-dataset.cs.washington.edu/software.html RGB-D (Kinect) Object Dataset

0 Kudos
fwang22
Beginner
2,944 Views

Hi, Marty

Fortunately, the approach you recommend is feasible, but as I said earlier, converting a depth image to a point cloud requires some camera parameters, such as:

focal length:

I saw two focal lengths in the camera data,I don't know which one I choose,

the other parameter :The physical dimensions of each pixel on the x and y axes of the image plane。That is the red part of the formula.

For clarity, here is the conversion formula:

0 Kudos
bche
Beginner
2,943 Views

Hi,

Why must you use Matlab? C++ code is here and like this:

/librealsense-2.16.0/examples/pointcloud/rs-pointcloud.cpp

// Declare pointcloud object, for calculating pointclouds and texture mappings rs2::pointcloud pc; // We want the points object to be persistent so we can display the last cloud when a frame drops rs2::points points; // Declare RealSense pipeline, encapsulating the actual device and sensors rs2::pipeline pipe; // Start streaming with default recommended configuration pipe.start(); auto frames = pipe.wait_for_frames(); auto depth = frames.get_depth_frame(); // Ge...
0 Kudos
MartyG
Honored Contributor III
2,943 Views

The D415's focal length is 1.88 mm and the D435's focal length is 1.93 mm.

The Dx / DY question is a harder one. It has been said that it is dependent on sensor size. The rolling shutter of the D415 has a pixel size of 1.4um x 1.4 um and the global shutter of the D435 has a sensor size of 3um x 3 um.

RealSense expert Samontab once gave the following explanation of pixel size:

https://software.intel.com/en-us/forums/realsense/topic/671453# comment-1879445 R200 pixel size

0 Kudos
fwang22
Beginner
2,943 Views

Thank you very much! This is very useful to me. Because I have been working on matlab, and I am not very familiar with c++, so I want to use matlab. I'll try to use c++.

Thanks again!

Bset reagrds!

0 Kudos
fwang22
Beginner
2,943 Views

Hi,I have run the above program and researched ahout it ,but it only sample one point cloud .Unfortunately,that's not what I'm trying to accomplish!

Thank you!

0 Kudos
idata
Employee
2,943 Views

Hello Nevyn,

 

 

A .ply represents a pointcloud format that contains the information from only one frame.

 

If this is not what you are trying to accomplish, could you please give us more details in order to help you?

 

 

Thank you best regards,

 

Eliza

 

0 Kudos
fwang22
Beginner
2,943 Views

Hi Eliza,

I want to get color image and corresponding point cloud video sequence, but the only way is converting the depth image to point cloud.I know the transformation formual ,but don't know the camer paremeters.

Do you have other ways to finish it.

Thanke you,

Nevyn

0 Kudos
idata
Employee
2,943 Views

Hello Nevyn,

 

 

Thank you for the clarifications!

 

 

In this case you are interested in knowing the Intrinsic Parameters, you can read about it on https://github.com/IntelRealSense/librealsense/wiki/Projection-in-RealSense-SDK-2.0# intrinsic-camera-parameters Intrinsic Camera Parameters 2. Deprojection.

 

 

You might also find https://github.com/IntelRealSense/librealsense/issues/1706 this GitHub thread helpful.

 

 

Please let us know if this information helps you with your project!

 

 

Thank you and best regards,

 

Eliza
0 Kudos
fwang22
Beginner
2,943 Views

Hello Eliza,

When I start the Intel RealSense Viewer use 'record' to recoed a sequence , I get the error message:

Unknown in rs2_create_record_device(device:0000020C63E7EB90, file:C):

Error opening file: C:\Users\\20180912_155849.bag

I don't know how to do it ,can you help me?

Thanks.

0 Kudos
idata
Employee
2,943 Views

Hello Nevyn,

 

 

This seems to be a permission error.

 

If you save a snapshot, do you received the same error?

 

 

Thank you,

 

Eliza
0 Kudos
fwang22
Beginner
2,943 Views

Hello Eliza,

I have solved the problem,but I still need to look at the previous questions about camera parameters.

Thank you,

Nevyn

0 Kudos
fwang22
Beginner
2,943 Views

Hello Eliza,

Fortunately, the information is useful, but I am not familiar with C program, do not know how to use the above files.

Please forgive me for always wanting to know the camera parameters in order to MATLAB code. Because I don't do program development,

I just do some research using it.

Thank you !

Best Regards,

Nevyn

0 Kudos
idata
Employee
2,943 Views

Hello Nevyn,

 

 

You can check out https://github.com/IntelRealSense/librealsense/blob/master/wrappers/matlab/video_stream_profile.m Video Stream Profile Matlab example, as it contains a function that gets the intrinsic parameters from the camera.

 

 

I hope this information helps you!

 

 

Thank you,

 

Eliza
0 Kudos
idata
Employee
2,943 Views

Hello Nevyn,

 

 

Could you please let us know if the example provided helped you?

 

 

Thank you,

 

Eliza
0 Kudos
fwang22
Beginner
2,943 Views

Hi Eliza,

Unfortunately, I didn't find he parameters I wanted. and I tried another way, but it didn't work.I still hope to use matlab to convert the depth chart point cloud, I don't know you have a better way?

Thank you,

Nevyn

0 Kudos
idata
Employee
2,943 Views

Hello Nevyn,

Could you please let us know exactly what are the parameters that you are interested in?

 

From one of the posts you made on this thread, I understood that you are interested in knowing:

 

Focal Length: which for D435 is 1.93mm and for D415 is 1.88mm. You can find this information in the https://www.intel.com/content/dam/support/us/en/documents/emerging-technologies/intel-realsense-technology/Intel-RealSense-D400-Series-Datasheet.pdf datasheet, at page number 32.

 

Physical dimensions of each pixel on the x and y axes of the image plane: The image sensor for D415 has the pixel size 1.4um x 1.4um and the Image sensor for D435 has the pixel size 3um x 3um. You can find this information https://newsroom.intel.com/wp-content/uploads/sites/11/2018/01/realsense-d415-d435-fact-sheet.pdf here at page number 2.

Besides this, what other parameters are you interested to know?

Thank you in advance,

 

Eliza
0 Kudos
fwang22
Beginner
2,494 Views

Hi Eliza ,

My goal is to use matlab to turn the depth chart into a point cloud,so I need the parameters of the camera.

But through yesterday's communication, I have implemented this function by rs-convert.Unfortunately, there are also a problem about the program.

When I use this program to convert to a PNG file, there will be inconsistency between the number of depth images and color images.so this problem needs to be solved.

 

Thank you ,

Best Regards,

Nevyn

0 Kudos
Reply