Depth Value conversion Texture2D Unity3D

[JLaco]

Hi everyone,

I am currently using the intel R200 with the 2016 R2 SDK in Unity3D.

With this SDK, it is is possible to store the depth data into a Texture2D in order to render it ( For instance : https://software.intel.com/sites/default/files/managed/cd/b1/Unity_Capturing_Raw_Streams.pdf https://software.intel.com/sites/default/files/managed/cd/b1/Unity_Capturing_Raw_Streams.pdf ). I wonder : how can I convert a pixel value (Grey-Scale RGBA32) stored in this texture to the depth value in meter?

My goal is to achieve this in a shader.

Thank You,

Jeremy

[MartyG]

Apologies for the delay in responding. I have been carefully considering what you need to do and trying to come up with an appropriate answer for you.

From discussions in the past and present involving others who seek to convert a pixel value into a depth value, it is likely to be too much trouble to do it manually with an equation, as you can get most of the values of the equation but not all of them. It will be easier to use an SDK function to do the conversion automatically for you and then store the result in a variable and use that in your shader.

The QueryVertices instruction may be appropriate for your needs, as it "calculates the vertices from the depth image. The vertices contain the world coordinates in mm."

https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/queryvertices_pxcprojection.html QueryVertices

[JLaco]

Dear MartyG, thank you a lot for your quick and precise answer.

I have tried the QueryVertices method but it has an important process time and I would like to reconstruct the point cloud at 60fps. (For now, I reach 15fps).

I am currently trying this to get the depth in meter :

PXCMImage.ImageData data;

pxcmStatus sts=depth.AcquireAccess(PXCMImage.Access.ACCESS_READ,PXCMImage.PixelFormat.PIXEL_FORMAT_DEPTH_F32, out data);

float[] array = new float[image.info.width * image.info.height];

data.ToFloatArray(0, array);

However, when I try to upload this array in a Texture2D (with depth values normalized between 0 and 1), I only get something that looklikes a noise texture.

I think the problem is caused by the pixel ordrer in the float array. Do you have any idea on how to fix this?

Thank you again.

[MartyG]

Pixel to depth conversion is outside of my field of expertise, sadly as I have never had a need to practice it myself. There are developers on this forum who are very knowledgeable about it though, so hopefully one of them will see your message and offer some expert advice.

I do recognize the code you are using though. It has caused developers problems quite often because it often returns values as an Int instead of a Float. On a recent forum, a contributor offered a script that they said returned the float on SR-300 but not on R200 (which you are using). The current status of the problem is that it is unsolved, and Intel staffer Pablo was going to look into it.

[JLaco]

Thank you again for your answers. I have tried some solutions proposes in the link you provided but unfortunatly it did not work.

Nevertheless I got something really close with the following code :

The texture is created in float format :

_texture = new Texture2D((int)size.width, (int)size.height, TextureFormat.RFloat, false);

I access the depth data in depth float format :

pxcmStatus sts=image.AcquireAccess(PXCMImage.Access.ACCESS_READ,PXCMImage.PixelFormat.PIXEL_FORMAT_DEPTH_F32, out data);

I reimplemented the toTexture2D method :

public Texture2D ToTexture2DTest(PXCMImage.ImageData imageData ,int index, Texture2D image)

{

buffer = new float [((Texture)image).width * ((Texture)image).height * 4];

GCHandle local_0 = GCHandle.Alloc(buffer, GCHandleType.Pinned);

PXCMImage_ImageData_EndianSwapC(imageData.planes[index], imageData.pitches[index], local_0.AddrOfPinnedObject(), ((Texture)image).width *4, ((Texture)image).width, ((Texture)image).height);

local_0.Free();

byte [] finalArray = new byte[buffer.Length * 4];

System.Buffer.BlockCopy(buffer, 0, finalArray, 0, finalArray.Length);

image.LoadRawTextureData(finalArray) ;

return image;

}

For now, this code does not exactly give me the correct depth values but something close to it. Moreover, it also causes some random crashes.

Jeremy

[MartyG]

I'm glad you made significant progress, Jeremy. Please feel free to comment again if you need further assistance!

[JLaco]

I think I found a solution. The values I get in my shader seem ok.

So, for the ones who want to upload the float values of the depth map in a Unity3D Texture2D, these are the steps i followed :

1) The texture must be created in float format :

_texture = new Texture2D((int)size.width, (int)size.height, TextureFormat.RFloat, false);

2) Then access the depth map in float format :

pxcmStatus sts=image.AcquireAccess(PXCMImage.Access.ACCESS_READ,PXCMImage.PixelFormat.PIXEL_FORMAT_DEPTH_F32, out data);

3) Copy the image buffer in a float array

Marshal.Copy(imageData.planes[index], buffer, 0, trueLength);

4) Make a copy in a temporary float array considering the image pitch

int indexBuffer = 0;

int indexNew = 0;

float[] newBufferTest = new float[trueLength];

for (int j = 0; j < image.height; j++)

{

for (int i = 0; i < imageData.pitches[0]/4; i++)

{

if (i < image.width)

{

newBufferTest[indexNew] = buffer[indexBuffer];

indexNew++;

}

indexBuffer++;

}

}

5) Convert this float buffer to a byte array and upload the raw data in the texture

byte [] finalArray = new byte[imageLength * 4];

System.Buffer.BlockCopy(newBufferTest, 0, finalArray, 0, finalArray.Length);

_texture.LoadRawTextureData(finalArray) ;

6) Last, in the shader, perform the mm to m conversion

float depth = tex2D(_MainTex, i.uv).r /1000.0 ;

For now the code is not optimized and not very clean but it seems to work and it can be a good starting point for the ones who want to achieve that.

Jeremy

[MartyG]

Thanks so much for sharing your solution with the community, Jeremy!