1 2 Previous Next 20 Replies Latest reply on May 3, 2017 4:40 AM by jb455

    Getting float depth data

    jb455

      Hi,

       

      I'm trying to obtain float depth data, as the rounded/truncated integer depth data I'm currently getting is causing issues due to the 'stepping' (discontinuity when it goes from one millimetre to the next).

       

      I remember a while ago there was an issue with the then-current SDK that QueryVertices was returning rounded/truncated integers rather than floats for depth - that is no longer the case with the 2016R2 SDK; thanks for fixing that, but:

       

      I'm currently using the folowing code to get depth data:

      var mappedImage = projection.CreateDepthImageMappedToColor(depth, , image);
      mappedImage.AcquireAccess(PXCMImage.Access.ACCESS_READ, PXCMImage.PixelFormat.PIXEL_FORMAT_DEPTH_F32, out mdata);
      var mwidth = mappedImage.info.width;
      var mheight = mappedImage.info.height;
      var mappedPixels = mdata.ToFloatArray(0, mwidth * mheight);
      

       

      And then feeding those pixels to ProjectColorToCamera() to obtain the world coordinates.

      The problem is that, even though I'm using the float depth format and sending it to a float array, I'm only getting integer depth values back (the values in mappedPixels are always ".0").

       

      So: is there a bug with CreateDepthImageMappedToColor which means it only ever does integer depth, or is there something else I should be doing to get the full depth data?

      I should note that I need this to be compatible with all cameras, so using the most recent SDK is not an option, nor is a fix (if this is a bug) going into a future SDK which doesn't support the R200.

       

      Thanks,

      James

        • 1. Re: Getting float depth data
          MartyG

          Getting floats instead of integers in a depth script is something that people have had trouble with in the past.  A developer called x.c posted a script a about 18 months ago that they said could get float data.

           

            Mat depthimage8;  

              depthimage8.create(480,640,CV_8UC1);

           

           

              PXCImage::ImageData data_depth;  unsigned short *depth_data;

           

           

                        sample->depth->AcquireAccess(PXCImage::ACCESS_READ,PXCImage::PIXEL_FORMAT_DEPTH, &data_depth);  

                              depth_data = (unsigned short*)data_depth.planes[0];  

                              unsigned char cVal;

                              for(int y=0; y<H; y++)

                              {

                                  for(int x=0; x<W; x++)

                                  {

                                      if(depth_data[y*W+x] >  fMaxValue)    //fMaxValue = 1000

                                          depth_data[y*W+x] =  fMaxValue;

                                      cVal = (unsigned char)((1.0*depth_data[y*W+x])/fMaxValue*255);

                                      if(cVal!=0)

                                          cVal = 255 - cVal;  

                                      depthimage8.at<unsigned char>(y,x) = cVal;

                                  }

                              }

          • 2. Re: Getting float depth data
            jb455

            Thanks for the reply Marty!

             

            My attempt to translate that to C# has resulted in this:

            var depth_data = new short[mappedPixels.Length];
            var ptr = mdata2.planes[0];
            var psize = Marshal.SizeOf(typeof(short));
            for(int i = 0; i < mappedPixels.Length; i++)
            {
                 depth_data[i] = (short)Marshal.PtrToStructure(ptr, typeof(short));
                 ptr = new IntPtr(ptr.ToInt64() + psize);
            }
            
            var dep = depth_data.Where(x => x > 0);
            for (int k = 0; k < mappedPixels.Length; k++)
            {
                 if (depth_data[k] > 1000)
                      depth_data[k] = 1000;
            
                 byte cval = (byte)(1.0 * depth_data[k] / (1000 * 255));
                 if (cval != 0)
                      cval = (byte)(255 - cval);
                 dpixels[k] = cval;
            }
            

            (where mappedPixels is an array colour.length * colour.width big)

             

            Problem is, line 18 is never hit, so everything in dpixels is 0. depth_data fills out nicely though; dep contains lots of nonzero elements which seem reasonable, so if anyone wants to get the data out of ImageData without using ToShortArray etc for some reason, lines 1-8 will do that for them.

            I don't really understand what they're trying to do in the second half of the snippet which doesn't help! (here is the original thread for info)

            Can anyone see where I'm going wrong, or suggest any other solutions?

            • 3. Re: Getting float depth data
              MartyG

              Yes I see, because cval = 0, it does not reach line 18 because it cannot satisfy the If statement's requirement that cval does not = 0.  'For' statements are my pet hate in programming!

              • 4. Re: Getting float depth data
                jb455

                So if I use PIXEL_FORMAT_DEPTH_RAW and convert each element in the array like so:

                 

                mappedImage.AcquireAccess(PXCMImage.Access.ACCESS_READ, PXCMImage.PixelFormat.PIXEL_FORMAT_DEPTH_RAW, out mdata);
                var mwidth = mappedImage.info.width;
                var mheight = mappedImage.info.height;
                
                var mp = mdata.ToShortArray(0, mwidth * mheight);
                var mappedPixels = new float[mp.Length];
                var du = device.QueryDepthUnit();
                for (int i = 0; i < mappedPixels.Length; i++)
                     mappedPixels[i] = mp[i] * du / 1000;
                

                 

                ...I still just get integers . And, actually, the values are way off too. Am I doing the conversion (line 9) right?

                • 5. Re: Getting float depth data
                  MartyG

                  From what I can gather, you use

                   

                  PXCImage::PIXEL_FORMAT_DEPTH_F32

                   

                  to get a float because that gets you a 32-bit float point (hence F32).

                   

                  Intel staffer David Lu recommended to someone else with floating point troubles to look at the C# source code of the RealSense SDK sample script 'Rawstreams' at

                  C:\Program Files (x86)\Intel\RSSDK\framework\CSharp\DF_RawStreams.cs

                  • 6. Re: Getting float depth data
                    jb455

                    You'd think, but as noted in my original post, that just returns integers.

                    I can't find anything to do with depth data in Raw Streams...

                    • 7. Re: Getting float depth data
                      MartyG

                      Another source reckoned that "If a device fails to determine the depth of a given image pixel, a value of zero will be stored in the depth image.  This is a reasonable sentinel for 'no depth' because all pixels with a depth of zero would correspond to the same physical location, the location of the imager itself." 

                       

                      "The default scale of an R200 device is one millimeter, allowing for a maximum expressive range of ~65 meters.  The depth scale can be modified by calling rs_set_device_option(...) with RS_OPTION_R200_DEPTH_UNITS, which specifies the number of micrometers per one increment of depth.  1000 would indicate millimeter scale, 10000 would indicate centimeter scale, while 31 would roughly approximate the F200's 1/32th of a millimeter scale."

                       

                      Sorry if I'm not being very helpful on this case! 

                      • 8. Re: Getting float depth data
                        jb455

                        Thanks for looking Marty, I appreciate your effort!

                         

                        If I do as in post #4 but with the standard depth image instead of the result of CreateDepthImageMappedToColor then I do get float depth values (in multiples of 1/8 of a millimetre), my problem is that it seems CreateDepthImageMappedToColor clips the data down to integers, losing the extra precision with no way to get it back.

                        I guess if it is bugged I'll have to use another method of mapping between the colour and depth images to get the camera coordinates. I have tried a few different methods in the past, but none of them worked as well as my current way of doing it.

                        • 9. Re: Getting float depth data
                          jb455

                          Any comments from Intel? How did the devs intend for us to get camera coordinates with float depths, aligned to the colour camera?

                          • 10. Re: Getting float depth data
                            jb455

                            Ok, managed to get this working using the Inverse UV Map:

                             

                            PXCMImage.ImageData ddata; 
                            depth.AcquireAccess(PXCMImage.Access.ACCESS_READ, PXCMImage.PixelFormat.PIXEL_FORMAT_DEPTH_F32, out ddata);
                            var dwidth = depth.info.width;
                            var dheight = depth.info.height; 
                            var dPixels = ddata.ToFloatArray(0, dwidth * dheight); 
                            depth.ReleaseAccess(ddata); 
                            
                            var invuvmap = new PXCMPointF32[color.info.width * color.info.height];
                            projection.QueryInvUVMap(depth, invuvmap);
                            var mappedPixels = new float[cheight * cwidth];
                            for (int i = 0; i < invuvmap.Length; i++) 
                            {     
                                 int u = (int)(invuvmap[i].x * dwidth);
                                 int v = (int)(invuvmap[i].y * dheight);
                                 if (u >= 0 && v >= 0 && u + v * dwidth < dPixels.Length)
                                 {
                                       mappedPixels[i] = dPixels[u + v * dwidth];
                                 } 
                            }
                            

                            With this, I now have float depth data aligned to the colour image, which I use in ProjectColorToCamera to get the camera coordinates for each point in the colour image.

                            2 of 2 people found this helpful
                            • 11. Re: Getting float depth data
                              MartyG

                              Thanks for sharing, JB!

                              • 12. Re: Getting float depth data
                                jb455

                                Grr, just tried this with the R200 (above was all testing with an SR300) and I'm getting ints again! @intel: is it possible to get float depth data from an R200 or is that int only?

                                • 13. Re: Getting float depth data
                                  Intel Corporation
                                  This message was posted on behalf of Intel Corporation

                                  Hi James,

                                  Let us investigate if this is possible. We’ll get back to you as soon as we have some updates on your case.
                                  Did you try using the exact same snippet you used for the SR300? We just want to make sure you didn’t change anything in your code.

                                  Regards,
                                  -Pablo

                                  • 14. Re: Getting float depth data
                                    jb455

                                    Yes, just plugged the R200 in using the same code. Thanks!

                                    1 2 Previous Next