1 2 Previous Next 16 Replies Latest reply on Feb 2, 2018 12:56 PM by BjarneG

    Value of unknown pixels in depth-map

    BjarneG

      Hello forum.

       

      Processing values in depth-streams from a D435-camers, I observe both 0 and 65535 (== max 16 bit int). I was under the impression that 0 indicates an "unknown" pixel (i.e. its depth is sufficiently uncertain, based on selected heuristics). But what does a 65535-value indicate? (I see this both in C++ and in the Python-wrapper.)

        • 1. Re: Value of unknown pixels in depth-map
          MartyG

          The 0 to 65535 range is apparently related to the brightness of the pixel, where '0' is black and 65535 is white.  In terms of general image processing principles, if 65535 appears in your results then it can sometimes mean that this part of the image is light-saturated.  It is okay to have some of these non-useful pixel values in your image though, and may be unavoidable.

          • 2. Re: Value of unknown pixels in depth-map
            BjarneG

            Hi Marty.

             

            Thanks for your quick reply, however is this really true for pixels in the depth-map? Do you have links to documentation where this is described?

             

            Regards,

            - Bjarne

            • 3. Re: Value of unknown pixels in depth-map
              MartyG

              The 0 to 65535 range can be used in a variety of settings related to color brightness, such as hue and intensity.  Googling for 'pixel value 65535' can get some good search results on the subject.

               

              Here is my particular source I quoted though.  If you do a Find operation on the page for 65535, it will highlight all the instances where it is mentioned in relation to pixel brightness.

               

              http://wiki.freepascal.org/BGRABitmap_Pixel_types

               

              Edit: apparently the '0 to 65535' description in terms of brightness bests fits greyscale images.  It gets a bit more complex when dealing with color images.

              • 4. Re: Value of unknown pixels in depth-map
                BjarneG

                Thanks Marty, but I believe this document refers to BGRA-bitmaps. I'm talking about the depth-matrix which should contain distance (in mm) from the camera.

                • 5. Re: Value of unknown pixels in depth-map
                  MartyG

                  I was just using that particular document to illustrate how the 0-65535 range is applicable to a variety of different lightness / color settings.

                   

                  In general, a white or black pixel may represent areas that the camera was not able to perceive detail on and create useful data representing that area.  This may be because the lighting cast on the object may be too dark, too strong or too reflective / light-absorbing.  Non-useful values may also be returned if the area being scanned has a shape or edges that make it harder for the camera to read.

                   

                  If these values are occurring in small numbers then my advice would be to just ignore them, as they are not likely to greatly impact the quality of the depth image.

                  • 6. Re: Value of unknown pixels in depth-map
                    BjarneG

                    Marty,

                     

                    do you mean that 0 and 65535 are both used to indicate pixels of unknown depth? Is there a document or source/header mentioning this, or perhaps someone from Intel would confirm?

                     

                    Regards,

                    - Bjarne

                    • 7. Re: Value of unknown pixels in depth-map
                      MartyG

                      My experience is that RealSense usually describes unknown detail in terms of '0'.

                       

                      An Intel support agent can probably confirm the meaning of 65535 in the context of RealSense depth scans better than I can.

                      • 8. Re: Value of unknown pixels in depth-map
                        BjarneG

                        Your experience is the same as mine, Marty.   That is, '0' is normally denoting a pixel of unknown depth.

                         

                        So this boils down to my original question: What do the 65355-values in the depth-map mean? Anyone?

                         

                        Regards,

                        - Bjarne

                        • 9. Re: Value of unknown pixels in depth-map
                          Intel Corporation
                          This message was posted on behalf of Intel Corporation

                          Hello BjarneG,

                          Depth values of 65535 correspond to the maximum depth that the camera can read.  Does this make sense based on your observations? If not, please clarify under which circumstances you are seeing 0 vs 65535.

                          Regards,
                          Jesus
                          Intel Customer Support

                          • 10. Re: Value of unknown pixels in depth-map
                            BjarneG

                            Hi Jesus.

                             

                            Although I agree that 65355 is the max possible depth a D400-camera can read (65355 == 2^16-1 === max value for a 16-bit unsigned int), it doesn't make sense to get these values in my setting. I see 65355 scattered around in the depth-matrix, although less frequent and not in similar patterns as 0-values. I'm indoor with a mix of natural light from windows and T8-tubes. Depth should be between 300 and 8000 cm.

                             

                            My workaround is an extra pass over the matrix to filter out 65535, otherwise basic statistics is polluted and simple operations like contrast-stretch doesn't make much sense. For me, this works at the moment but such wasted cycles is unacceptable in our finished solution. (Besides, it is a little confusing before you become aware that both 0 and 65535 may occur.   )

                             

                            I use this little snippet

                             

                            ---------------

                            #include <librealsense2/rs.hpp>
                            #include <opencv2/opencv.hpp>
                            
                            using namespace std;
                            using namespace cv;
                            
                            int main()
                            {
                                rs2::pipeline pipe;
                                rs2::config cfg;
                            
                                cfg.enable_stream(RS2_STREAM_COLOR, 640, 480, RS2_FORMAT_BGR8, 30);
                                cfg.enable_stream(RS2_STREAM_DEPTH, 640, 480, RS2_FORMAT_Z16, 30);
                            
                                pipe.start(cfg);
                            
                                rs2::frameset frames;
                                while (true)
                                {
                                    // Wait for a coherent pair of frames: depth and color
                                    frames = pipe.wait_for_frames();
                                    rs2::depth_frame depth_frame = frames.get_depth_frame();
                                    rs2::video_frame color_frame = frames.get_color_frame();
                            
                                    unsigned int d_w = depth_frame.get_width();
                                    unsigned int d_h = depth_frame.get_height();
                                    unsigned int c_w = color_frame.get_width();
                                    unsigned int c_h = color_frame.get_height();
                            
                                    void *depth_data = (void*)depth_frame.get_data();
                                    void *color_data = (void*)color_frame.get_data();
                            
                                    // Creating OpenCV Matrices
                                    Mat depth_raw(Size(d_w, d_h), CV_16U, depth_data, Mat::AUTO_STEP);
                                    Mat color(Size(c_w, c_h), CV_8UC3, color_data, Mat::AUTO_STEP);
                            
                                    Mat filtered_0(Size(d_w, d_h), CV_8U, cv::Scalar(0));
                                    Mat filtered_65535(Size(d_w, d_h), CV_8U, cv::Scalar(0));
                                    filtered_0.setTo(255, depth_raw == 0);
                                    filtered_65535.setTo(255, depth_raw == 65535);
                            
                                    // Display
                                    imshow("Image", color);
                                    imshow("Filtered 0", filtered_0);
                                    imshow("Filtered 65535", filtered_65535);
                            
                                    int k = waitKey(10);
                                    if (k == 27)
                                        break;
                                }
                            
                            
                                return 0;
                            }
                            

                            ---------------

                             

                            What I would really like is a post-processing option for the camera to "fill in" unknown values based on some (possible user-selected) heuristics. I believe this is most effectively done at the disparity-map level, hence it belongs in the ASIC / camera, but methods in the SDK might also make sense.

                             

                            Regards,

                            - Bjarne

                            • 11. Re: Value of unknown pixels in depth-map
                              Intel Corporation
                              This message was posted on behalf of Intel Corporation

                              Hello BJarneG,

                              I received feedback from the engineering team.  A depth value of 0 means the value is unknown and we know it. A depth value of 65535 means we don't know the value is unknown, but rather think it is valid, but it is not valid. This is just what the ASIC is reporting. At the moment we don't offer invalid pixel filtering but I will raise this as a customer feature request to the development team.

                              Regards,
                              Jesus G.
                              Intel Customer Support

                              1 of 1 people found this helpful
                              • 12. Re: Value of unknown pixels in depth-map
                                BjarneG

                                Thanks for following up on this Jesus.

                                 

                                Not 100% sure I grasp the full meaning of "we don't know the value is unknown, but rather think it is valid, but it is not valid" But it is nice to know that both can be dealt with in the same way.

                                 

                                Looking forward to the filter to "fill in" these values.

                                 

                                Regards,

                                - Bjarne

                                 

                                 

                                1 of 1 people found this helpful
                                • 13. Re: Value of unknown pixels in depth-map
                                  Intel Corporation
                                  This message was posted on behalf of Intel Corporation

                                  Hello BjarneG,

                                  Sorry for the confusing expression. The ASIC erroneously assigns a value of 65535 to those pixels it thinks are valid. However, the pixels are not valid. That's it.

                                  Regards,
                                  Jesus

                                  • 14. Re: Value of unknown pixels in depth-map
                                    BjarneG

                                    Thanks for clarifying Jesus,

                                     

                                    I observe that "hole filling" seems to be implemented in the spatial post-processing filter. The measure example shows how to this in code, and there is a dropdown in realsense-viewer.

                                     

                                    Regards,

                                    - Bjarne

                                    1 2 Previous Next