6 Replies Latest reply on Sep 15, 2017 3:55 AM by MartyG

    Depth map of black pixels in an image is not coming

    kishankk

      Camera Model: SR300

      Firmware Version: firmware': '3.21.0.0'

      Operating System & Version: macOS sierra 10.12.6

      librealsense version: 1.12.1

      pyrealsense version: 2.1

       

      Required info:

      I am trying to get the depth map using intel realsense SR300. However i am not getting the depth map for the black pixels in an image. example: for my hair i am getting 0 value in the depth map.

      Screen Shot 2017-09-13 at 7.09.59 PM.png

      I am wearing a stripes t-shirt having grey and red colour. Depth map for red stripe is generated however for the grey colour the depth map is not created. How to solve this problem? Even the depth of my hair, beard or anything black is not coming.

      CODE:

       

      import os
      import sys
      import cv2
      import pyrealsense as pyrs
      import numpy as np

      def create_directory(directory):
        if not os.path.exists(directory):
        os.makedirs(directory)
        return 1
        else:
        print(directory, '- already exists')
        return 0


      def check_directories():
        try:
        create_directory('./data')
        create_directory('./data/depth')
        create_directory('./data/color')
        create_directory('./data/cad')
        create_directory('./data/dac')
        except:
        print("Unexpected error:", sys.exc_info()[0])
        return -1
        return 0

      def main():
        file_structure = check_directories()
        if file_structure == -1:
             print('\nERROR: Directories can\'t be created, error thrown')
             return -1
        else:
             print('\nDirectories created successfully...\nLaunching camera module...')

        # Fire camera & launch streams
        serv = pyrs.Service()
        serv.start()
        cam = serv.Device(device_id = 0, streams = [pyrs.stream.ColorStream(fps=10),
                                 pyrs.stream.DepthStream(fps=10),
                                 pyrs.stream.CADStream(fps=10),
                                 pyrs.stream.DACStream(fps=10),
                                 pyrs.stream.InfraredStream(fps=10),
                                 pyrs.stream.PointStream(fps=10)])
        scale = cam.depth_scale * 1000
        print "scale = ", scale
        # Some important variables
        flag_save_frames = True
        file_num = 0

        # Start fetching Buffer
        print('Starting Buffer...')
        while(True):
             cam.wait_for_frames()
             current_color = cam.color[:,:,::-1]
             # print current_color.shape
             current_depth = cam.depth * scale
             current_cad = cam.cad[:,:,::-1]
             current_dac = cam.dac * scale
             print "max depth", np.max(current_dac)
             print "print depth val = ",np.max(cam.dac)
        
             cv2.imshow('Color',current_color)
             cv2.imshow('Depth',current_depth/1000)
             cv2.imshow('CAD',current_cad)
             cv2.imshow('DAC',current_dac/1000)

        

             if flag_save_frames:
                  num = format(file_num, '08')
                  cv2.imwrite('./data/depth/' + str(num) + '.png', cam.depth)
                  cv2.imwrite('./data/color/' + str(num) + '.png', current_color)
                  cv2.imwrite('./data/dac/' + str(num) + '.png', cam.dac)
                  cv2.imwrite('./data/cad/' + str(num) + '.png', current_cad)
                  file_num += 1

             k = cv2.waitKey(1)
             if k == ord('q'):
                  print('Q Pressed...\nEnding execution')
                  break
             if k == ord('f'):
                  if flag_save_frames:
                       print('F Pressed...\nStopped fetching frames...')
                       flag_save_frames = False
                  else:
                       print('F Pressed...\nStarted fetching frames...')
                       flag_save_frames = True

        cam.stop()
        serv.stop()

        return 0


      if __name__ == '__main__':
        print(__doc__)
        main()

       

       

       

      Are there any methods in pyrealsesnse that help me create proper depth maps with less noise? Currently i am getting a lot of noise in my depth maps as well.

        • 1. Re: Depth map of black pixels in an image is not coming
          MartyG

          Causes of an image being black like this can include:

           

          1.  The light source in the room is too dark or too bright.  The camera has a component called an IR emitter that assists the IR sensor with exposure, and if the lighting is too low or too high, the emitter can cause the IR sensor to become saturated.  The IR emitter can be turned off with scripting in Librealsense.

           

          * Enables / disables R200 emitter

               */

              public void setEmitterEnabled(int value) {

                  setOption(RealSense.RS_OPTION_R200_EMITTER_ENABLED, value);

              }

           

          2.  The 'gain' value may be too low.  Increasing gain can make an image brighter.

           

          Also, if you have a problem with depth scans that are too noisy, using a 'point cloud' type of depth scan may yield better results.

          • 2. Re: Depth map of black pixels in an image is not coming
            kishankk

            MartyG the light source is good enough. It is always the case that the depth for the objects black in colour is undefined. Are black objects a limitation for SR300?

             

            The depth images are very noisy and so are there any predefined methods that helps us reduce noise?

            • 3. Re: Depth map of black pixels in an image is not coming
              MartyG

              Further research revealed that it is apparently a fundamental principle of 3D scanning in general (not just RealSense) that black / dark objects do not scan well because the color absorbs the laser.  A tip that was shared many times for scanning black objects was to cover them in a spray or a powder fine enough to spray on, such as foot powder or baby powder so that the laser can see the surface.

               

              I recently found a link to a professional 3D scanning spray.

               

              3DScanSpray.com

               

              In your case, I think I would think that coating your t-shirt in baby powder would be a nicer option than foot powder!

               

              As I mentioned above, doing a 'point cloud' depth scan may provide better results if you have noise on your scan.

               

              Intel® RealSense™ Technology and the Point Cloud | Intel® Software

              1 of 1 people found this helpful
              • 4. Re: Depth map of black pixels in an image is not coming
                McCool

                As I noted before, shiny and/or transparent objects have a similar problem (in this case, the light is not absorbed, but reflected or refracted in a direction not visible to the camera).   Hair is yet ANOTHER problem: it's just too fine to reflect the dot pattern in a coherent way, and often individual hairs are below the camera resolution.  Of course if the hair is black you have more than one problem.  Edges of objects are also often not picked up well as they are at an extreme angle with respect to the camera (and may be shadowed from one camera or the other), and also pick up relatively small amounts of light from the projector.  As an added complication, objects that are "glossy" tend to act more and more "shiny" (more specular, or mirror-like) at glancing angles.

                 

                These are not RealSense problems per se but general problems with depth sensing.

                 

                Unfortunately using a spray or otherwise modifying the object to be scanned (eg using a hair cap) is not always feasible.   Your best strategy is probably to make your algorithms robust to holes in the data, and using hole-filling algorithms were appropriate.  Don't forget you also have color pixels even where there is no depth so you can peek at them to see if a hole is a "real" hole or maybe due to the pixel being dark (which you can easily determine from the color channel).

                2 of 2 people found this helpful
                • 5. Re: Depth map of black pixels in an image is not coming
                  kishankk

                  Hi MartyG Thanks for your response.

                   

                  R200 camera uses a similar technology. However, such a problem is not present there. I had my friend using it and he was able to get the depth maps for the black objects as well.

                  • 6. Re: Depth map of black pixels in an image is not coming
                    MartyG

                    I did further research into scanning black objects, and every information source repeated the earlier suggestions of changing the lighting conditions or using a spray / powder.  The only new tip I did see suggested was to turn the camera 90 degrees to change how the laser light falls onto the object being scanned.

                    1 of 1 people found this helpful