Items with no label
3335 Discussions

Depth map of black pixels in an image is not coming

KKedi
Beginner
2,722 Views

Camera Model: SR300

Firmware Version: firmware': '3.21.0.0'

Operating System & Version: macOS sierra 10.12.6

librealsense version: 1.12.1

pyrealsense version: 2.1

Required info:

I am trying to get the depth map using intel realsense SR300. However i am not getting the depth map for the black pixels in an image. example: for my hair i am getting 0 value in the depth map.

I am wearing a stripes t-shirt having grey and red colour. Depth map for red stripe is generated however for the grey colour the depth map is not created. How to solve this problem? Even the depth of my hair, beard or anything black is not coming.

CODE:

import os 

import sys

 

import cv2

 

import pyrealsense as pyrs

 

import numpy as np

 

 

def create_directory(directory):

 

if not os.path.exists(directory):

 

os.makedirs(directory)

 

return 1

 

else:

 

print(directory, '- already exists')

 

return 0

 

 

 

def check_directories():

 

try:

 

create_directory('./data')

 

create_directory('./data/depth')

 

create_directory('./data/color')

 

create_directory('./data/cad')

 

create_directory('./data/dac')

 

except:

 

print("Unexpected error:", sys.exc_info()[0])

 

return -1

 

return 0

 

 

def main():

 

file_structure = check_directories()

 

if file_structure == -1:

 

print('\nERROR: Directories can\'t be created, error thrown')

 

return -1

 

else:

 

print('\nDirectories created successfully...\nLaunching camera module...')

 

 

# Fire camera & launch streams

 

serv = pyrs.Service()

 

serv.start()

 

cam = serv.Device(device_id = 0, streams = [pyrs.stream.ColorStream(fps=10),

 

pyrs.stream.DepthStream(fps=10),

 

pyrs.stream.CADStream(fps=10),

 

pyrs.stream.DACStream(fps=10),

 

pyrs.stream.InfraredStream(fps=10),

 

pyrs.stream.PointStream(fps=10)])

 

scale = cam.depth_scale * 1000

 

print "scale = ", scale

 

# Some important variables

 

flag_save_frames = True

 

file_num = 0

 

 

# Start fetching Buffer

 

print('Starting Buffer...')

 

while(True):

 

cam.wait_for_frames()

 

current_color = cam.color[:,:,::-1]

 

# print current_color.shape

 

current_depth = cam.depth * scale

 

current_cad = cam.cad[:,:,::-1]

 

current_dac = cam.dac * scale

 

print "max depth", np.max(current_dac)

 

print "print depth val = ",np.max(cam.dac)

 

 

cv2.imshow('Color',current_color)

 

cv2.imshow('Depth',current_depth/1000)

 

cv2.imshow('CAD',current_cad)

 

cv2.imshow('DAC',current_dac/1000)

 

 

 

 

if flag_save_frames:

 

num = format(file_num, '08')

 

cv2.imwrite('./data/depth/' + str(num) + '.png', cam.depth)

 

cv2.imwrite('./data/color/' + str(num) + '.png', current_color)

 

cv2.imwrite('./data/dac/' + str(num) + '.png', cam.dac)

 

cv2.imwrite('./data/cad/' + str(num) + '.png', current_cad)

 

file_num += 1

 

 

k = cv2.waitKey(1)

 

if k == ord('q'):

 

print('Q Pressed...\nEnding execution')

 

break

 

if k == ord('f'):

 

if flag_save_frames:

 

print('F Pressed...\nStopped fetching frames...')

 

flag_save_frames = False

 

else:

 

print('F Pressed...\nStarted fetching frames...')

 

flag_save_frames = True

XXXWHITESPA...

0 Kudos
8 Replies
MartyG
Honored Contributor III
1,312 Views

Causes of an image being black like this can include:

1. The light source in the room is too dark or too bright. The camera has a component called an IR emitter that assists the IR sensor with exposure, and if the lighting is too low or too high, the emitter can cause the IR sensor to become saturated. The IR emitter can be turned off with scripting in Librealsense.

* Enables / disables R200 emitter

*/

public void setEmitterEnabled(int value) {

setOption(RealSense.RS_OPTION_R200_EMITTER_ENABLED, value);

}

2. The 'gain' value may be too low. Increasing gain can make an image brighter.

Also, if you have a problem with depth scans that are too noisy, using a 'point cloud' type of depth scan may yield better results.

0 Kudos
KKedi
Beginner
1,312 Views

MartyG the light source is good enough. It is always the case that the depth for the objects black in colour is undefined. Are black objects a limitation for SR300?

The depth images are very noisy and so are there any predefined methods that helps us reduce noise?

0 Kudos
MartyG
Honored Contributor III
1,312 Views

Further research revealed that it is apparently a fundamental principle of 3D scanning in general (not just RealSense) that black / dark objects do not scan well because the color absorbs the laser. A tip that was shared many times for scanning black objects was to cover them in a spray or a powder fine enough to spray on, such as foot powder or baby powder so that the laser can see the surface.

I recently found a link to a professional 3D scanning spray.

https://3dscanspray.com/ 3DScanSpray.com

In your case, I think I would think that coating your t-shirt in baby powder would be a nicer option than foot powder!

As I mentioned above, doing a 'point cloud' depth scan may provide better results if you have noise on your scan.

https://software.intel.com/en-us/articles/intel-realsense-technology-and-the-point-cloud Intel® RealSense™ Technology and the Point Cloud | Intel® Software

0 Kudos
KKedi
Beginner
1,312 Views

Hi MartyG Thanks for your response.

R200 camera uses a similar technology. However, such a problem is not present there. I had my friend using it and he was able to get the depth maps for the black objects as well.

0 Kudos
MartyG
Honored Contributor III
1,312 Views

I did further research into scanning black objects, and every information source repeated the earlier suggestions of changing the lighting conditions or using a spray / powder. The only new tip I did see suggested was to turn the camera 90 degrees to change how the laser light falls onto the object being scanned.

0 Kudos
Michael_M_Intel1
Employee
1,312 Views

As I noted before, shiny and/or transparent objects have a similar problem (in this case, the light is not absorbed, but reflected or refracted in a direction not visible to the camera). Hair is yet ANOTHER problem: it's just too fine to reflect the dot pattern in a coherent way, and often individual hairs are below the camera resolution. Of course if the hair is black you have more than one problem. Edges of objects are also often not picked up well as they are at an extreme angle with respect to the camera (and may be shadowed from one camera or the other), and also pick up relatively small amounts of light from the projector. As an added complication, objects that are "glossy" tend to act more and more "shiny" (more specular, or mirror-like) at glancing angles.

These are not RealSense problems per se but general problems with depth sensing.

Unfortunately using a spray or otherwise modifying the object to be scanned (eg using a hair cap) is not always feasible. Your best strategy is probably to make your algorithms robust to holes in the data, and using hole-filling algorithms were appropriate. Don't forget you also have color pixels even where there is no depth so you can peek at them to see if a hole is a "real" hole or maybe due to the pixel being dark (which you can easily determine from the color channel).

PPola
Beginner
1,312 Views

Do you know if new generation of realsense D415 & D335 deal better with this problem?

0 Kudos
MartyG
Honored Contributor III
1,312 Views

Whilst the new cameras would still be subject to the same general physics problem of dark / black colors absorbing laser light, they do have a new Advanced Mode that enables users to have greater control over depth generation and color correction.

https://github.com/IntelRealSense/librealsense/blob/development/doc/rs400/rs400_advanced_mode.md librealsense/rs400_advanced_mode.md at development · IntelRealSense/librealsense · GitHub

Reply