Producing high quality depth maps

• US 8,355,565 B1
• Filed: 10/29/2009
• Issued: 01/15/2013
• Est. Priority Date: 10/29/2009
• Status: Expired due to Fees

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Yang, Qingxiong, Tan, Kar-Han
• Primary Examiner(s): JEAN, FRANTZ B
• Application Number:
  US12/608,394
• Time in Patent Office: 1,174 Days
• Field of Search: 382/154, 382/216
• US Class Current: 382/154
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/593 from stereo images

Abstract

Producing high quality depth maps. A time of flight camera and stereo cameras of a stereo camera arrangement are mapped into a common coordinate system. A correspondence between an RGB image taken from the time of flight camera and an RGB image taken from a reference stereo camera of the stereo camera arrangement is established. Intensity values of the plurality of pixels of a stereo camera left of the reference stereo camera, a stereo camera right of the reference stereo camera and the reference stereo camera are determined. These intensity values of the stereo camera left of the reference stereo camera and the stereo camera right of the reference stereo camera are compared to the reference camera. Based on the comparing, matching intensity values are determined.

Claims

1. A computer-implemented method for producing high quality depth maps in real time, said computer-implemented method comprising:

mapping a time of flight camera and stereo cameras of a stereo camera arrangement into a common coordinate system;

establishing a correspondence between a red, green and blue (RGB) image taken from said time of flight camera and an RGB image taken from a reference stereo camera of said stereo camera arrangement, wherein said RGB images taken from said time of flight camera and stereo cameras within said stereo camera arrangement comprise a plurality of pixels;

determining intensity values of said plurality of pixels of a stereo camera left of said reference stereo camera, a stereo camera right of said reference stereo camera and said reference stereo camera;

comparing said intensity values of said stereo camera left of said reference stereo camera and said stereo camera right of said reference stereo camera as to said reference camera; and

determining matching intensity values based on said comparing.

2. The computer-implemented method of claim 1, wherein said comparing said intensity values of said stereo camera left of said reference stereo camera and said stereo camera right of said reference stereo camera as to said reference stereo camera comprises:

finding a first plurality of differences between intensity values of pixels at said one or more depths of said reference stereo camera and intensity values of pixels at said one or more depths of said stereo camera that is positioned to the left of said reference stereo camera; and

finding a second plurality of differences between locations of pixels at said one or more depths of said reference stereo camera and locations of pixels at said one or more depths of said stereo camera thatis positioned to the right of said reference stereo camera.

3. The computer-implemented method of claim 1, further comprising:

generating a depth map based on said matching intensity values.

4. The computer-implemented method of claim 3, further comprising:

modifying said intensity values of said depth map based on an up-sampled active brightness image and a depth image captured by a time of flight sensor of said time of flight camera; and

generating a first redefined depth map based on said modifying.

5. The computer-implemented method of claim 4, further comprising:

generating a second redefined depth map by applying a stereo confidence map to said first redefined depth map.

6. The computer-implemented method of claim 1, further comprising:

before said mapping, generating estimated intrinsic and extrinsic matrices and radial distortion coefficients of said RGB images taken from said time of flight camera and said stereo cameras; and

removing distortion from said RGB images.

7. The computer-implemented method of claim 1, further comprising:

before said establishing a correspondence between said RGB image taken from said time of flight camera and said RGB image taken from said reference stereo camera, establishing a correspondence between sensors of said time of flight camera and said stereo camera to enable said sensors to capture images from a same point.

8. The computer-implemented method of claim 1, further comprising:

denoising said stereo cameras by applying at least one of the following methods to said intensity values of said stereo camera left of said reference stereo camera and said intensity values of said stereo camera right of said reference stereo camera;

a local optimization method; and

a global optimization method.

9. The computer-implemented method of claim 1, further comprising:

determining intensity values at every accessed depth plane for said stereo camera left of said reference stereo camera and said stereo camera right of said reference stereo camera; and

selecting at said every accessed depth plane a smallest intensity value of said stereo camera left of said reference stereo camera and said stereo camera right of said reference stereo camera.

10. The computer-implemented method of claim 9, further comprising:

computing a stereo confidence map utilizing said intensity values selected at said every accessed depth plane.

11. The computer-implemented method of claim 1, further comprising:

at every accessed depth plane, generating determined intensity values for said stereo camera left of said reference stereo camera and said stereo camera right of said reference stereo camera; and

selecting a smallest intensity value of a range of said intensity values for said stereo camera left of said reference stereo camera and a range of said intensity values for said stereo camera right of said reference stereo camera.

12. A computer system for producing high quality depth maps, said system comprising:

a common coordinate system mapper configured for mapping a time of flight camera and stereo cameras of a stereo camera arrangement into a common coordinate system;

a calibrator configured for establishing a correspondence between a red, green and blue (RGB) image taken from said time of flight camera and an RGB image taken from a reference stereo camera of said stereo camera arrangement, wherein said RGB images taken from said time of flight camera and stereo cameras within said stereo camera arrangement comprise a plurality of pixels;

an intensity value determiner configured for determining intensity values of said plurality of pixels of a stereo camera left of said reference stereo camera, a stereo camera right of said reference stereo camera and said reference stereo camera;

an intensity value comparor configured for comparing said intensity values of said stereo camera left of said reference stereo camera and said stereo camera right of said reference stereo camera as to said reference camera; and

a matching intensity value determiner configured for determining matching intensity values based on said comparing.

13. The computer system of claim 12, wherein said intensity value comparor comprises:

a difference determiner configured for finding a first plurality of differences between intensity values of pixels at said one or more depths of said reference stereo camera and intensity values of pixels at said one or more depths of said stereo camera that is positioned to the left of said reference stereo camera and finding a second plurality of differences between locations of pixels at said one or more depths of said reference stereo camera and locations of pixels at said one or more depths of said stereo camera that is positioned to the right of said reference stereo camera.

14. The computer system of claim 12, further comprising:

a depth map generator configured for generating a depth map based on said matching intensity values.

15. The computer system of claim 14, further comprising:

a time of flight sensor confidence mapper configured for modifying said intensity values of said depth map based on an up-sampled active brightness image and a depth image captured by a time of flight sensor of said time of flight camera and generating a first redefined depth map based on said modifying.

16. The computer system of claim 15, further comprising:

a stereo confidence mapper configured for generating a second redefined depth map by applying a stereo confidence map to said first redefined depth map.

17. A computer usable storage medium comprising instructions that when executed cause a computer system to perform a method for producing high quality depth maps, said method comprising:

mapping a time of flight camera and stereo cameras of a stereo camera arrangement into a common coordinate system;

establishing a correspondence between a red, green and blue (RGB) image taken from said time of flight camera and an RGB image taken from a reference stereo camera of said stereo camera arrangement, wherein said RGB images taken from said time of flight camera and said stereo cameras comprise a plurality of pixels;

matching intensity values of said plurality of pixels of a stereo camera left of said reference stereo camera and a stereo camera right of said reference stereo camera to said reference stereo camera to generate a depth map; and

computing a time of flight confidence depth map utilizing said matching intensity values and an up-sampled active brightness image and a depth image captured by a time of flight sensor of said time of flight camera, said time of flight confidence depth map modifying said depth map and generating a first redefined depth map.

18. The method of claim 17, further comprising:

modifying said intensity values of said first redefined depth map utilizing a stereo confidence map; and

generating a second redefined depth map based on said modifying.

19. The method of claim 17, further comprising:

determining intensity values at every accessed depth plane for said stereo camera left of said reference stereo camera and said stereo camera right of said reference stereo camera; and

selecting at said every accessed depth plane a smallest intensity value of said stereo camera left of said reference stereo camera and said stereo camera right of said reference stereo camera.

20. The method of claim 19, further comprising:

computing a stereo confidence map utilizing said intensity values selected at said every accessed depth plane.