Generating a depth map

• US 9,098,908 B2
• Filed: 10/21/2011
• Issued: 08/04/2015
• Est. Priority Date: 10/21/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Kirk, Adam G., Eshet, Yaron, Patiejunas, Kestutis, Kang, Sing Bing, Zitnick, Charles Lawrence III, Eraker, David, Winder, Simon
• Primary Examiner(s): Czekaj, Dave
• Assistant Examiner(s): Brumfield, Shanika
• Application Number:
  US13/278,184
• Publication Number:
  US 20130100256A1
• Time in Patent Office: 1,383 Days
• Field of Search: 348 42- 60, 382/100, 382/104, 382/154
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/593 from stereo images

Abstract

Methods and systems for generating a depth map are provided. The method includes projecting an infrared (IR) dot pattern onto a scene. The method also includes capturing stereo images from each of two or more synchronized IR cameras, detecting a number of dots within the stereo images, computing a number of feature descriptors for the dots in the stereo images, and computing a disparity map between the stereo images. The method further includes generating a depth map for the scene using the disparity map.

Claims

1. A method performed at least in part by a hardware processor, the method comprising:

projecting a random infrared dot pattern onto a scene;

genlocking a first infrared camera and a second infrared camera;

capturing a first infrared image of the random infrared dot pattern using the first infrared camera and a second infrared image of the random infrared dot pattern using the second infrared camera, wherein the genlocking causes the first infrared image to be temporally coherent with the second infrared image;

detecting a first plurality of first dots within the first infrared image captured using the first infrared camera;

detecting a second plurality of second dots within the second infrared image captured using the second infrared camera;

computing a first feature descriptor for an individual first dot in the first infrared image based on a corresponding first location of a neighboring first dot in the first infrared image that neighbors the individual first dot in the first infrared image;

computing a second feature descriptor for an individual second dot in the second infrared image based on a corresponding second location of a neighboring second dot in the second infrared image that neighbors the individual second dot in the second infrared image;

matching the individual first dot to the individual second dot based on similarity of the first feature descriptor to the second feature descriptor;

computing a disparity map reflecting pixel shifts across matched dots of the first infrared image and the second infrared image, the matched dots including the individual first dot and the individual second dot; and

generating a depth map for the scene using the disparity map.

2. The method of claim 1, wherein the first infrared camera and the second infrared camera are part of an active infrared stereo module, the method further comprising:

generating another depth map using another active infrared stereo module comprising other genlocked infrared cameras.

3. The method of claim 2, further comprising:

genlocking the active infrared stereo module and the another active infrared stereo module using a synchronization signal such that the first infrared camera and the second infrared camera are also genlocked with each of the other genlocked infrared cameras.

4. The method of claim 2, further comprising:

combining the depth map and the another depth map to create a constructive view of the scene.

5. The method of claim 2, the method further comprising:

projecting another random infrared dot pattern onto the scene with another infrared projector of the another active infrared stereo module.

6. The method of claim 5, wherein the random infrared dot pattern and the another random infrared dot pattern overlap.

7. The method of claim 1, further comprising:

capturing a color image of the scene with a color camera and supplementing the depth map with the color image.

8. The method of claim 1, further comprising:

assigning the individual first dot, the neighboring first dot, the individual second dot, and the neighboring second dot to specific coordinate locations of a grid; and

using the specific coordinate locations to determine the first feature descriptor and the second feature descriptor.

9. The method of claim 1, further comprising:

computing the first feature descriptor as a first radial descriptor based on a distance and reference angle between the individual first dot and the neighboring first dot; and

computing the second feature descriptor as a second radial descriptor based on a distance and reference angle between the individual second dot and the neighboring second dot.

10. A system comprising:

a hardware processor; and

a memory device storing instructions that configure the hardware processor to;

cause an infrared projector to project a random infrared feature pattern onto a scene, the random infrared feature pattern comprising a plurality of randomly-projected recognizable features;

obtain temporally-coherent infrared stereo images of the random infrared feature pattern, the temporally-coherent infrared stereo images being captured by a group of genlocked infrared cameras;

identify individual randomly-projected recognizable features within the temporally-coherent infrared stereo images captured by the group of genlocked infrared cameras;

compute a plurality of feature descriptors corresponding to the individual randomly-projected recognizable features in the temporally-coherent infrared stereo images captured by the group of genlocked infrared cameras;

use the plurality of feature descriptors of the individual randomly-projected recognizable features to compute a disparity map for the temporally-coherent infrared stereo images captured by the group of genlocked infrared cameras; and

generate a depth map for the scene using the disparity map.

11. The system of claim 10, further comprising:

a projector interface configured to connect the system to the infrared projector; and

a camera interface configured to connect the system to the group of genlocked infrared cameras.

12. The system of claim 11, further comprising the infrared projector connected to the projector interface and the group of genlocked infrared cameras connected to the camera interface.

13. The system of claim 10, wherein the instructions further configure the hardware processor to:

cause another infrared projector to project another infrared random infrared feature pattern onto the scene;

obtain another depth map using another group of genlocked infrared cameras; and

obtain a constructive view of the scene using the depth map and the another depth map.

14. The system of claim 10, wherein the individual randomly-projected recognizable features comprise dots, triangles, or grids.

15. The system of claim 10, wherein the instructions configure the hardware processor to:

compute the disparity map between the temporally-coherent infrared stereo images using differences between the plurality of feature descriptors.

16. The system of claim 10, wherein the instructions configure the hardware processor to:

in a first instance, project the individual randomly-projected recognizable features as dots;

in a second instance, project the individual randomly-projected recognizable features as triangles; and

in a third instance, project the individual randomly-projected recognizable features as grids.

17. A method performed by a hardware processor of a computing system, the method comprising:

obtaining temporally-coherent infrared stereo images of a random infrared feature pattern projected onto a scene, wherein the temporally-coherent infrared stereo images were captured by a group of synchronized infrared cameras;

identifying individual randomly-projected recognizable features within the temporally-coherent infrared stereo images that were captured by the group of synchronized infrared cameras;

determining a plurality of feature descriptors corresponding to the individual randomly-projected recognizable features in the temporally-coherent infrared stereo images that were captured by the group of synchronized infrared cameras;

using the plurality of feature descriptors of the individual randomly-projected recognizable features to compute a disparity map for the temporally-coherent infrared stereo images that were captured by the group of synchronized infrared cameras; and

generating a depth map for the scene using the disparity map.

18. The method of claim 17, further comprising:

determining at least some of the feature descriptors for certain randomly-projected recognizable features based on coordinates of neighboring randomly-projected recognizable features.

19. The method of claim 17, further comprising:

causing the random infrared feature pattern to be projected onto the scene.

20. The method of claim 17, further comprising:

generating a synchronization signal to synchronize the group of infrared cameras.