Depth Mapping Based on Pattern Matching and Stereoscopic Information

• US 20110025827A1
• Filed: 07/28/2010
• Published: 02/03/2011
• Est. Priority Date: 07/30/2009
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: PrimeSense Ltd (Apple Inc.)
• Inventors: Medioni, Gerard, Deitch, Ronen, Cohen, Daniel, Shpunt, Alexander, Sali, Erez
• Granted Patent: US 9,582,889 B2
• Time in Patent Office: Days
• US Class Current: 348/47
• CPC Class Codes: G01B 11/22 for measuring depth
• CPC Class Codes: G01B 11/25 by projecting a pattern, e.g. one or more lines, moiré fringes on the object G01B11/255 takes precedence ; image analysis for depth or shape recovery G06T7/50
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

A method for depth mapping includes projecting a pattern of optical radiation onto an object. A first image of the pattern on the object is captured using a first image sensor, and this image is processed to generate pattern-based depth data with respect to the object. A second image of the object is captured using a second image sensor, and the second image is processed together with another image to generate stereoscopic depth data with respect to the object. The pattern-based depth data is combined with the stereoscopic depth data to create a depth map of the object.

Claims

1. A method for depth mapping, comprising:

projecting a pattern of optical radiation onto an object;

capturing a first image of the pattern on the object using a first image sensor, and processing the first image to generate pattern-based depth data with respect to the object;

capturing a second image of the object using a second image sensor, and processing the second image together with another image to generate stereoscopic depth data with respect to the object; and

combining the pattern-based depth data with the stereoscopic depth data to create a depth map of the object.

2. The method according to claim 1, wherein processing the second image together with the other image comprises processing the second image together with the first image.

3. The method according to claim 2, wherein projecting the pattern comprises projecting infrared (IR) radiation onto the object, and wherein capturing the first image comprises capturing the IR radiation that is reflected from the object.

4. The method according to claim 3, wherein capturing the second image comprises capturing a color image of the object.

5. The method according to claim 4, wherein the color image comprises pixels, and the depth map comprises depth values, and wherein the method comprises outputting the color image to a display together with the depth coordinates that are associated with the pixels.

6. The method according to claim 1, wherein projecting the pattern comprises projecting multiple spots onto the object, and wherein processing the first image comprises finding respective transverse shifts between the spots on the object and the spots in a reference image of the pattern, and computing the depth data based on the transverse shifts.

7. The method according to claim 1, wherein combining the pattern-based depth data with the stereoscopic depth data comprises computing respective measures of confidence associated with the pattern-based depth data and stereoscopic depth data, and selecting depth coordinates from among the pattern-based and stereoscopic depth data responsively to the respective measures of confidence.

8. The method according to claim 1, wherein combining the pattern-based depth data with the stereoscopic depth data comprises defining multiple candidate depth coordinates for each of a plurality of pixels in the depth map, and selecting one of the candidate depth coordinates at each pixel for inclusion in the depth map.

9. The method according to claim 8, wherein the multiple candidate depth coordinates comprise, for at least some of the pixels, a null coordinate indicating that no valid depth coordinate was found.

10. The method according to claim 8, wherein selecting the one of the candidate depth coordinates comprises applying weighted tensor voting among the pixels in order to select the one of the candidate depth coordinates based on the candidate depth coordinates at neighboring pixels.

11. The method according to claim 1, wherein combining the pattern-based depth data with the stereoscopic depth data comprises applying a calibration procedure to the first and second images so as to correct for a misalignment between the first and second images.

12. The method according to claim 11, wherein applying the calibration procedure comprises correcting for a change in alignment between the pattern of optical radiation and the first image sensor.

13. A method for depth mapping, comprising:

receiving at least one image of an object, captured by an image sensor, the image comprising multiple pixels;

processing the at least one image to generate depth data comprising multiple candidate depth coordinates for each of a plurality of the pixels;

applying a weighted voting process to the depth data in order to select one of the candidate depth coordinates at each pixel; and

outputting a depth map of the object comprising the selected one of the candidate depth coordinates at each pixel.

14. The method according to claim 13, wherein processing the at least one image comprises computing respective measures of confidence associated with the candidate depth coordinates, and wherein applying the weighted voting process comprises weighting votes for the candidate depth coordinates responsively to the respective measures of confidence.

15. The method according to claim 13, wherein the multiple candidate depth coordinates comprise, for at least some of the pixels, a null coordinate indicating that no valid depth coordinate was found.

16. The method according to claim 13, wherein applying the weighted voting process comprises applying weighted tensor voting among the pixels in order to select the one of the candidate depth coordinates based on the candidate depth coordinates at neighboring pixels.

17. The method according to claim 16, wherein applying the weighted tensor voting comprises computing a weighted sum of covariance matrices over the neighboring pixels, and selecting the one of the candidate depth coordinates based on a difference between eigenvalues of the summed covariance matrices.

18. Apparatus for depth mapping, comprising:

an illumination subassembly, which is configured to project a pattern of optical radiation onto an object;

a first image sensor, which is configured to capture a first image of the pattern on the object;

at least a second image sensor, which is configured to capture at least a second image of the object; and

a processor, which is configured to process the first image to generate pattern-based depth data with respect to the object, to process a pair of images including at least the second image to generate stereoscopic depth data with respect to the object, and to combine the pattern-based depth data with the stereoscopic depth data to create a depth map of the object.

19. The apparatus according to claim 18, wherein the pair of the images comprises the first image and the second image.

20. The apparatus according to claim 19, wherein the illumination subassembly is configured to project infrared (IR) radiation onto the object, and wherein the first image sensor is configured to capture the IR radiation that is reflected from the object.

21. The apparatus according to claim 20, wherein the second image comprises a color image of the object.

22. The apparatus according to claim 21, wherein the color image comprises pixels, and the depth map comprises depth values, and wherein the processor is configured to output the color image to a display together with the depth coordinates that are associated with the pixels.

23. The apparatus according to claim 18, wherein the projected pattern comprises multiple spots that are projected onto the object, and wherein the processor is configured to find respective transverse shifts between the spots on the object and the spots in a reference image of the pattern, and to compute the depth data based on the transverse shifts.

24. The apparatus according to claim 18, wherein the processor is configured to associate respective measures of confidence with the pattern-based depth data and stereoscopic depth data, and to select depth coordinates from among the pattern-based and stereoscopic depth data responsively to the respective measures of confidence.

25. The apparatus according to claim 18, wherein the processor is configured to define multiple candidate depth coordinates for each of a plurality of pixels in the depth map, and to select one of the candidate depth coordinates at each pixel for inclusion in the depth map.

26. The apparatus according to claim 25, wherein the multiple candidate depth coordinates comprise, for at least some of the pixels, a null coordinate indicating that no valid depth coordinate was found.

27. The apparatus according to claim 25, wherein the processor is configured to apply weighted tensor voting among the pixels in order to select the one of the candidate depth coordinates based on the candidate depth coordinates at neighboring pixels.

28. The apparatus according to claim 18, wherein the processor is configured to apply a calibration procedure to the first and second images so as to correct for a misalignment between the first and second images.

29. The apparatus according to claim 28, wherein the calibration procedure comprises correcting for a change in alignment between the pattern of optical radiation and the first image sensor.

30. Apparatus for depth mapping, comprising:

at least one image sensor, which is configured to capture at least one image of an object, the image comprising multiple pixels; and

a processor, which is configured to process the at least one image to generate depth data comprising multiple candidate depth coordinates for each of a plurality of the pixels, to apply a weighted voting process to the depth data in order to select one of the candidate depth coordinates at each pixel, and to output a depth map of the object comprising the selected one of the candidate depth coordinates at each pixel.

31. The apparatus according to claim 30, wherein the processor is configured to associate respective measures of confidence with the candidate depth coordinates, and to weight votes for the candidate depth coordinates responsively to the respective measures of confidence.

32. The apparatus according to claim 31, wherein the multiple candidate depth coordinates comprise, for at least some of the pixels, a null coordinate indicating that no valid depth coordinate was found.

33. The apparatus according to claim 31, wherein the processor is configured to apply weighted tensor voting among the pixels in order to select the one of the candidate depth coordinates based on the candidate depth coordinates at neighboring pixels.

34. The apparatus according to claim 33, wherein the processor is configured to compute a weighted sum of covariance matrices over the neighboring pixels, and to select the one of the candidate depth coordinates based on a difference between eigenvalues of the summed covariance matrices.

35. A computer software product, comprising a computer-readable medium in which program instructions are stored, which instructions, when read by a processor, cause the processor to receive a first image of a pattern that has been projected onto an object and to receive at least a second image of the object, and to process the first image to generate pattern-based depth data with respect to the object, to process a pair of images including at least the second image to generate stereoscopic depth data with respect to the object, and to combine the pattern-based depth data with the stereoscopic depth data to create a depth map of the object.

36. A computer software product, comprising a computer-readable medium in which program instructions are stored, which instructions, when read by a processor, cause the processor to receive at least one image of an object, the image comprising multiple pixels, to process the at least one image to generate depth data comprising multiple candidate depth coordinates for each of a plurality of the pixels, to apply a weighted voting process to the depth data in order to select one of the candidate depth coordinates at each pixel, and to output a depth map of the object comprising the selected one of the candidate depth coordinates at each pixel.

37. A method for depth mapping, comprising:

capturing first and second images of an object using first and second image capture subassemblies, respectively;

comparing the first and second images in order to estimate a misalignment between the first and second image capture subassemblies;

processing the first and second images together while correcting for the misalignment so as to generate stereoscopic depth data with respect to the object; and

outputting a depth map comprising the stereoscopic depth data.

38. The method according to claim 37, wherein comparing the first and second images comprises selecting pixels in a first depth map responsively to the depth data, collecting statistics with respect to the selected pixels in subsequent images captured by the first and second image capture subassemblies, and applying the statistics in updating the estimate of the misalignment for use creating a second, subsequent depth map.

39. The method according to claim 37, wherein comparing the first and second images comprises estimating a difference in relative magnification between the first and second images.

40. The method according to claim 37, wherein comparing the first and second images comprises estimating a shift between the first and second images.

41. The method according to claim 40, wherein correcting the misalignment comprises applying corrected shift values x_nom in generating the depth data, incorporating a correction dx_nom given by a formula:

42. Apparatus for depth mapping, comprising:

first and second image capture subassemblies, which are configured to capture respective first and second images of an object; and

a processor, which is configured to compare the first and second images in order to estimate a misalignment between the first and second image capture subassemblies, to process the first and second images together while correcting for the misalignment so as to generate stereoscopic depth data with respect to the object, and to output a depth map comprising the stereoscopic depth data.

43. The apparatus according to claim 42, wherein the processor is configured to select pixels in a first depth map responsively to the depth data, to collect statistics with respect to the selected pixels in subsequent images captured by the first and second image capture subassemblies, and to apply the statistics in updating the estimate of the misalignment for use creating a second, subsequent depth map.

44. The apparatus according to claim 42, wherein the misalignment estimated by the processor comprises a difference in relative magnification between the first and second images.

45. The apparatus according to claim 42, wherein the misalignment estimated by the processor comprises a shift between the first and second images.

46. The apparatus according to claim 45, wherein the processor is configured to apply corrected shift values x_nom in generating the depth data, incorporating a correction dx_nom given by a formula: