Depth mapping with enhanced resolution

• US 9,836,645 B2
• Filed: 03/18/2015
• Issued: 12/05/2017
• Est. Priority Date: 10/30/2012
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Gurman, Amiad
• Primary Examiner(s): Broome, Said
• Application Number:
  US14/661,123
• Publication Number:
  US 20150193657A1
• Time in Patent Office: 993 Days
• Field of Search: None
• CPC Class Codes: G01B 11/254 Projection of a pattern, viewing through a pattern, e.g. moiré
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 15/30 Clipping
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/162 involving graph-based methods
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/426 Graphical representations
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/11 Hand-related biometrics; Hand pose recognition
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00

Abstract

A method for depth mapping includes receiving an image of a pattern of spots that has been projected onto a scene, which includes a hand having fingers. The image is processed in order to segment and find a three-dimensional (3D) location of the hand. Based on the spots appearing on the hand in the 3D location, a first depth value that is characteristic of the hand and a second depth value that is characteristic of a background of the scene behind the hand are computed. The spots in a vicinity of the hand in the image between the first and second depth values are sorted in order to extract separate, respective contours of each of the fingers. The respective contours are processed in order to identify a posture of the hand and fingers.

Claims

1. A method for depth mapping, comprising:

receiving in a digital processor an image of a pattern of spots that has been projected onto a scene, which includes a hand having fingers,

wherein the fingers have a transverse dimension that is less than twice an average distance between the spots in the pattern that is projected onto the scene;

processing the image in order to segment and find a three-dimensional (3D) location of the hand, wherein processing the image comprises computing 3D coordinates of points on the hand and fingers based on transverse shifts of the spots in the image;

based on the spots appearing on the hand in the 3D location, computing a first depth value that is characteristic of the hand and a second depth value that is characteristic of a background of the scene behind the hand;

sorting the spots in a vicinity of the hand in the image between the first and second depth values in order to extract separate, respective contours of each of the fingers; and

processing the respective contours in order to identify a posture of the hand and fingers.

2. The method according to claim 1, wherein sorting the spots comprises delineating a respective contour of at least one of the fingers that has a transverse dimension that is less than the average distance between the spots in the pattern.

3. The method according to claim 1, and comprising finding depth coordinates of the points on the hand and fingers with a resolution finer than a depth increment corresponding to a transverse shift equal to the average distance between the spots in the image.

4. The method according to claim 1, wherein sorting the spots comprises delineating a respective contour of at least one of the fingers while no more than a single chain of the spots is connected along a length of each of the at least one of the fingers.

5. The method according to claim 1, wherein processing the respective contours comprises detecting gestures of the hand, and wherein the method comprises controlling an application running on a computer responsively to the gestures.

6. The method according to claim 1, wherein sorting the spots comprises:

identifying in the image an area of shadow between the fingers;

adding further points to the image in the area of the shadow, and assigning the second depth value to the further points; and

applying the further points in delineating the contours of the fingers.

14. The method according to claim 1, wherein extracting the separate, respective contours comprises drawing the contours by choosing and connecting together points on the cut-edges.

16. The method according to claim 1, wherein sorting the spots comprises constructing a graph having vertices corresponding to the spots in the image, and identifying cut-edges of the graph in order to find the contours, wherein the cut-edges connect between the spots at the first and second depth values.

7. Apparatus for depth mapping, comprising:

an imaging assembly, which is configured to capture an image of a pattern of spots that has been projected onto a scene, which includes a hand having fingers,

wherein the fingers have a transverse dimension that is less than twice an average distance between the spots in the pattern that is projected onto the scene; and

a processor, which is configured to process the image in order to compute 3D coordinates of points on the hand and fingers based on transverse shifts of the spots in the image, to segment and find a three-dimensional (3D) location of the hand, and to compute, based on the spots appearing on the hand in the 3D location, a first depth value that is characteristic of the hand and a second depth value that is characteristic of a background of the scene behind the hand, to sort the spots in a vicinity of the hand in the image between the first and second depth values in order to extract separate, respective contours of each of the fingers, and to process the respective contours in order to identify a posture of the hand and fingers.

8. The apparatus according to claim 7, wherein the processor is configured to delineate a respective contour of at least one of the fingers that has a transverse dimension that is less than the average distance between the spots in the pattern.

9. The apparatus according to claim 7, wherein the processor is configured to find depth coordinates of the points on the hand and fingers with a resolution finer than a depth increment corresponding to a transverse shift equal to the average distance between the spots in the image.

10. The apparatus according to claim 7, wherein the processor is configured to delineate a respective contour of at least one of the fingers while no more than a single chain of the spots is connected along a length of each of the at least one of the fingers.

11. The apparatus according to claim 7, wherein the processor is configured to detect gestures of the hand based on the extracted contours, and to control an application running on a computer responsively to the gestures.

12. The apparatus according to claim 7, wherein the processor is configured to identify in the image an area of shadow between the fingers, to add further points to the image in the area of the shadow, to assign the second depth coordinate to the further points, and to apply the further points in delineating the contours of the fingers.

15. The apparatus according to claim 7, wherein the processor is configured to draw the contours by choosing and connecting together points on the cut-edges.

17. The apparatus according to claim 7, wherein the processor is configured to construct a graph having vertices corresponding to the spots in the image, and to identify cut-edges of the graph in order to find the contours, wherein the cut-edges connect between the spots at the first and second depth values.

13. A computer software product, comprising a non-transitory computer-readable medium in which program instructions are stored, which instructions, when read by a processor, cause the processor to receive a an image of a pattern of spots that has been projected onto a scene, which includes a hand having fingers, wherein the fingers have a transverse dimension that is less than twice an average distance between the spots in the pattern that is projected onto the scene, to process the image in order to compute 3D coordinates of points on the hand and fingers based on transverse shifts of the spots in the image, to segment and find a three-dimensional (3D) location of the hand, and to compute, based on the spots appearing on the hand in the 3D location, a first depth value that is characteristic of the hand and a second depth value that is characteristic of a background of the scene behind the hand, to sort the spots in a vicinity of the hand in the image between the first and second depth values, to extract separate, respective contours of each of the fingers using the identified cut edges, and to process the respective contours in order to identify a posture of the hand and fingers.