Depth mapping with enhanced resolution

• US 20150193657A1
• Filed: 03/18/2015
• Published: 07/09/2015
• Est. Priority Date: 10/30/2012
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Gurman, Amiad
• Granted Patent: US 9,836,645 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G01B 11/254 Projection of a pattern, viewing through a pattern, e.g. moiré
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 15/30 Clipping
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/162 involving graph-based methods
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/426 Graphical representations
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/11 Hand-related biometrics; Hand pose recognition
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00

Abstract

A method for depth mapping includes receiving an image of a pattern of spots that has been projected onto a scene, which includes a hand having fingers. The image is processed in order to segment and find a three-dimensional (3D) location of the hand. Based on the spots appearing on the hand in the 3D location, a first depth value that is characteristic of the hand and a second depth value that is characteristic of a background of the scene behind the hand are computed. The spots in a vicinity of the hand in the image between the first and second depth values are sorted in order to extract separate, respective contours of each of the fingers. The respective contours are processed in order to identify a posture of the hand and fingers.

Claims

1. A method for depth mapping, comprising:

receiving in a digital processor an image of a pattern of spots that has been projected onto a scene, which includes a hand having fingers;

processing the image in order to segment and find a three-dimensional (3D) location of the hand;

based on the spots appearing on the hand in the 3D location, computing a first depth value that is characteristic of the hand and a second depth value that is characteristic of a background of the scene behind the hand, and sorting the spots in a vicinity of the hand in the image between the first and second depth values in order to extract separate, respective contours of each of the fingers; and

processing the respective contours in order to identify a posture of the hand and fingers.

2. The method according to claim 1, wherein the spots have respective positions in the pattern that are uncorrelated, and wherein processing the image comprises computing 3D coordinates of points on the hand and fingers based on transverse shifts of the spots in the image.

3. The method according to claim 2, wherein the fingers have a transverse dimension that is less than twice an average distance between the spots in the pattern that is projected onto the feature.

4. The method according to claim 3, wherein sorting the spots comprises delineating a respective contour of at least one of the fingers that has a transverse dimension that is less than the average distance between the spots in the pattern.

5. The method according to claim 2, and comprising finding depth coordinates of the points on the hand and fingers with a resolution finer than a depth increment corresponding to a transverse shift equal to the average distance between the spots in the image.

6. The method according to claim 1, wherein sorting the spots comprises delineating a respective contour of at least one of the fingers while no more than a single chain of the spots is connected along a length of each of the at least one of the fingers.

7. The method according to claim 1, wherein processing the respective contours comprises detecting gestures of the hand, and wherein the method comprises controlling an application running on a computer responsively to the gestures.

8. The method according to claim 1, wherein sorting the spots comprises:

identifying in the image an area of shadow between the fingers;

adding further points to the image in the area of the shadow, and assigning the second depth value to the further points; and

applying the further points in delineating the contours of the fingers.

9. The method according to claim 1, wherein sorting the spots comprises constructing a graph having vertices corresponding to the spots in the image, and identifying cut-edges of the graph in order to find the contours.

10. Apparatus for depth mapping, comprising:

an imaging assembly, which is configured to capture an image of a pattern of spots that has been projected onto a scene, which includes a hand having fingers; and

a processor, which is configured to process the image in order to segment and find a three-dimensional (3D) location of the hand, and to compute, based on the spots appearing on the hand in the 3D location, a first depth value that is characteristic of the hand and a second depth value that is characteristic of a background of the scene behind the hand, and to sort the spots in a vicinity of the hand in the image between the first and second depth values in order to extract separate, respective contours of each of the fingers, and to process the respective contours in order to identify a posture of the hand and fingers.

11. The apparatus according to claim 10, wherein the spots have respective positions in the pattern that are uncorrelated, and wherein the processor is configured to compute 3D coordinates of points on the hand and fingers based on transverse shifts of the spots in the image.

12. The method according to claim 11, wherein the fingers have a transverse dimension that is less than twice an average distance between the spots in the pattern that is projected onto the feature.

13. The apparatus according to claim 12, wherein the processor is configured to delineate a respective contour of at least one of the fingers that has a transverse dimension that is less than the average distance between the spots in the pattern.

14. The apparatus according to claim 11, wherein the processor is configured to find depth coordinates of the points on the hand and fingers with a resolution finer than a depth increment corresponding to a transverse shift equal to the average distance between the spots in the image.

15. The apparatus according to claim 10, wherein the processor is configured to delineate a respective contour of at least one of the fingers while no more than a single chain of the spots is connected along a length of each of the at least one of the fingers.

16. The apparatus according to claim 10, wherein the processor is configured to detect gestures of the hand based on the extracted contours, and to control an application running on a computer responsively to the gestures.

17. The apparatus according to claim 10, wherein the processor is configured to identify in the image an area of shadow between the fingers, to add further points to the image in the area of the shadow, to assign the second depth coordinate to the further points, and to apply the further points in delineating the contours of the fingers.

18. The apparatus according to claim 10, wherein the processor is configured to construct a graph having vertices corresponding to the spots in the image, and to identify cut-edges of the graph in order to find the contours.

19. A computer software product, comprising a non-transitory computer-readable medium in which program instructions are stored, which instructions, when read by a processor, cause the processor to receive a an image of a pattern of spots that has been projected onto a scene, which includes a hand having fingers, to process the image in order to segment and find a three-dimensional (3D) location of the hand, and to compute, based on the spots appearing on the hand in the 3D location, a first depth value that is characteristic of the hand and a second depth value that is characteristic of a background of the scene behind the hand, and to sort the spots in a vicinity of the hand in the image between the first and second depth values in order to extract separate, respective contours of each of the fingers, and to process the respective contours in order to identify a posture of the hand and fingers.

20. The product according to claim 19, wherein the spots have respective positions in the pattern that are uncorrelated, and wherein the instructions cause the processor to compute 3D coordinates of points on the feature based on transverse shifts of the spots in the image.