Depth mapping with enhanced resolution

• US 9,019,267 B2
• Filed: 10/30/2012
• Issued: 04/28/2015
• Est. Priority Date: 10/30/2012
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Gurman, Amiad
• Primary Examiner(s): BROOME, SAID A
• Application Number:
  US13/663,518
• Publication Number:
  US 20140118335A1
• Time in Patent Office: 910 Days
• Field of Search: None
• US Class Current: 345/419
• CPC Class Codes: G01B 11/254 Projection of a pattern, viewing through a pattern, e.g. moiré
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 15/30 Clipping
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/162 involving graph-based methods
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/426 Graphical representations
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/11 Hand-related biometrics; Hand pose recognition
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00

Abstract

A method for depth mapping includes receiving an image of a pattern of spots that has been projected onto a scene, which includes a feature having a set of elongate appendages, which have respective transverse dimensions that are less than twice an average distance between the spots in the pattern that is projected onto the feature. The image is processed in order to segment and find a three-dimensional (3D) location of the feature. The spots appearing on the feature in the 3D location are connected in order to extract separate, respective contours of the appendages.

Claims

1. A method for depth mapping, comprising:

receiving in a digital processor an image of a pattern of spots that has been projected onto a scene, which includes a feature having a set of elongate appendages, which have respective transverse dimensions that are less than twice an average distance between the spots in the pattern that is projected onto the feature;

processing the image in order to segment and find a three-dimensional (3D) location of the feature; and

connecting the spots appearing on the feature in the 3D location in order to extract separate, respective contours of each of the elongate appendages in the set.

2. The method according to claim 1, wherein the spots have respective positions in the pattern that are uncorrelated, and wherein processing the image comprises computing 3D coordinates of points on the feature based on transverse shifts of the spots in the image.

3. The method according to claim 2, wherein connecting the spots comprises finding depth coordinates of the points on the feature with a resolution finer than a depth increment corresponding to a transverse shift equal to the average distance between the spots in the image.

4. The method according to claim 1, wherein connecting the spots comprises delineating a respective contour of one of the appendages that has a transverse dimension that is less than the average distance between the spots in the pattern.

5. The method according to claim 1, wherein connecting the spots comprises delineating a respective contour of one of the appendages while no more than a single chain of the spots is connected along a length of the one of the appendages.

6. The method according to claim 1, wherein the feature includes a hand, and wherein the appendages are fingers of the hand.

7. The method according to claim 6, wherein the extracted contours are indicative of a posture of the hand and fingers, and wherein the method comprises detecting gestures of the hand, and controlling an application running on a computer responsively to the gestures.

8. The method according to claim 1, wherein connecting the spots comprises computing a first depth value that is characteristic of the feature and a second depth value that is characteristic of a background of the scene behind the feature, and sorting the spots in a vicinity of the feature in the image between the first and second depth values.

9. The method according to claim 8, wherein connecting the spots comprises:

identifying in the image an area of shadow between the appendages;

adding further points to the image in the area of the shadow, and assigning the second depth coordinate to the further points; and

applying the further points in delineating the contours of the appendages.

10. The method according to claim 1, wherein connecting the spots comprises constructing a graph having vertices corresponding to the spots in the image, and identifying cut-edges of the graph in order to find the contours.

11. The method according to claim 10, wherein connecting the spots comprises identifying features of the image in a vicinity of the appendages, and finding the features that correspond to the contours responsively to the graph.

12. Apparatus for depth mapping, comprising:

an imaging assembly, which comprises a sensor that is configured to capture an image of a pattern of spots that has been projected onto a scene, which includes a feature having a set of elongate appendages, which have respective transverse dimensions that are less than twice an average distance between the spots in the pattern that is projected onto the feature; and

a processor, which is configured to process the image in order to segment and find a three-dimensional (3D) location of the feature and to connect the spots appearing on the feature in the 3D location in order to extract separate, respective contours of each of the elongate appendages in the set.

13. The apparatus according to claim 12, wherein the spots have respective positions in the pattern that are uncorrelated, and wherein the processor is configured to compute 3D coordinates of points on the feature based on transverse shifts of the spots in the image.

14. The apparatus according to claim 13, wherein the processor is configured to find depth coordinates of the points on the feature with a resolution finer than a depth increment corresponding to a transverse shift equal to the average distance between the spots in the image.

15. The apparatus according to claim 12, wherein the processor is configured to delineate a respective contour of one of the appendages that has a transverse dimension that is less than the average distance between the spots in the pattern.

16. The apparatus according to claim 12, wherein the processor is configured to delineate a respective contour of one of the appendages while no more than a single chain of the spots is connected along a length of the one of the appendages.

17. The apparatus according to claim 12, wherein the feature includes a hand, and wherein the appendages are fingers of the hand.

18. The apparatus according to claim 17, wherein the extracted contours are indicative of a posture of the hand and fingers, and wherein the processor is configured to detect gestures of the hand based on the extracted contours, and to control an application running on a computer responsively to the gestures.

19. The apparatus according to claim 12, wherein the processor is configured to compute a first depth value that is characteristic of the feature and a second depth value that is characteristic of a background of the scene behind the feature, and to sort the spots in a vicinity of the feature in the image between the first and second depth values.

20. The apparatus according to claim 19, wherein the processor is configured to identify in the image an area of shadow between the appendages, to add further points to the image in the area of the shadow, to assign the second depth coordinate to the further points, and to apply the further points in delineating the contours of the appendages.

21. The apparatus according to claim 12, wherein the processor is configured to construct a graph having vertices corresponding to the spots in the image, and to identify cut-edges of the graph in order to find the contours.

22. The apparatus according to claim 21, wherein the processor is configured to identify features of the image in a vicinity of the appendages, and to find the features that correspond to the contours responsively to the graph.

23. A computer software product, comprising a non-transitory computer-readable medium in which program instructions are stored, which instructions, when read by a processor, cause the processor to receive an image of a pattern of spots that has been projected onto a scene, which includes a feature having a set of elongate appendages, which have respective transverse dimensions that are less than twice an average distance between the spots in the pattern that is projected onto the feature, and to process the image in order to segment and find a three-dimensional (3D) location of the feature and to connect the spots appearing on the feature in the 3D location in order to extract separate, respective contours of each of the elongate appendages in the set.

24. The product according to claim 23, wherein the spots have respective positions in the pattern that are uncorrelated, and wherein the instructions cause the processor to compute 3D coordinates of points on the feature based on transverse shifts of the spots in the image.

25. The product according to claim 24, wherein the instructions cause the processor to find depth coordinates of the points on the feature with a resolution finer than a depth increment corresponding to a transverse shift equal to the average distance between the spots in the image.

26. The product according to claim 23, wherein the instructions cause the processor to delineate a respective contour of one of the appendages that has a transverse dimension that is less than the average distance between the spots in the pattern.

27. The product according to claim 23, wherein the instructions cause the processor to delineate a respective contour of one of the appendages while no more than a single chain of the spots is connected along a length of the one of the appendages.

28. The product according to claim 23, wherein the feature includes a hand, and wherein the appendages are fingers of the hand.

29. The product according to claim 28, wherein the extracted contours are indicative of a posture of the hand and fingers, and wherein the instructions cause the processor to detect gestures of the hand based on the extracted contours, and to control an application running on a computer responsively to the gestures.

30. The product according to claim 23, wherein the instructions cause the processor to compute a first depth value that is characteristic of the feature and a second depth value that is characteristic of a background of the scene behind the feature, and to sort the spots in a vicinity of the feature in the image between the first and second depth values.

31. The product according to claim 30, wherein the instructions cause the processor to identify in the image an area of shadow between the appendages, to add further points to the image in the area of the shadow, to assign the second depth coordinate to the further points, and to apply the further points in delineating the contours of the appendages.

32. The product according to claim 23, wherein the instructions cause the processor to construct a graph having vertices corresponding to the spots in the image, and to identify cut-edges of the graph in order to find the contours.

33. The product according to claim 32, wherein the instructions cause the processor to identify features of the image in a vicinity of the appendages, and to find the features that correspond to the contours responsively to the graph.