LEARNING-BASED ESTIMATION OF HAND AND FINGER POSE

• US 20130236089A1
• Filed: 03/06/2013
• Published: 09/12/2013
• Est. Priority Date: 09/11/2011
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: PrimeSense Ltd (Apple Inc.)
• Inventors: Litvak, Shai, Brailovsky, Leonid, Yanir, Tomer
• Granted Patent: US 9,002,099 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/11 Hand-related biometrics; Hand pose recognition

Abstract

A method for processing data includes receiving a depth map of a scene containing a human hand, the depth map consisting of a matrix of pixels having respective pixel depth values. The method continues by extracting from the depth map respective descriptors based on the depth values in a plurality of patches distributed in respective positions over the human hand, and matching the extracted descriptors to previously-stored descriptors in a database. A pose of the human hand is estimated based on stored information associated with the matched descriptors.

Claims

1. A method for processing data, comprising:

receiving a depth map of a scene containing a human hand, the depth map comprising a matrix of pixels having respective pixel depth values;

extracting from the depth map respective descriptors based on the depth values in a plurality of patches distributed in respective positions over the human hand;

matching the extracted descriptors to previously-stored descriptors in a database; and

estimating a pose of the human hand based on stored information associated with the matched descriptors.

2. The method according to claim 1, wherein estimating the pose comprises applying kinematics based on anatomical constraints of the hand in processing the descriptors.

3. The method according to claim 1, and comprising receiving a color or grayscale image of the human hand, wherein extracting the descriptors comprises incorporating information from the color or grayscale image in the descriptors together with the depth values.

4. The method according to claim 1, wherein estimating the pose comprises detecting that a part of the hand is occluded in the depth map, and excluding the occluded part from estimation of the pose.

5. The method according to claim 4, wherein estimating the pose comprises choosing a most anatomically probable hand configuration in response to detecting that the part of the hand is occluded.

6. The method according to claim 1, wherein estimating the pose comprises expressing the pose in terms of a hand posture description language.

7. The method according to claim 6, wherein the hand description language comprises assigning one of a set of positions to each finger of the hand so as to define postures of the hand.

8. The method according to claim 7, wherein defining postures of the hand comprises defining one or more invalid postures of the hand, and wherein estimating the pose of the hand comprises excluding the one or more invalid postures.

9. The method according to claim 1, wherein the previously-stored descriptors in the database are associated with corresponding pointers to respective locations of anatomical landmarks on the human hand, and

wherein estimating the pose comprises applying the pointers to the respective positions of the patches from which the matching descriptors were extracted in order to estimate the locations of the landmarks.

10. The method according to claim 9, wherein the landmarks comprise at least one element selected from a group comprising a fingertip, a joint, a palm plane, and a base of the hand.

11. The method according to claim 9, wherein estimating the pose comprises reconstructing the pose by applying reverse kinematics using at least one of the locations of the landmarks.

12. The method according to claim 9, and comprising creating the database by processing a set of training maps in which ground-truth locations of the anatomical landmarks are indicated in order to find the corresponding pointers.

13. The method according to claim 9, wherein estimating the pose comprises associating respective weights with the estimated locations of the landmarks, and applying a weighted voting process using the weights to find the locations of the landmarks.

14. The method according to claim 13, wherein associating the respective weights comprises computing the weights based on at least one weighting term that is selected from a group of weighting terms consisting of:

a similarity term, based on a descriptor distance between the matched descriptors;

a patch distance term, based on a Euclidean distance between a patch position and a landmark location;

a landmark distance term, based on a landmark distance between a given landmark location and a parent landmark location that has already been estimated;

a predictive term, based on a previous landmark location derived from a preceding depth map;

a variance term, based on a variance of the landmark location determined in creating the database;

a bone length term, based on a length-difference between a current estimated bone length and an expected bone length derived from locations of joints of a given bone; and

a bone angle term, based on an angle-difference between a current estimated bone angle and an expected bone angle of the given bone derived from anatomical constraints.

15. The method according to claim 14, wherein associating the respective weights comprises assessing a reliability of the patches providing the estimated locations, and assigning reliability values to the estimated locations based on the assessed reliability.

16. The method according to claim 1, and comprising normalizing a depth of the depth map by finding a representative depth coordinate of the human hand in the depth map and projecting a point cloud derived from the depth map responsively to the representative depth coordinate, and applying the normalized depth in matching the descriptors and estimating the pose.

17. The method according to claim 1, wherein estimating the pose comprises finding respective locations of landmarks of the human hand, and wherein the method comprises calibrating a scale of the human hand by finding a distance between the locations of the landmarks and scaling the depth map responsively to the distance, and applying the calibrated scale in matching the descriptors and estimating the pose.

18. The method according to claim 1, wherein receiving the depth map comprises receiving a sequence of depth maps, and wherein estimating the pose comprises tracking movement of the human hand over multiple frames in the sequence.

19. The method according to claim 18, and comprising controlling a computer application responsively to the tracked movement.

20. Mapping apparatus, comprising:

an imaging assembly, which is configured to provide a depth map of a scene containing a human hand, the depth map comprising a matrix of pixels having respective pixel depth values; and

a processor, which is configured to extract from the depth map respective descriptors based on the depth values in a plurality of patches distributed in respective positions over the human hand, to match the extracted descriptors to previously-stored descriptors in a database, and to estimate a pose of the human hand based on stored information associated with the matched descriptors.

21. The apparatus according to claim 20, wherein the processor is configured to apply kinematics based on anatomical constraints of the hand in processing the descriptors.

22. The apparatus according to claim 20, wherein the processor is configured to receive a color or grayscale image of the human hand, and to incorporate information from the color or grayscale image in the descriptors together with the depth values.

23. The apparatus according to claim 20, wherein the processor is configured to detect that a part of the hand is occluded in the depth map, and to exclude the occluded part from estimation of the pose.

24. The apparatus according to claim 23, wherein the processor is configured to choose a most anatomically probable hand configuration in response to detecting that the part of the hand is occluded.

25. The apparatus according to claim 20, wherein the processor is configured to express the pose in terms of a hand posture description language.

26. The apparatus according to claim 25, wherein the hand description language comprises assigning one of a set of positions to each finger of the hand so as to define postures of the hand.

27. The apparatus according to claim 26, wherein the processor is configured to define one or more invalid postures of the hand, and wherein estimating the pose of the hand comprises excluding the one or more invalid postures.

28. The apparatus according to claim 20, wherein the previously-stored descriptors in the database are associated with corresponding pointers to respective locations of anatomical landmarks on the human hand, and

wherein the processor is configured to apply the pointers to the respective positions of the patches from which the matching descriptors were extracted in order to estimate the locations of the landmarks.

29. The apparatus according to claim 28, wherein the landmarks comprise at least one element selected from a group comprising a fingertip, a joint, a palm plane, and a base of the hand.

30. The apparatus according to claim 28, wherein the processor is configured to reconstruct the pose by applying reverse kinematics using at least one of the locations of the landmarks.

31. The apparatus according to claim 28, and comprising creating the database by processing a set of training maps in which ground-truth locations of the anatomical landmarks are indicated in order to find the corresponding pointers.

32. The apparatus according to claim 28, wherein the processor is configured to associate respective weights with the estimated locations of the landmarks, and apply a weighted voting process using the weights to find the locations of the landmarks.

33. The apparatus according to claim 32, wherein associating the respective weights comprises computing the weights based on at least one weighting term that is selected from a group of weighting terms consisting of:

a similarity term, based on a descriptor distance between the matched descriptors;

a patch distance term, based on a Euclidean distance between a patch position and a landmark location;

a landmark distance term, based on a landmark distance between a given landmark location and a parent landmark location that has already been estimated;

a predictive term, based on a previous landmark location derived from a preceding depth map;

a variance term, based on a variance of the landmark location determined in creating the database;

a bone length term, based on a length-difference between a current estimated bone length and an expected bone length derived from locations of joints of a given bone; and

a bone angle term, based on an angle-difference between a current estimated bone angle and an expected bone angle of the given bone derived from anatomical constraints.

34. The apparatus according to claim 33, wherein associating the respective weights comprises assessing a reliability of the patches providing the estimated locations, and assigning reliability values to the estimated locations based on the assessed reliability.

35. The apparatus according to claim 20, wherein the processor is configured to normalize a depth of the depth map by finding a representative depth coordinate of the human hand in the depth map and to project a point cloud derived from the depth map responsively to the representative depth coordinate, and to apply the normalized depth in matching the descriptors and estimating the pose.

36. The apparatus according to claim 20, wherein estimating the pose comprises finding respective locations of landmarks of the human hand, and wherein the processor is configured to calibrate a scale of the human hand by finding a distance between the locations of the landmarks and to scale the depth map responsively to the distance, and to apply the calibrated scale in matching the descriptors and in estimating the pose.

37. The apparatus according to claim 20, wherein the imaging assembly is configured to provide a sequence of depth maps, and wherein the processor is configured to track movement of the human hand over multiple frames in the sequence.

38. The apparatus according to claim 37, wherein the processor is configured to control a computer application responsively to the tracked movement.

39. A computer software product, comprising a computer-readable medium in which program instructions are stored, which instructions, when read by a computer, cause the computer to receive a depth map of a scene containing a human hand, the depth map comprising a matrix of pixels having respective pixel depth values, to extract from the depth map respective descriptors based on the depth values in a plurality of patches distributed in respective positions over the human hand, to match the extracted descriptors to previously-stored descriptors in a database, and to estimate a pose of the human hand based on stored information associated with the matched descriptors.

40. The product according to claim 39, wherein the instructions cause the computer to apply kinematics based on anatomical constraints of the hand in processing the descriptors.

41. The product according to claim 39, wherein the instructions cause the computer to receive a color or grayscale image of the human hand, and to incorporate information from the color or grayscale image in the descriptors together with the depth values.

42. The product according to claim 39, wherein the instructions cause the computer to detect that a part of the hand is occluded in the depth map, and to exclude the occluded part from estimation of the pose.

43. The product according to claim 42, wherein the instructions cause the computer to choose a most anatomically probable hand configuration in response to detecting that the part of the hand is occluded.

44. The product according to claim 39, wherein the instructions cause the computer to express the pose in terms of a hand posture description language.

45. The product according to claim 44, wherein the hand description language comprises assigning one of a set of positions to each finger of the hand so as to define postures of the hand.

46. The product according to claim 45, wherein the instructions cause the computer to define one or more invalid postures of the hand, and to estimate the pose of the hand by excluding the one or more invalid postures.

47. The product according to claim 39, wherein the previously-stored descriptors in the database are associated with corresponding pointers to respective locations of anatomical landmarks on the human hand, and

wherein the instructions cause the computer to estimate the pose by applying the pointers to the respective positions of the patches from which the matching descriptors were extracted in order to estimate the locations of the landmarks.

48. The product according to claim 47, wherein the landmarks comprise at least one element selected from a group comprising a fingertip, a joint, a palm plane, and a base of the hand.

49. The product according to claim 47, wherein the instructions cause the computer to reconstruct the pose by applying reverse kinematics using at least one of the locations of the landmarks.

50. The product according to claim 47, wherein the instructions cause to computer to create the database by processing a set of training maps in which ground-truth locations of the anatomical landmarks are indicated in order to find the corresponding pointers.

51. The product according to claim 47, wherein the instructions cause the computer to associate respective weights with the estimated locations of the landmarks, and to apply a weighted voting process using the weights to find the locations of the landmarks.

52. The product according to claim 51, wherein the weights comprise at least one weighting term that is selected from a group of weighting terms consisting of:

a similarity term, based on a descriptor distance between the matched descriptors;

a patch distance term, based on a Euclidean distance between a patch position and a landmark location;

a landmark distance term, based on a landmark distance between a given landmark location and a parent landmark location that has already been estimated;

a predictive term, based on a previous landmark location derived from a preceding depth map;

a variance term, based on a variance of the landmark location determined in creating the database;

a bone length term, based on a length-difference between a current estimated bone length and an expected bone length derived from locations of joints of a given bone; and

a bone angle term, based on an angle-difference between a current estimated bone angle and an expected bone angle of the given bone derived from constraints by bones other than the given bone.

53. The product according to claim 52, wherein the instructions cause the computer to assess a reliability of the patches providing the estimated locations, and to assign reliability values to the estimated locations based on the assessed reliability.

54. The product according to claim 39, wherein the instructions cause the computer to normalize a depth of the depth map by finding a representative depth coordinate of the human hand in the depth map and to project a point cloud derived from the depth map responsively to the representative depth coordinate, and to apply the normalized depth in matching the descriptors and in estimating the pose.

56. The product according to claim 39, wherein the instructions cause the computer to find respective locations of landmarks of the human hand, and to calibrate a scale of the human hand by finding a distance between the locations of the landmarks and to scale the depth map responsively to the distance, and to apply the calibrated scale in matching the descriptors and estimating the pose.

57. The product according to claim 39, wherein the instructions cause the computer to receive a sequence of depth maps, and to track movement of the human hand over multiple frames in the sequence.

58. The product according to claim 57, wherein the instructions cause the computer to control a computer application responsively to the tracked movement.