LEARNING-BASED POSE ESTIMATION FROM DEPTH MAPS

• US 20140037191A1
• Filed: 10/07/2013
• Published: 02/06/2014
• Est. Priority Date: 09/16/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: PrimeSense Ltd (Apple Inc.)
• Inventors: Litvak, Shai
• Granted Patent: US 8,824,781 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/113 Recognition of static hand signs

Abstract

A method for processing data includes receiving a depth map of a scene containing a humanoid form. Respective descriptors are extracted from the depth map based on the depth values in a plurality of patches distributed in respective positions over the humanoid form. The extracted descriptors are matched to previously-stored descriptors in a database. A pose of the humanoid form is estimated based on stored information associated with the matched descriptors.

Claims

1. A method for processing data, comprising:

receiving a depth map of a scene containing a humanoid form, the depth map comprising a matrix of pixels having respective pixel depth values;

extracting from the depth map respective descriptors based on the depth values in a plurality of patches distributed in respective positions over the humanoid form;

matching the extracted descriptors to previously-stored descriptors in a database; and

estimating a pose of the humanoid form based on stored information associated with the matched descriptors.

2. The method according to claim 1, wherein extracting the respective descriptors comprises dividing each patch into an array of spatial bins, and computing a vector of descriptor values corresponding to the pixel depth values in each of the spatial bins.

3. The method according to claim 2, wherein each patch has a center point, and wherein the spatial bins that are adjacent to the center point have smaller respective areas than the spatial bins at a periphery of the patch.

4. The method according to claim 2, wherein each patch has a center point, and wherein the spatial bins are arranged radially around the center point.

5. The method according to claim 2, wherein the descriptor values are indicative of a distribution of at least one type of depth feature in each bin, selected from the group of depth features consisting of depth edges and depth ridges.

6. The method according to claim 1, wherein matching the extracted descriptors comprises finding a respective approximate nearest neighbor of each of the matched extracted descriptors among the stored descriptors in the database.

7. The method according to claim 1, wherein the descriptors in the database are associated with corresponding pointers to respective locations of body joints, and

wherein estimating the pose comprises applying the pointers to the respective positions of the patches from which the matching descriptors were extracted in order to estimate the locations of the joints of the humanoid form.

8. The method according to claim 9, and comprising creating the database by processing a set of training maps in which ground-truth locations of the body joints are indicated in order to find the corresponding pointers.

9. The method according to claim 1, wherein receiving the depth map comprises receiving a sequence of depth maps, and wherein estimating the pose comprises tracking movement of the humanoid form over multiple frames in the sequence.

10. The method according to claim 9, and comprising controlling a computer application responsively to the tracked movement.

11. Mapping apparatus, comprising:

an imaging assembly, which is configured to provide a depth map of a scene containing a humanoid form, the depth map comprising a matrix of pixels having respective pixel depth values; and

a processor, which is configured to extract from the depth map respective descriptors based on the depth values in a plurality of patches distributed in respective positions over the humanoid form, to match the extracted descriptors to previously-stored descriptors in a database, and to estimate a pose of the humanoid form based on stored information associated with the matched descriptors.

12. The apparatus according to claim 11, wherein the processor is configured to divide each patch into an array of spatial bins, and to compute the descriptor as a vector of descriptor values corresponding to the pixel depth values in each of the spatial bins.

13. The apparatus according to claim 12, wherein each patch has a center point, and wherein the spatial bins that are adjacent to the center point have smaller respective areas than the spatial bins at a periphery of the patch.

14. The apparatus according to claim 12, wherein each patch has a center point, and wherein the spatial bins are arranged radially around the center point.

15. The apparatus according to claim 12, wherein the descriptor values are indicative of a distribution of at least one type of depth feature in each bin, selected from the group of depth features consisting of depth edges and depth ridges.

16. The apparatus according to claim 15, wherein the distribution is indicative of at least one characteristic of the depth features, selected from the group of characteristics consisting of a spatial distribution of the depth features and a directional distribution of the depth features.

17. The apparatus according to claim 11, wherein the processor is configured to match the extracted descriptors by finding a respective approximate nearest neighbor of each of the matched extracted descriptors among the stored descriptors in the database.

18. The apparatus according to claim 11, wherein the descriptors in the database are associated with corresponding pointers to respective locations of body joints, and

wherein the processor is configured to estimate the pose by applying the pointers to the respective positions of the patches from which the matching descriptors were extracted in order to estimate the locations of the joints of the humanoid form.

19. The apparatus according to claim 18, wherein the database is created by processing a set of training maps in which ground-truth locations of the body joints are indicated in order to find the corresponding pointers.

20. The apparatus according to claim 18, wherein the processor is configured to associate respective weights with the estimated locations of the joints provided by the extracted descriptors, and to apply a weighted voting process using the weights to find the locations of the joints.

21. The apparatus according to claim 20, wherein the weights comprise at least one weighting term that is selected from a group of weighting terms consisting of:

a similarity term, based on a descriptor distance between the matched descriptors;

a patch distance term, based on a Euclidean distance between a patch position and a joint location;

a joint distance term, based on a joint distance between a given joint location and a parent joint location that has already been estimated;

a predictive term, based on a previous joint location derived from a preceding depth map;

a variance term, based on a variance of the joint location determined in creating the database; and

a bone length term, based on distance between a current estimated bone length and an expected bone length derived from the locations of the joints.

22. The apparatus according to claim 20, wherein the processor is configured to assess a reliability of the patches providing the estimated locations, and to assign reliability values to the estimated locations based on the assessed reliability.

23. The apparatus according to claim 11, wherein the processor is configured to normalize a depth of the depth map by finding a representative depth coordinate of the humanoid form in the depth map and projecting a point cloud derived from the depth map responsively to the representative depth coordinate, and to apply the normalized depth in matching the descriptors and estimating the pose.

24. The apparatus according to claim 11, wherein the processor is configured to find respective locations of joints of the humanoid form, and to calibrate a scale of the humanoid form by finding a distance between the locations of the joints and scaling the depth map responsively to the distance, and to apply the calibrated scale in matching the descriptors and estimating the pose.

25. The apparatus according to claim 11, wherein the imaging assembly is configured to provide a sequence of depth maps, and wherein the processor is configured to track movement of the humanoid form over multiple frames in the sequence.

26. The apparatus according to claim 25, wherein the processor is configured to control a computer application responsively to the tracked movement.

27. A computer software product, comprising a non-transitory computer-readable medium in which program instructions are stored, which instructions, when read by a computer, cause the computer to receive a depth map of a scene containing a humanoid form, the depth map comprising a matrix of pixels having respective pixel depth values, to extract from the depth map respective descriptors based on the depth values in a plurality of patches distributed in respective positions over the humanoid form, to match the extracted descriptors to previously-stored descriptors in a database, and to estimate a pose of the humanoid form based on stored information associated with the matched descriptors.