Extraction of skeletons from 3D maps

• US 8,565,479 B2
• Filed: 08/11/2010
• Issued: 10/22/2013
• Est. Priority Date: 08/13/2009
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: PrimeSense Ltd (Apple Inc.)
• Inventors: Gurman, Amiad, Yanir, Tomer, Sali, Erez
• Primary Examiner(s): Lu, Tom Y
• Application Number:
  US12/854,188
• Publication Number:
  US 20110052006A1
• Time in Patent Office: 1,168 Days
• Field of Search: 382/103, 382/154
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30172 Centreline of tubular or elongated structure
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

A method for processing data includes receiving a temporal sequence of depth maps of a scene containing a humanoid form having a head. The depth maps include a matrix of pixels having respective pixel depth values. A digital processor processes at least one of the depth maps so as to find a location of the head and estimates dimensions of the humanoid form based on the location. The processor tracks movements of the humanoid form over the sequence using the estimated dimensions.

Claims

1. A method for processing data, comprising:

receiving a temporal sequence of depth maps of a scene containing a humanoid form having a head, the depth maps comprising a matrix of pixels having respective pixel depth values;

using a digital processor, processing at least one of the depth maps so as to find a location of the head and so as to identify a planar surface corresponding to a floor on which the humanoid form is standing;

extracting a height of the humanoid form from the at least one of the depth maps by measuring a distance from the head to the planar surface;

estimating dimensions of the humanoid form based on the height; and

tracking movements of the humanoid form over the sequence using the estimated dimensions.

2. The method according to claim 1, wherein extracting the height comprises locating a foot of the humanoid form in the at least one of the depth maps, and measuring a distance from the head to the foot.

3. The method according to claim 1, wherein processing the at least one of the depth maps comprises identifying left and right arms of the humanoid form, and searching to find the head between the arms.

4. The method according to claim 3, wherein identifying the left and right arms comprises capturing the at least of the depth maps while the humanoid form stands in a calibration pose, in which the left and right arms are raised.

5. The method according to claim 4, wherein the left and right arms are raised above a shoulder level of the humanoid form in the calibration pose.

6. The method according to claim 3, wherein identifying the left and right arms comprises extracting edges of the humanoid form from the at least one depth map, finding three-dimensional (3D) medial axes and extreme points of limbs of the humanoid form based on the edges, and identifying joints in the limbs based on the medial axes.

7. The method according to claim 6, wherein identifying the joints comprises locating left and right shoulders of the humanoid form, and wherein estimating the dimensions comprises extracting a height of the humanoid form from the at least one of the depth maps based on the location of the head, and computing a width between the shoulders, and estimating the dimensions of other parts of the humanoid form using the height and the width.

9. The method according to claim 1, and comprising refining the estimated dimension responsively to the depth maps in the sequence while tracking the movements.

8. A method for processing data, comprising:

receiving a temporal sequence of depth maps of a scene containing a humanoid form having a head, the depth maps comprising a matrix of pixels having respective pixel depth values;

capturing one or more two-dimensional (2D) images of the humanoid form;

using a digital processor, detecting a face of the humanoid form in the 2D images;

processing the at least one of the depth maps by registering the depth maps with the 2D images, and finding the location of the head in the at least one of the depth maps using the detected face;

estimating dimensions of the humanoid form based on the location of the head; and

tracking movements of the humanoid form over the sequence using the estimated dimensions.

10. Apparatus for processing data, comprising:

an imaging assembly, which is configured to capture a temporal sequence of depth maps of a scene containing a humanoid form having a head, the depth maps comprising a matrix of pixels having respective pixel depth values; and

a processor, which is configured to process at least one of the depth maps so as to find a location of the head and to identify a planar surface corresponding to a floor on which the humanoid form is standing, to extract a height of the humanoid form from the at least one of the depth maps by measuring a distance from the head to the planar surface, to estimate dimensions of the humanoid form based on the height, and to track movements of the humanoid form over the sequence using the estimated dimensions.

11. The apparatus according to claim 10, wherein the processor is configured to extract the height by locating a foot of the humanoid form in the at least one of the depth maps, and measuring a distance from the head to the foot.

12. The apparatus according to claim 10, wherein the processor is configured to identify left and right arms of the humanoid form in the at least one of the depth maps and to find the head by searching between the arms.

13. The apparatus according to claim 12, wherein the at least of the depth maps is captured while the humanoid form stands in a calibration pose, in which the left and right arms are raised.

14. The apparatus according to claim 13, wherein the left and right arms are raised above a shoulder level of the humanoid form in the calibration pose.

15. The apparatus according to claim 12, wherein the processor is configured to identify the left and right arms by extracting edges of the humanoid form from the at least one depth map, finding three-dimensional (3D) medial axes and extreme points of limbs of the humanoid form based on the edges, and identifying joints in the limbs based on the medial axes.

16. The apparatus according to claim 15, wherein the joints identified by the processor comprise left and right shoulders of the humanoid form, and wherein the processor is configured to extract a height of the humanoid form from the at least one of the depth maps based on the location of the head, to compute a width between the shoulders, and to estimate the dimensions of other parts of the humanoid form using the height and the width.

17. The apparatus according to claim 10, wherein the imaging assembly is configured to capture one or more two-dimensional (2D) images of the humanoid form, and wherein the processor is configured to detect a face of the humanoid form in the 2D images, to register the depth maps with the 2D images, and to find the location of the head using the detected face.

18. The apparatus according to claim 10, wherein the processor is configured to refine the estimated dimension responsively to the depth maps in the sequence while tracking the movements.

19. A computer software product, comprising a non-transitory computer-readable medium in which program instructions are stored, which instructions, when read by a computer, cause the computer to receive a temporal sequence of depth maps of a scene containing a humanoid form having a head, the depth maps comprising a matrix of pixels having respective pixel depth values, to process at least one of the depth maps so as to find a location of the head and to identify a planar surface corresponding to a floor on which the humanoid form is standing, to extract a height of the humanoid form from the at least one of the depth maps by measuring a distance from the head to the planar surface, to estimate dimensions of the humanoid form based on the height, and to track movements of the humanoid form over the sequence using the estimated dimensions.

20. The product according to claim 19, wherein the instructions cause the computer to extract the height by locating a foot of the humanoid form in the at least one of the depth maps, and measuring a distance from the head to the foot.

21. The product according to claim 19, wherein the instructions cause the computer to identify left and right arms of the humanoid form in the at least one of the depth maps and to find the head by searching between the arms.

22. The product according to claim 21, wherein the at least of the depth maps is captured while the humanoid form stands in a calibration pose, in which the left and right arms are raised.

23. The product according to claim 22, wherein the left and right arms are raised above a shoulder level of the humanoid form in the calibration pose.

24. The product according to claim 21, wherein the instructions cause the computer to identify the left and right arms by extracting edges of the humanoid form from the at least one depth map, finding three-dimensional (3D) medial axes and extreme points of limbs of the humanoid form based on the edges, and identifying joints in the limbs based on the medial axes.

25. The product according to claim 24, wherein the joints identified by the computer comprise left and right shoulders of the humanoid form, and wherein the instructions cause the computer to extract a height of the humanoid form from the at least one of the depth maps based on the location of the head, to compute a width between the shoulders, and to estimate the dimensions of other parts of the humanoid form using the height and the width.

26. The product according to claim 19, wherein the instructions cause the computer to receive one or more two-dimensional (2D) images of the humanoid form, to detect a face of the humanoid form in the 2D images, to register the depth maps with the 2D images, and to find the location of the head using the detected face.

27. The product according to claim 19, wherein the instructions cause the computer to refine the estimated dimension responsively to the depth maps in the sequence while tracking the movements.