EXTRACTION OF SKELETONS FROM 3D MAPS

• US 20110052006A1
• Filed: 08/11/2010
• Published: 03/03/2011
• Est. Priority Date: 08/13/2009
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: PrimeSense Ltd (Apple Inc.)
• Inventors: Yanir, Tomer, Gurman, Amiad, Sali, Erez
• Granted Patent: US 8,565,479 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30172 Centreline of tubular or elongated structure
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

A method for processing data includes receiving a temporal sequence of depth maps of a scene containing a humanoid form having a head. The depth maps include a matrix of pixels having respective pixel depth values. A digital processor processes at least one of the depth maps so as to find a location of the head and estimates dimensions of the humanoid form based on the location. The processor tracks movements of the humanoid form over the sequence using the estimated dimensions.

Claims

1. A method for processing data, comprising:

receiving a temporal sequence of depth maps of a scene containing a humanoid form having a head, the depth maps comprising a matrix of pixels having respective pixel depth values;

using a digital processor, processing at least one of the depth maps so as to find a location of the head;

estimating dimensions of the humanoid form based on the location; and

tracking movements of the humanoid form over the sequence using the estimated dimensions.

2. The method according to claim 1, wherein estimating the dimension comprises extracting a height of the humanoid form from the at least one of the depth maps based on the location of the head.

3. The method according to claim 2, wherein extracting the height comprises locating a foot of the humanoid form in the at least one of the depth maps, and measuring a distance from the head to the foot.

4. The method according to claim 2, wherein extracting the height comprises processing the at least one of the depth maps so as to identify a planar surface corresponding to a floor on which the humanoid form is standing, and measuring a distance from the head to the planar surface.

5. The method according to claim 1, wherein processing the at least one of the depth maps comprises identifying left and right arms of the humanoid form, and searching to find the head between the arms.

6. The method according to claim 5, wherein identifying the left and right arms comprises capturing the at least of the depth maps while the humanoid form stands in a calibration pose, in which the left and right arms are raised.

7. The method according to claim 6, wherein the left and right arms are raised above a shoulder level of the humanoid form in the calibration pose.

8. The method according to claim 5, wherein identifying the left and right arms comprises extracting edges of the humanoid form from the at least one depth map, finding three-dimensional (3D) medial axes and extreme points of limbs of the humanoid form based on the edges, and identifying joints in the limbs based on the medial axes.

9. The method according to claim 8, wherein identifying the joints comprises locating left and right shoulders of the humanoid form, and wherein estimating the dimensions comprises extracting a height of the humanoid form from the at least one of the depth maps based on the location of the head, and computing a width between the shoulders, and estimating the dimensions of other parts of the humanoid form using the height and the width.

10. The method according to claim 1, and comprising capturing one or more two-dimensional (2D) images of the humanoid form, and detecting a face of the humanoid form in the 2D images, wherein processing the at least one of the depth maps comprises registering the depth maps with the 2D images, and finding the location of the head using the detected face.

11. The method according to claim 1, and comprising refining the estimated dimension responsively to the depth maps in the sequence while tracking the movements.

12. Apparatus for processing data, comprising:

an imaging assembly, which is configured to capture a temporal sequence of depth maps of a scene containing a humanoid form having a head, the depth maps comprising a matrix of pixels having respective pixel depth values; and

a processor, which is configured to process at least one of the depth maps so as to find a location of the head, to estimate dimensions of the humanoid form based on the location, and to track movements of the humanoid form over the sequence using the estimated dimensions.

13. The apparatus according to claim 12, wherein the processor is configured to extract a height of the humanoid form from the at least one of the depth maps based on the location of the head.

14. The apparatus according to claim 13, wherein the processor is configured to extract the height by locating a foot of the humanoid form in the at least one of the depth maps, and measuring a distance from the head to the foot.

15. The apparatus according to claim 13, wherein the processor is configured to extract the height by processing the at least one of the depth maps so as to identify a planar surface corresponding to a floor on which the humanoid form is standing, and measuring a distance from the head to the planar surface.

16. The apparatus according to claim 12, wherein the processor is configured to identify left and right arms of the humanoid form in the at least one of the depth maps and to find the head by searching between the arms.

17. The apparatus according to claim 16, wherein the at least of the depth maps is captured while the humanoid form stands in a calibration pose, in which the left and right arms are raised.

18. The apparatus according to claim 17, wherein the left and right arms are raised above a shoulder level of the humanoid form in the calibration pose.

19. The apparatus according to claim 16, wherein the processor is configured to identify the left and right arms by extracting edges of the humanoid form from the at least one depth map, finding three-dimensional (3D) medial axes and extreme points of limbs of the humanoid form based on the edges, and identifying joints in the limbs based on the medial axes.

20. The apparatus according to claim 19, wherein the joints identified by the processor comprise left and right shoulders of the humanoid form, and wherein the processor is configured to extract a height of the humanoid form from the at least one of the depth maps based on the location of the head, to compute a width between the shoulders, and to estimate the dimensions of other parts of the humanoid form using the height and the width.

21. The apparatus according to claim 12, wherein the imaging assembly is configured to capture one or more two-dimensional (2D) images of the humanoid form, and wherein the processor is configured to detect a face of the humanoid form in the 2D images, to register the depth maps with the 2D images, and to find the location of the head using the detected face.

22. The apparatus according to claim 12, wherein the processor is configured to refine the estimated dimension responsively to the depth maps in the sequence while tracking the movements.

23. A computer software product, comprising a computer-readable medium in which program instructions are stored, which instructions, when read by a computer, cause the computer to receive a temporal sequence of depth maps of a scene containing a humanoid form having a head, the depth maps comprising a matrix of pixels having respective pixel depth values, to process at least one of the depth maps so as to find a location of the head, to estimate dimensions of the humanoid form based on the location, and to track movements of the humanoid form over the sequence using the estimated dimensions.

24. The product according to claim 23, wherein the instructions cause the computer to extract a height of the humanoid form from the at least one of the depth maps based on the location of the head.

25. The product according to claim 24, wherein the instructions cause the computer to extract the height by locating a foot of the humanoid form in the at least one of the depth maps, and measuring a distance from the head to the foot.

26. The product according to claim 24, wherein the instructions cause the computer to extract the height by processing the at least one of the depth maps so as to identify a planar surface corresponding to a floor on which the humanoid form is standing, and measuring a distance from the head to the planar surface.

27. The product according to claim 23, wherein the instructions cause the computer to identify left and right arms of the humanoid form in the at least one of the depth maps and to find the head by searching between the arms.

28. The product according to claim 27, wherein the at least of the depth maps is captured while the humanoid form stands in a calibration pose, in which the left and right arms are raised.

29. The product according to claim 28, wherein the left and right arms are raised above a shoulder level of the humanoid form in the calibration pose.

30. The product according to claim 27, wherein the instructions cause the computer to identify the left and right arms by extracting edges of the humanoid form from the at least one depth map, finding three-dimensional (3D) medial axes and extreme points of limbs of the humanoid form based on the edges, and identifying joints in the limbs based on the medial axes.

31. The product according to claim 30, wherein the joints identified by the computer comprise left and right shoulders of the humanoid form, and wherein the instructions cause the computer to extract a height of the humanoid form from the at least one of the depth maps based on the location of the head, to compute a width between the shoulders, and to estimate the dimensions of other parts of the humanoid form using the height and the width.

32. The product according to claim 23, wherein the instructions cause the computer to receive one or more two-dimensional (2D) images of the humanoid form, to detect a face of the humanoid form in the 2D images, to register the depth maps with the 2D images, and to find the location of the head using the detected face.

33. The product according to claim 23, wherein the instructions cause the computer to refine the estimated dimension responsively to the depth maps in the sequence while tracking the movements.