Modeling of humanoid forms from depth maps

• US 8,249,334 B2
• Filed: 05/10/2007
• Issued: 08/21/2012
• Est. Priority Date: 05/11/2006
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: PrimeSense Ltd (Apple Inc.)
• Inventors: Berliner, Tamir, Hendel, Ziv, Shpunt, Alexander, Danowitz, Jeffrey, Rais, Dmitri, Mor, Oren, Shpigelmacher, Michael
• Primary Examiner(s): Bella, Matthew
• Assistant Examiner(s): Li, Ruiping
• Application Number:
  US12/300,086
• Publication Number:
  US 20100034457A1
• Time in Patent Office: 1,930 Days
• Field of Search: 382/154, 382/103, 382/106, 382/107, 345/419, 356/511, 356/35.5
• US Class Current: 382/154
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Abstract

A computer-implemented method includes receiving a depth map (30) of a scene containing a body of a humanoid subject (28). The depth map includes a matrix of pixels (32), each corresponding to a respective location in the scene and having a respective pixel value indicative of a distance from a reference location to the respective location. The depth map is segmented so as to find a contour (64) of the body. The contour is processed in order to identify a torso (70) and one or more limbs (76, 78, 80, 82) of the subject. An input is generated to control an application program running on a computer by analyzing a disposition of at least one of the identified limbs in the depth map.

Claims

1. A computer-implemented method for processing data, comprising:

receiving a depth map of a scene containing a body of a humanoid subject, the depth map comprising a matrix of pixels, each pixel corresponding to a respective location in the scene and having a respective pixel value indicative of a distance from a reference location to the respective location;

segmenting the depth map so as to find a contour of the body;

processing the contour in order to identify a torso and one or more limbs of the subject by;

identifying an axis of the body in the depth map;

computing a transverse thickness of the contour along multiple first lines perpendicular to the axis;

computing a longitudinal thickness of the contour along multiple second lines parallel to the axis;

designating as the torso a part of the area within the contour that is defined by the first and second lines along which the transverse thickness and longitudinal thickness are no less than a specified transverse threshold and longitudinal threshold, respectively; and

identifying a second part of the area that is connected to the torso as at least one of the limbs; and

generating an input to control an application program running on a computer by analyzing a disposition of at least one of the identified limbs in the depth map.

2. The method according to claim 1, wherein receiving the depth map comprises acquiring the depth map without affixing any dedicated marker to the body.

3. The method according to claim 1, wherein receiving depth map comprises acquiring the depth map using a single, stationary imaging device.

4. The method according to claim 3, wherein acquiring the depth map comprises projecting an uncorrelated pattern of spots onto the subject, capturing an image of the projected pattern using the imaging device, and processing the image in order to generate the depth map.

5. The method according to claim 4, wherein projecting the uncorrelated pattern of spots comprises forming a primary speckle pattern using a coherent light source.

6. The method according to claim 1, wherein receiving the depth map comprises acquiring a sequence of depth maps at a frame rate of at least 25 frames/sec, and wherein generating the control input comprises updating the control input, responsively to the sequence of depth maps, at an update rate that is equal at least to the frame rate.

7. The method according to claim 1, wherein segmenting the depth map comprises identifying a background object in the scene that is not connected to the body, and removing the background object from the scene prior to finding the contour.

8. The method according to claim 1, wherein identifying the second part of the area comprises determining, based on the depth map, a representative distance of the torso from the reference location, and identifying as an arm of the subject a region overlying the torso in which the distance of the pixels is less than the representative distance.

9. The method according to claim 1, wherein analyzing the disposition comprises finding one or more respective medial axes of the one or more limbs.

10. The method according to claim 9, wherein finding the one or more respective medial axes comprises iteratively removing the pixels from at least one of the limbs.

11. The method according to claim 9, wherein the medial axes are curved, and wherein analyzing the disposition comprises deriving stick-figure representations of the one or more limbs by finding straight segments that fit the medial axes.

12. The method according to claim 1, wherein the one or more limbs comprise an arm of the subject, wherein analyzing the disposition comprises deriving a stick-figure representation of the arm by identifying a shoulder point on the torso and a palm point on the arm, and identifying two straight segments that pass through the shoulder and palm points, respectively, and meet at an elbow of the arm.

13. Apparatus for processing data, comprising:

an imaging device, which is configured to capture an image of a scene containing a body of a humanoid subject; and

a processor, which is coupled to process the image so as to create a depth map, which comprises a matrix of pixels, each pixel corresponding to a respective location in the scene and having a respective pixel value indicative of a distance from a reference location to the respective location, and which is configured to segment the depth map so as to find a contour of the body, to process the contour in order to identify a torso and one or more limbs of the subject, and to generate an input to control an application program running on a computer by analyzing a disposition of at least one of the identified limbs in the depth map,

wherein the processor is configured to identify the torso and the one or more limbs by identifying an axis of the body in the depth map, computing a transverse thickness of the contour along multiple first lines perpendicular to the axis, computing a longitudinal thickness of the contour along multiple second lines parallel to the axis, designating as the torso a part of the area within the contour defined by the first and second lines along which the transverse thickness and longitudinal are no less than a specified transverse threshold longitudinal threshold, respectively, and identifying a second part of the area that is connected to the torso as at least one of the limbs.

14. The apparatus according to claim 13, wherein the processor is configured to create the depth snap without any dedicated marker having been affixed to the body.

15. The apparatus according to claim 13, wherein the processor is configured to create the depth map using a single, stationary imaging device.

16. The apparatus according to claim 15, wherein the imaging device is configured to project an uncorrelated pattern of spots onto the subject, and to capture the image of the projected pattern.

17. The apparatus according to claim 16, wherein the uncorrelated pattern of spots comprises a primary speckle pattern formed by a coherent light source.

18. The apparatus according to claim 13, wherein the processor is configured to generate a sequence of depth maps at a frame rate of at least 25 frames/sec, responsively to images captured by the imaging device, and to update the control input, responsively to the sequence of depth maps, at an update rate that is equal at least to the frame rate.

19. The apparatus according to claim 13, wherein the processor is configured to identify a background object in the scene that is not connected to the body, and to remove the background object from the scene prior to finding the contour.

20. The apparatus according to claim 13, wherein the processor is configured to determine, based on the depth map, a representative distance of the torso from the reference location, and to identify as an arm of the subject a region overlying the torso in which the distance of the pixels is less than the representative distance.

21. The apparatus according to claim 13, wherein the processor is configured to analyze the contour so as to find one or more respective medial axes of the one or more limbs.

22. The apparatus according to claim 21, wherein the processor is configured to find the one or more respective medial axes by iteratively eroding the contour surrounding at least one of the limbs.

23. The apparatus according to claim 21, wherein the medial axes are curved, and wherein the processor is configured to derive stick-figure representations of the one or more limbs by finding straight segments that fit the medial axes.

24. The apparatus according to claim 13, wherein the one or more limbs comprise an arm of the subject, and wherein the processor is configured to derive a stick-figure representation of the arm by identifying a shoulder point on the torso and a palm point on the arm, and identifying two straight segments that pass through the shoulder and palm points, respectively, and meet at an elbow of the arm.

25. A computer software product, comprising a non-transitory computer-readable medium in which program instructions are stored, which instructions, when read by a computer, cause the computer to acquire a depth map of a scene containing a body of a humanoid subject, the depth map comprising a matrix of pixels, each pixel corresponding to a respective location in the scene and having a respective pixel value indicative of distance from a reference location to the respective location, and to segment the depth map so as to find a contour of the body, to process the contour in order to identify a torso and one or more limbs of the subject, and to generate an input to control an application program running on a computer by lysing a disposition of at least one of the identified limbs in the depth map,

wherein the instructions cause the computer to identify the torso and the one limbs by identifying an axis of the body in the depth map, computing a transverse thickness of the contour along multiple first lines perpendicular to the axis, computing a longitudinal thickness of the contour along multiple second lines parallel to the axis designating as the torso a part of the area within contour that is defined by the first and second lines along which the transverse thickness and longitudinal thickness are no less than a specified transverse threshold and longitudinal threshold, respectively, and identifying a second part of the area that is connected to the torso as at least one of the limbs.

26. The product according to claim 25, wherein the instructions cause the computer to create the depth map without any dedicated marker having been affixed to the body.

27. The product according to claim 25, wherein the instructions cause the computer to create the depth map using a single, stationary imaging device.

28. The product according to claim 27, wherein an uncorrelated pattern of spots is projected onto the subject, and wherein the instructions cause the computer to create the depth map by processing an image of the projected pattern.

29. The product according to claim 28, wherein the uncorrelated pattern of spots comprises a primary speckle pattern formed by a coherent light source.

30. The product according to claim 25, wherein the instructions cause the computer to generate a sequence of depth maps at a frame rate of at least 25 frames/sec, and to update the control input, responsively to the sequence of depth maps, at an update rate that is equal at least to the frame rate.

31. The product according to claim 25, wherein the instructions cause the computer to identify a background object in the scene that is not connected to the body, and to remove the background object from the scene prior to finding the contour.

32. The product according to claim 25, wherein the instructions cause the computer to determine, based on the depth map, a representative distance of the torso from the reference location, and to identify as an arm of the subject a region overlying the torso in which the distance of the pixels is less than the representative distance.

33. The product according to claim 25, wherein the instructions cause the computer to analyze the contour so as to find one or more respective medial axes of the one or more limbs.

34. The product according to claim 33, wherein the instructions cause the computer to find the one or more respective medial axes by iteratively eroding the contour surrounding at least one of the limbs.

35. The product according to claim 33, wherein the medial axes are curved, and wherein the instructions cause the computer to derive stick-figure representations of the one or more limbs by finding straight segments that fit the medial axes.

36. The product according to claim 25, wherein the one or more limbs comprise an arm of the subject, and wherein the instructions cause the computer to derive a stick-figure representation of the arm by identifying a shoulder point on the torso and a palm point on the arm, and identifying two straight segments that pass through the shoulder and palm points, respectively, and meet at an elbow of the arm.