Identifying components of a humanoid form in three-dimensional scenes

• US 8,824,737 B2
• Filed: 04/21/2013
• Issued: 09/02/2014
• Est. Priority Date: 05/31/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: PrimeSense Ltd (Apple Inc.)
• Inventors: Gurman, Amiad, Yanir, Tomer, Brand, Michael, Guendelman, Eran
• Primary Examiner(s): Le, Vu
• Assistant Examiner(s): PARK, SOO JIN
• Application Number:
  US13/867,085
• Publication Number:
  US 20130230215A1
• Time in Patent Office: 499 Days
• Field of Search: 382/103, 382/106, 382/107, 382/154
• US Class Current: 382/103
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 15/06 Ray-tracing
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2215/12 Shadow map, environment map
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/507 from shading G06T7/586 takes precedence
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

A method for processing data includes receiving a depth map of a scene containing a humanoid form. The depth map is processed so as to identify three-dimensional (3D) connected components in the scene, each connected component including a set of the pixels that are mutually adjacent and have mutually-adjacent depth values. Separate, first and second connected components are identified as both belonging to the humanoid form, and a representation of the humanoid form is generated including both of the first and second connected components.

Claims

1. A method for processing data, comprising:

receiving a temporal sequence of depth maps of a scene containing a humanoid form, each depth map comprising a matrix of pixels, at least some of which have respective pixel depth values and correspond to respective locations in the scene;

using a digital processor, processing the depth maps so as to identify three-dimensional (3D) connected components in the scene, each connected component comprising a set of the pixels that are mutually adjacent and have mutually-adjacent depth values;

identifying separate, first and second connected components as both belonging to the humanoid form,

wherein identifying the first and second connected components comprises tracking the connected components over multiple, successive depth maps in the sequence by assigning a common identifier to the first and second connected components in a first depth map in the sequence, and labeling the pixels of the first and second connected components in the depth map with the common identifier in order to determine that the first and second connected components belong to the humanoid form in a second, subsequent depth map; and

generating a representation of the humanoid form comprising both of the first and second connected components, such that when the humanoid form in the second depth map is in contact with another object, thereby causing the second connected component to incorporate the other object, the common identifier is used to separate the labeled pixels of the second connected component that belong to the humanoid form from the pixels belonging to other object in order to generate the representation of the humanoid form appearing in the second depth map.

2. The method according to claim 1, wherein processing the depth map comprises constructing a background model of the scene based on the depth map, removing the background model from the depth map in order to generate a foreground map, and identifying the 3D connected components in the foreground map.

3. The method according to claim 1, wherein processing the depth map comprises locating edges in the depth map and blocks of pixels between the edges, and clustering adjacent blocks of the pixels in three dimensions in order to identify the 3D connected components.

4. The method according to claim 1, wherein tracking the connected components comprises using the common identifier to determine that the first and second connected components belong to the humanoid form when the first and second connected components have been separated in the second depth map.

5. The method according to claim 1, wherein tracking the connected components comprises estimating a velocity of motion of the connected components, and wherein generating the representation comprises maintaining an identification of an occluded part of the humanoid form using the estimated velocity.

6. The method according to claim 1, wherein the humanoid form in the depth map is partly occluded by another object, and wherein the first and second connected components correspond to parts of the humanoid form that are separated by the occluding object, and wherein generating the representation comprises incorporating both of the first and second connected components in the representation of the humanoid form.

7. The method according to claim 1, wherein the first and second connected components correspond to first and second parts of the humanoid form, wherein the second part partly occludes the first part in the depth map, and wherein generating the representation comprises incorporating both of the first and second connected components in the representation of the humanoid form.

8. Apparatus for processing data, comprising:

an imaging assembly, which is configured to produce a temporal sequence of depth maps of a scene containing a humanoid form, each depth map comprising a matrix of pixels, at least some of which have respective pixel depth values and correspond to respective locations in the scene; and

a processor, which is configured to process the depth maps so as to identify three-dimensional (3D) connected components in the scene, each connected component comprising a set of the pixels that are mutually adjacent and have mutually-adjacent depth values, and to identify separate, first and second connected components as both belonging to the humanoid form, wherein the processor is configured to track the connected components over multiple, successive depth maps in the sequence by assigning a common identifier to the first and second connected components in a first depth map in the sequence, and labeling the pixels of the first and second connected components in the depth map with the common identifier in order to determine that the first and second connected components belong to the humanoid form in a second, subsequent depth map, and

wherein the processor is configured to generate a representation of the humanoid form comprising both of the first and second connected components, such that when the humanoid form in the second depth map is in contact with another object, thereby causing the second connected component to incorporate the other object, the common identifier is used to separate the labeled pixels of the second connected component that belong to the humanoid form from the pixels belonging to other object in order to generate the representation of the humanoid form appearing in the second depth map.

9. The apparatus according to claim 8, wherein the processor is configured to construct a background model of the scene based on the depth map, to remove the background model from the depth map in order to generate a foreground map, and to identify the 3D connected components in the foreground map.

10. The apparatus according to claim 8, wherein the processor is configured to locate edges in the depth map and blocks of pixels between the edges, and to cluster adjacent blocks of the pixels in three dimensions in order to identify the 3D connected components.

11. The apparatus according to claim 8, wherein the processor is configured to use the common identifier to determine that the first and second connected components belong to the humanoid form when the first and second connected components have been separated in the second depth map.

12. The apparatus according to claim 8, wherein the processor is configured to estimate a velocity of motion of the connected components, and to maintain an identification of an occluded part of the humanoid form using the estimated velocity.

13. The apparatus according to claim 8, wherein the humanoid form in the depth map is partly occluded by another object, and wherein the first and second connected components correspond to parts of the humanoid form that are separated by the occluding object, and wherein the processor is configured to incorporate both of the first and second connected components in the representation of the humanoid form.

14. The apparatus according to claim 8, wherein the first and second connected components correspond to first and second parts of the humanoid form, wherein the second part partly occludes the first part in the depth map, and wherein the processor is configured to incorporate both of the first and second connected components in the representation of the humanoid form.

15. A computer software product, comprising a non-transitory computer-readable medium in which program instructions are stored, which instructions, when read by a computer, cause the computer to receive a temporal sequence of depth maps of a scene containing a humanoid form, each depth map comprising a matrix of pixels, at least some of which have respective pixel depth values and correspond to respective locations in the scene,

wherein the instructions cause the computer to process the depth maps so as to identify three-dimensional (3D) connected components in the scene, each connected component comprising a set of the pixels that are mutually adjacent and have mutually-adjacent depth values, and to identify separate, first and second connected components as both belonging to the humanoid form, wherein the instructions cause the computer to track the connected components over multiple, successive depth maps in the sequence by assigning a common identifier to the first and second connected components in a first depth map in the sequence, and labeling the pixels of the first and second connected components in the depth map with the common identifier in order to determine that the first and second connected components belong to the humanoid form in a second, subsequent depth map, and

wherein the instructions cause the computer to generate a representation of the humanoid form comprising both of the first and second connected components, such that when the humanoid form in the second depth map is in contact with another object, thereby causing the second connected component to incorporate the other object, the common identifier is used to separate the labeled pixels of the second connected component that belong to the humanoid form from the pixels belonging to other object in order to generate the representation of the humanoid form appearing in the second depth map.

16. The product according to claim 15, wherein the instructions cause the computer to construct a background model of the scene based on the depth map, to remove the background model from the depth map in order to generate a foreground map, and to identify the 3D connected components in the foreground map.

17. The product according to claim 15, wherein the instructions cause the computer to locate edges in the depth map and blocks of pixels between the edges, and to cluster adjacent blocks of the pixels in three dimensions in order to identify the 3D connected components.

18. The product according to claim 15, wherein the instructions cause the computer to use the common identifier to determine that the first and second connected components belong to the humanoid form when the first and second connected components have been separated in the second depth map.

19. The product according to claim 15, wherein the instructions cause the computer to estimate a velocity of motion of the connected components, and to maintain an identification of an occluded part of the humanoid form using the estimated velocity.

20. The product according to claim 15, wherein the humanoid form in the depth map is partly occluded by another object, and wherein the first and second connected components correspond to parts of the humanoid form that are separated by the occluding object, and wherein the instructions cause the computer to incorporate both of the first and second connected components in the representation of the humanoid form.

21. The product according to claim 15, wherein the first and second connected components correspond to first and second parts of the humanoid form, wherein the second part partly occludes the first part in the depth map, and wherein the instructions cause the computer to incorporate both of the first and second connected components in the representation of the humanoid form.