Analysis of three-dimensional scenes with a surface model

• US 8,781,217 B2
• Filed: 04/21/2013
• Issued: 07/15/2014
• Est. Priority Date: 05/31/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: PrimeSense Ltd (Apple Inc.)
• Inventors: Gurman, Amiad, Yanir, Tomer, Brand, Michael, Guendelman, Eran
• Primary Examiner(s): Bhatnagar, Anand
• Assistant Examiner(s): PARK, SOO JIN
• Application Number:
  US13/867,083
• Publication Number:
  US 20130230234A1
• Time in Patent Office: 450 Days
• Field of Search: 382/106, 382/173
• US Class Current: 382/154
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 15/06 Ray-tracing
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2215/12 Shadow map, environment map
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/507 from shading G06T7/586 takes precedence
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

A method for processing data includes receiving a depth map of a scene containing a humanoid form. The depth map is processed so as to identify three-dimensional (3D) connected components in the scene, each connected component including a set of the pixels that are mutually adjacent and have mutually-adjacent depth values. Separate, first and second connected components are identified as both belonging to the humanoid form, and a representation of the humanoid form is generated including both of the first and second connected components.

Claims

1. A method for processing data, comprising:

receiving a depth map, captured by an imaging assembly, of a scene containing one or more objects positioned in relation to a planar surface corresponding to a floor on which at least one of the objects is positioned, the depth map comprising a matrix of pixels, at least some of which have respective pixel depth values and correspond to respective locations in the scene;

using a digital processor, processing the depth map so as to identify and generate a parametric model of the planar surface by;

computing local normals at the pixels in the depth map;

processing respective depth values of at least some of the pixels in order to predict a height of the imaging assembly with respect to each pixel;

clustering the pixels according to the local normals and to the height of the imaging assembly predicted with respect to the pixels; and

choosing as the floor a cluster of the pixels that is located, according to the predicted height of the imaging assembly for the cluster, at the lowest plane in space among a set of clusters containing more than a predetermined minimal area in the depth map; and

segmenting one or more of the objects in the scene by removing the planar surface represented by the parametric model from the depth map.

2. The method according to claim 1, wherein at least one of the objects is a humanoid figure, and wherein segmenting the one or more of the objects comprises extracting the humanoid figure from the depth map.

3. The method according to claim 1, wherein removing the planar surface comprises assigning null depth values to the pixels having coordinates within the planar surface.

4. The method according to claim 1, wherein processing the depth map comprises calculating a confidence score by comparing a first number of the pixels within the plane of the floor to a second number of pixels below the plane of the floor, and accepting or rejecting a choice of the floor responsively to the confidence score.

5. Apparatus for processing data, comprising:

an imaging assembly, which is configured to produce a depth map of a scene containing one or more objects positioned in relation to a planar surface corresponding to a floor on which at least one of the objects is positioned, the depth map comprising a matrix of pixels, at least some of which have respective pixel depth values and correspond to respective locations in the scene; and

a processor, which is configured to process the depth map so as to identify and generate a parametric model of the planar surface by;

computing local normals at the pixels in the depth map;

processing respective depth values of at least some of the pixels in order to predict a height of the imaging assembly with respect to each pixel;

clustering the pixels according to the local normals and to the height of the imaging assembly predicted with respect to the pixels; and

choosing as the floor a cluster of the pixels that is located, according to the predicted height of the imaging assembly for the cluster, at the lowest plane in space among a set of clusters containing more than a predetermined minimal area in the depth map, and

to segment one or more of the objects in the scene by removing the planar surface represented by the parametric model from the depth map.

6. The apparatus according to claim 5, wherein at least one of the objects is a humanoid figure, and wherein the processor is configured to extract the humanoid figure from the depth map.

7. The apparatus according to claim 5, wherein the processor is configured to assign null depth values to the pixels having coordinates within the planar surface.

8. The apparatus according to claim 5, wherein the processor is configured to calculate a confidence score by comparing a first number of the pixels within the plane of the floor to a second number of pixels below the plane of the floor, and to accept or reject a choice of the floor responsively to the confidence score.

9. A computer software product, comprising a tangible, non-transitory computer-readable medium in which program instructions are stored, which instructions, when read by a computer, cause the computer to receive a depth map, captured by an imaging assembly, of a scene containing one or more foreground objects positioned in relation to a planar surface corresponding to a floor on which at least one of the objects is positioned, the depth map comprising a matrix of pixels, at least some of which have respective pixel depth values and correspond to respective locations in the scene,

wherein the instructions cause the computer to process the depth map so as to identify and generate a parametric model of the planar surface by:

computing local normals at the pixels in the depth map;

processing respective depth values of at least some of the pixels in order to predict a height of the imaging assembly with respect to each pixel;

clustering the pixels according to the local normals and to the height of the imaging assembly predicted with respect to the pixels; and

choosing as the floor a cluster of the pixels that is located, according to the predicted height of the imaging assembly for the cluster, at the lowest plane in space among a set of clusters containing more than a predetermined minimal area in the depth map, and

to segment one or more of the objects in the scene by removing the planar surface represented by the parametric model from the depth map.

10. The product according to claim 9, wherein at least one of the objects is a humanoid figure, and wherein the instructions cause the computer to extract the humanoid figure from the depth map.

11. The product according to claim 9, wherein the instructions cause the computer to assign null depth values to the pixels having coordinates within the planar surface.

12. The product according to claim 9, wherein the instructions cause the computer to calculate a confidence score by comparing a first number of the pixels within the plane of the floor to a second number of pixels below the plane of the floor, and to accept or reject a choice of the floor responsively to the confidence score.