Robust detection and classification of body parts in a depth map

• US 10,043,279 B1
• Filed: 09/22/2016
• Issued: 08/07/2018
• Est. Priority Date: 12/07/2015
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Eshet, Yaron
• Primary Examiner(s): CHEN, XUEMEI G
• Application Number:
  US15/272,455
• Time in Patent Office: 684 Days
• Field of Search: 345158, 345419, 345423, 34820814, 348 46, 348169, 382115, 382128, 382173, 382199
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/66 of image moments or centre of gravity
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

A method for processing data includes receiving a depth map of a scene containing at least a part of a body of a humanoid form. The depth map includes a matrix of pixels having respective pixel depth values. A digital processor extracts from the depth map a curvature map of the scene. The curvature map includes respective curvature values and curvature orientations of at least some of the pixels in the matrix. The depth map is segmented using the curvature values and curvature orientations in the curvature map so as to extract three-dimensional (3D) coordinates of one or more limbs of the humanoid form.

Claims

1. A method for processing data, comprising:

receiving a depth map of a scene containing at least a part of a body of a humanoid form, the depth map comprising a matrix of pixels having respective pixel depth values;

using a digital processor, extracting from the depth map a curvature map of the scene, the curvature map comprising respective curvature values and curvature orientations of at least some of the pixels in the matrix; and

segmenting the depth map using the curvature values and curvature orientations in the curvature map so as to extract three-dimensional (3D) coordinates of one or more limbs of the humanoid form,

wherein segmenting the depth map comprises;

identifying blobs of the pixels in the depth map that correspond to the one or more limbs of the humanoid form;

computing, for each blob, a dominant curvature direction of the blob responsively to the curvature orientations;

defining a respective axis of each limb running perpendicular to the dominant curvature direction of a corresponding blob;

processing the depth map so as to locate ridges running along respective axes of the one or more limbs, each ridge comprising a chain of the pixels whose respective curvature values are local maxima; and

identifying locations and orientations of the one or more limbs based on the ridges.

2. The method according to claim 1, wherein extracting the curvature map comprises computing, for each of the at least some of the pixels in the curvature map, a dominant, signed value of the curvature and a corresponding curvature orientation.

3. The method according to claim 1, wherein computing the dominant curvature direction comprises, for each blob, constructing a histogram of curvature directions of the pixels in the blob, and identifying the dominant curvature direction with a mode of the histogram.

4. The method according to claim 1, wherein segmenting the depth map comprises identifying blobs in the curvature map over which the pixels have respective curvature values that are indicative of a convex surface.

5. The method according to claim 4, wherein identifying the blobs comprises finding edges of the blobs in the depth map at locations of sign changes in the curvature map.

6. Apparatus for processing data, comprising:

an imaging assembly, which is configured to capture a depth map of a scene containing at least a part of a body of a humanoid form, the depth map comprising a matrix of pixels having respective pixel depth values; and

a processor, which is configured to extract from the depth map a curvature map of the scene, the curvature map comprising respective curvature values and curvature orientations of at least some of the pixels in the matrix, and to segment the depth map using the curvature values and curvature orientations in the curvature map so as to extract three-dimensional (3D) coordinates of one or more limbs of the humanoid form,

wherein the processor is configured to identify blobs of the pixels in the depth map that correspond to the one or more limbs of the humanoid form, to compute, for each blob, a dominant curvature direction of the blob responsively to the curvature orientations, to define a respective axis of each limb running perpendicular to the dominant curvature direction of a corresponding blob, to process the depth map so as to locate ridges running along respective axes of the one or more limbs, each ridge comprising a chain of the pixels whose respective curvature values are local maxima, and to identify locations and orientations of the one or more limbs based on the ridges.

7. The apparatus according to claim 6, wherein the curvature map comprises, for each of the at least some of pixels in the curvature map, a dominant, signed value of the curvature and a corresponding curvature orientation.

8. The apparatus according to claim 6, wherein the processor is configured to construct, for each blob, a histogram of curvature directions of the pixels in the blob, and to identify the dominant curvature direction with a mode of the histogram.

9. The apparatus according to claim 6, wherein the processor is configured to segment the depth map by identifying blobs in the curvature map over which the pixels have respective curvature values that are indicative of a convex surface.

10. The apparatus according to claim 9, wherein the processor is configured to find edges of the blobs in the depth map at locations of sign changes in the curvature map.

11. A computer software product, comprising a non-transitory computer-readable medium in which program instructions are stored, which instructions, when read by a computer, cause the computer to receive a depth map of a scene containing at least a part of a body of a humanoid form, the depth map comprising a matrix of pixels having respective pixel depth values, to extract from the depth map a curvature map of the scene, the curvature map comprising respective curvature values and curvature orientations of at least some of the pixels in the matrix, and to segment the depth map using the curvature values and curvature orientations in the curvature map so as to extract three-dimensional (3D) coordinates of one or more limbs of the humanoid form,

wherein the instructions cause the computer to identify blobs of the pixels in the depth map that correspond to the one or more limbs of the humanoid form, to compute, for each blob, a dominant curvature direction of the blob responsively to the curvature orientations, to define a respective axis of each limb running perpendicular to the dominant curvature direction of a corresponding blob, to process the depth map so as to locate ridges running along respective axes of the one or more limbs, each ridge comprising a chain of the pixels whose respective curvature values are local maxima, and to identify locations and orientations of the one or more limbs based on the ridges.

12. The product according to claim 11, wherein the curvature map comprises, for each of the at least some of pixels in the curvature map, a dominant, signed value of the curvature and a corresponding curvature orientation.

13. The product according to claim 11, wherein the instructions cause the computer to construct, for each blob, a histogram of curvature directions of the pixels in the blob, and to identify the dominant curvature direction with a mode of the histogram.

14. The product according to claim 11, wherein the instructions cause the computer to segment the depth map by identifying blobs in the curvature map over which the pixels have respective curvature values that are indicative of a convex surface.

15. The product according to claim 14, wherein the instructions cause the computer to find edges of the blobs in the depth map at locations of sign changes in the curvature map.