Controlled human pose estimation from depth image streams

• US 20090252423A1
• Filed: 05/29/2009
• Published: 10/08/2009
• Est. Priority Date: 12/21/2007
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Zhu, Younding, Dariush, Behzad, Fujimura, Kikuo
• Granted Patent: US 9,165,199 B2
• Time in Patent Office: Days
• US Class Current: 382/209
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

A system, method, and computer program product for estimating human body pose are described. According to one aspect, anatomical features are detected in a depth image of a human actor. The method detects a head, neck, and trunk (H-N-T) template in the depth image, and detects limbs in the depth image based on the H-N-T template. The anatomical features are detected based on the H-N-T template and the limbs. An estimated pose of a human model is estimated based on the detected features and kinematic constraints of the human model.

Claims

1. A computer based method, for estimating a pose of a human actor, the method comprising:

receiving a depth image of the human actor;

detecting a head, neck, and trunk (H-N-T) template of the human actor in the depth image;

detecting limbs of the human actor in the depth image based on the H-N-T template;

detecting a plurality of features of the human actor in the depth image based on the detected H-N-T template and limbs;

augmenting missing features based on previously generated corresponding predicted features; and

estimating a pose of the human actor in a human model based on the augmented features and kinematic constraints of the human model.

2. The method of claim 1, further comprising:

segmenting image regions of the human actor from image regions of background in the depth image.

3. The method of claim 2, wherein segmenting the image regions of the human actor comprises:

identifying image regions in the depth image with depth values exceeding a predetermined depth working volume as background; and

identifying image regions with vertical depth image normal vectors as background.

4. The method of claim 2, wherein detecting the limbs of the human actor in the depth image comprises:

performing a skeleton analysis on the image regions of the human actor to generate a skeleton image of the human actor;

performing distance transformation on the skeleton image to generate a distance transformed skeleton image of the human actor;

detecting end points of the human actor in the distance transformed skeleton image; and

detecting the limbs by forming limb templates that track between the end points and the H-N-T template.

5. The method of claim 4, wherein detecting the limbs of the human actor in the depth image further comprises:

determining whether self occlusion is present in the depth image based on the detected end points; and

responsive to self occlusion being determined present in the depth image, conducting additional analysis of the depth image to detect and label limbs of the human actor.

6. The method of claim 5, wherein determining whether self occlusion is present comprises:

determining whether four of the detected end points correspond to hands and feet of the human actor;

determining whether lengths of skeletonized branches between the detected end points corresponding to the hands and feet of the human actor and corresponding entry points of the H-N-T template exceed a minimum distance;

determining that self occlusion is not present if (1) four of the detected end points correspond to the hands and feet of the human actor, and (2) the lengths of the skeletonized branches exceed the minimum distance; and

determining that self occlusion is present if (1) no more than three of the detected end points correspond to the hands and feet of the human actor, or (2) the length of at least one of the skeletonized branches does not exceed the minimum distance.

7. The method of claim 5, wherein conducting the additional analysis of the depth image comprises:

detecting an open limb region in the distance transformed skeleton image;

detecting a loop region formed by a limb and the H-N-T template in the distance transformed skeleton image; and

conducting depth slicing analysis to detect a limb region occluding the H-N-T template.

8. The method of claim 7, wherein conducting the additional analysis of the depth image further comprises:

calculating a probability for each pixel of a detected limb region for a likelihood of the pixel belonging to a particular limb in the depth image, the probability is calculated based on (1) previously generated predicted limb poses and (2) previous occlusion states of the limbs; and

determining a label to the limb region based on the calculated probabilities of its pixels.

9. The method of claim 8, wherein conducting the additional analysis of the depth image further comprises:

determining an occlusion state of the limb region based on the calculated probabilities of its pixels.

10. The method of claim 4, wherein detecting the plurality of features of the human actor comprises:

localizing the plurality of features of the human actor based on the H-N-T template, the detected end points and corresponding entry points on the H-N-T template, and intersection points on the detected limbs.

11. The method of claim 1, wherein the depth image is taken by a single time-of-flight camera.

12. The method of claim 1, wherein the augmented features comprises:

head center, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, waist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

13. The method of claim 1, further comprising:

generating interpolated features based on the detected plurality of features.

14. The method of claim 1, further comprising:

responsive to detecting multiple candidates for one of the plurality of features in the image, identify a candidate as the one of the plurality of features based on a previously generated corresponding predicted feature.

15. The method of claim 1, wherein estimating the pose of the human actor in the human model further comprises:

tracking the estimated pose of the human model with an observed pose of the human actor.

16. The method of claim 1, further comprising:

generating predict features based on the augmented features and the kinematic constraints of the human model.

17. The method of claim 1, wherein the kinematic constraints of the human model comprises joint limitations and self penetration avoidance.

18. The method of claim 17, further comprises:

constructing a virtual surface surrounding an actual surface of a body segment of the human model;

monitoring a distance between the body segment and an unconnected structure;

detecting that the unconnected structure penetrates the virtual surface;

determining a redirected joint motion that prevents the unconnected structure from colliding with the body segment; and

redirecting the body segment based on the redirected joint motion to avoid colliding with the unconnected structure.

19. A computer program product for estimating a pose of a human actor, the computer program product comprising a computer-readable storage medium containing executable computer program code for performing a method comprising:

receiving a depth image of the human actor;

detecting a head, neck, and trunk (H-N-T) template of the human actor in the depth image;

detecting limbs of the human actor in the depth image based on the H-N-T template;

detecting a plurality of features of the human actor in the depth image based on the detected H-N-T template and limbs;

augmenting missing features based on previously generated corresponding predicted features; and

estimating a pose of the human actor in a human model based on the augmented features and kinematic constraints of the human model.

20. A system for estimating a pose of a human actor, the system comprising:

a computer processor for executing executable computer program code;

a computer-readable storage medium containing the executable computer program code for performing a method comprising;

receiving a depth image of the human actor;

detecting a head, neck, and trunk (H-N-T) template of the human actor in the depth image;

detecting limbs of the human actor in the depth image based on the H-N-T template;

detecting a plurality of features of the human actor in the depth image based on the detected H-N-T template and limbs;

augmenting missing features based on previously generated corresponding predicted features; and

estimating a pose of the human actor in a human model based on the augmented features and kinematic constraints of the human model.