Controlled human pose estimation from depth image streams

• US 9,098,766 B2
• Filed: 12/19/2008
• Issued: 08/04/2015
• Est. Priority Date: 12/21/2007
• Status: Expired due to Fees

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Dariush, Behzad, Zhu, Youding, Fujimura, Kikuo
• Primary Examiner(s): Vu, Kim
• Assistant Examiner(s): HU, FRED H.
• Application Number:
  US12/317,369
• Publication Number:
  US 20090175540A1
• Time in Patent Office: 2,419 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Abstract

A system, method, and computer program product for estimating upper body human pose are described. According to one aspect, a plurality of anatomical features are detected in a depth image of the human actor. The method detects a head, neck, and torso (H-N-T) template in the depth image, and detects the features in the depth image based on the H-N-T template. An estimated pose of a human model is estimated based on the detected features and kinematic constraints of the human model.

Claims

1. A computer based method for estimating a pose of a human subject, the method comprising:

receiving an image of the human subject;

detecting a plurality of detected features of the human subject on the image, the detected features comprising representations of anatomical body parts on the human subject;

determining that at least one undetected feature is missing or occluded based on the detected features;

augmenting the detected features with at least one predicted feature to replace the at least one undetected feature, the at least one predicted feature generated using a previous pose estimated using a posture model and a previously received image; and

estimating a pose of the human subject using the posture model, the posture model estimating the pose using the detected features, the at least one predicted feature, and a plurality of kinematic constraints, the posture model using an inverse kinematic computation configured to, for each of the expected features;

reduce an error between the expected feature and a corresponding one of the detected features,

except that for the at least one undetected feature, instead reducing an error between the expected feature and the at least one corresponding predicted feature.

2. The method of claim 1, wherein detecting the plurality of features comprises:

detecting a head, neck, and torso (H-N-T) template in the image;

detecting arm blobs in the image based on the H-N-T template;

labeling left and right arms based on the detected arm blobs; and

localizing the plurality of feature points on the labeled arms.

3. The method of claim 2, wherein the image comprises a depth image, and detecting arm blobs comprises at least one of the following:

performing a skeleton analysis to detect the arm blobs; and

performing a depth slicing to detect the arm blobs.

4. The method of claim 3, wherein the depth image is taken by a single time-of-flight camera.

5. The method of claim 2, wherein the plurality of features comprises:

waist, head center, left shoulder, right shoulder, left elbow, right elbow, left wrist and right wrist.

6. The method of claim 1, further comprising:

generating interpolated features based on the detected plurality of features.

7. The method of claim 1, further comprising:

responsive to detecting multiple candidates for one of the plurality of features in the image, identify a candidate as the one of the plurality of features based on a previously generated corresponding predicted feature.

8. The method of claim 1, wherein estimating the pose of the human subject in the posture model further comprises:

tracking the estimated pose of the posture model with an observed pose of the human subject.

9. The method of claim 1, further comprising:

generating predicted features based on the augmented features and the kinematic constraints of the posture model.

10. The method of claim 1, wherein the kinematic constraints of the posture model comprises joint limitations and self penetration avoidance.

11. The method of claim 10, further comprises:

constructing a virtual surface surrounding an actual surface of a body segment of the posture model;

monitoring a distance between the body segment and an unconnected structure;

detecting that the unconnected structure penetrates the virtual surface;

determining a redirected joint motion that prevents the unconnected structure from colliding with the body segment; and

redirecting the body segment based on the redirected joint motion to avoid colliding with the unconnected structure.

21. The method of claim 1, wherein the posture model is further configured to prioritize the features based on a level of confidence for each feature.

22. The method of claim 1, wherein the posture model is further configured to use features from the previous image to estimate the pose.

12. A computer program product for estimating a pose of a human subject, the computer program product comprising a non-transitory computer-readable storage medium containing executable computer program code for performing a method comprising:

receiving an image of the human subject;

detecting a plurality of detected features of the human subject on the image, the detected features comprising representations of anatomical body parts on the human subject;

determining that at least one undetected feature is missing or occluded based on the detected features;

augmenting the detected features with at least one predicted feature to replace the at least one undetected feature, the at least one predicted feature generated using a previous pose estimated using a posture model and a previously received image; and

estimating a pose of the human subject using the posture model, the posture model estimating the pose using the detected features, the at least one predicted feature, and a plurality of kinematic constraints, the posture model using an inverse kinematic computation configured to, for each of the expected features;

reduce an error between the expected feature and a corresponding one of the detected features,

except that for the at least one undetected feature, instead reducing an error between the expected feature and the at least one corresponding predicted feature.

13. The computer program product of claim 12, wherein detecting the plurality of features comprises:

detecting a head, neck, and torso (H-N-T) template in the image;

detecting arm blobs in the image based on the H-N-T template;

labeling left and right arms based on the detected arm blobs; and

localizing the plurality of feature points on the labeled arms.

14. The computer program product of claim 13, wherein the image comprises a depth image, and detecting arm blobs comprises at least one of the following:

performing a skeleton analysis to detect the arm blobs; and

performing a depth slicing to detect the arm blobs.

15. The computer program product of claim 13, wherein the plurality of features comprises:

waist, head center, left shoulder, right shoulder, left elbow, right elbow, left wrist and right wrist.

16. The computer program product of claim 12, wherein the method further comprises:

generating interpolated features based on the detected plurality of features.

17. A system for estimating a pose of a human subject, the system comprising:

a computer processor for executing executable computer program code;

a computer-readable storage medium containing the executable computer program code for performing a method comprising;

receiving an image of the human subject;

detecting a plurality of detected features of the human subject on the image, the detected features comprising representations of anatomical body parts on the human subject;

determining that at least one undetected feature is missing or occluded based on the detected features;

augmenting the detected features with at least one predicted feature to replace the at least one undetected feature, the at least one predicted feature generated using a previous pose estimated using a posture model and a previously received image; and

estimating a pose of the human subject using the posture model, the posture model estimating the pose using the detected features, the at least one predicted feature, and a plurality of kinematic constraints, the posture model using an inverse kinematic computation configured to for each of the expected features;

reduce an error between the expected feature and a corresponding one of the detected features,

except that for the at least one undetected feature, instead reducing an error between the expected feature and the at least one corresponding predicted feature.

18. The system of claim 17, wherein detecting the plurality of features comprises:

detecting a head, neck, and torso (H-N-T) template in the image;

detecting arm blobs in the image based on the H-N-T template;

labeling left and right arms based on the detected arm blobs; and

localizing the plurality of feature points on the labeled arms.

19. The system of claim 18, wherein the image comprises a depth image, and detecting arm blobs comprises at least one of the following:

performing a skeleton analysis to detect the arm blobs; and

performing a depth slicing to detect the arm blobs.

20. The system of claim 18, wherein the plurality of features comprises:

waist, head center, left shoulder, right shoulder, left elbow, right elbow, left wrist and right wrist.