Controlled human pose estimation from depth image streams

• US 20090175540A1
• Filed: 12/19/2008
• Published: 07/09/2009
• Est. Priority Date: 12/21/2007
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Zhu, Youding, Dariush, Behzad, Fujimura, Kikuo
• Granted Patent: US 9,098,766 B2
• Time in Patent Office: Days
• US Class Current: 382/195
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Abstract

A system, method, and computer program product for estimating upper body human pose are described. According to one aspect, a plurality of anatomical features are detected in a depth image of the human actor. The method detects a head, neck, and torso (H-N-T) template in the depth image, and detects the features in the depth image based on the H-N-T template. An estimated pose of a human model is estimated based on the detected features and kinematic constraints of the human model.

Claims

1. A computer based method for estimating a pose of a motion generator, the method comprising:

receiving an image of the motion generator;

detecting a plurality of features of the motion generator on the image;

augmenting missing features based on previously generated corresponding predicted features; and

estimating a pose of the motion generator in a motion generator model based on the augmented features and kinematic constraints of the motion generator model.

2. The method of claim 1, wherein the motion generator comprises a human actor, and wherein detecting the plurality of features comprises:

detecting a head, neck, and torso (H-N-T) template in the image;

detecting arm blobs in the image based on the H-N-T template;

labeling left and right arms based on the detected arm blobs; and

localizing the plurality of feature points on the labeled arms.

3. The method of claim 2, wherein the image comprises a depth image, and detecting arm blobs comprises at least one of the following:

performing a skeleton analysis to detect the arm blobs; and

performing a depth slicing to detect the arm blobs.

4. The method of claim 3, wherein the depth image is taken by a single time-of-flight camera.

5. The method of claim 2, wherein the plurality of features comprises:

waist, head center, left shoulder, right shoulder, left elbow, right elbow, left wrist and right wrist.

6. The method of claim 1, further comprising:

generating interpolated features based on the detected plurality of features.

7. The method of claim 1, further comprising:

responsive to detecting multiple candidates for one of the plurality of features in the image, identify a candidate as the one of the plurality of features based on a previously generated corresponding predicted feature.

8. The method of claim 1, wherein estimating the pose of the motion generator in the motion generator model further comprises:

tracking the estimated pose of the motion generator model with an observed pose of the motion generator.

9. The method of claim 1, further comprising:

generating predict features based on the augmented features and the kinematic constraints of the motion generator model.

10. The method of claim 1, wherein the kinematic constraints of the motion generator model comprises joint limitations and self penetration avoidance.

11. The method of claim 10, further comprises:

constructing a virtual surface surrounding an actual surface of a body segment of the motion generator model;

monitoring a distance between the body segment and an unconnected structure;

detecting that the unconnected structure penetrates the virtual surface;

determining a redirected joint motion that prevents the unconnected structure from colliding with the body segment; and

redirecting the body segment based on the redirected joint motion to avoid colliding with the unconnected structure.

12. A computer program product for estimating a pose of a motion generator, the computer program product comprising a computer-readable storage medium containing executable computer program code for performing a method comprising:

receiving an image of the motion generator;

detecting a plurality of features of the motion generator on the image;

augmenting missing features based on previously generated corresponding predicted features; and

estimating a pose of the motion generator in a motion generator model based on the augmented features and kinematic constraints of the motion generator model.

13. The computer program product of claim 12, wherein the motion generator comprises a human actor, and wherein detecting the plurality of features comprises:

detecting a head, neck, and torso (H-N-T) template in the image;

detecting arm blobs in the image based on the H-N-T template;

labeling left and right arms based on the detected arm blobs; and

localizing the plurality of feature points on the labeled arms.

14. The computer program product of claim 13, wherein the image comprises a depth image, and detecting arm blobs comprises at least one of the following:

performing a skeleton analysis to detect the arm blobs; and

performing a depth slicing to detect the arm blobs.

15. The computer program product of claim 13, wherein the plurality of features comprises:

waist, head center, left shoulder, right shoulder, left elbow, right elbow, left wrist and right wrist.

16. The computer program product of claim 12, wherein the method further comprises:

generating interpolated features based on the detected plurality of features.

17. A system for estimating a pose of a motion generator, the system comprising:

a computer processor for executing executable computer program code;

a computer-readable storage medium containing the executable computer program code for performing a method comprising;

receiving an image of the motion generator;

detecting a plurality of features of the motion generator on the image;

augmenting missing features based on previously generated corresponding predicted features; and

estimating a pose of the motion generator in a motion generator model based on the augmented features and kinematic constraints of the motion generator model.

18. The system of claim 17, wherein the motion generator comprises a human actor, and wherein detecting the plurality of features comprises:

detecting a head, neck, and torso (H-N-T) template in the image;

detecting arm blobs in the image based on the H-N-T template;

labeling left and right arms based on the detected arm blobs; and

localizing the plurality of feature points on the labeled arms.

19. The system of claim 18, wherein the image comprises a depth image, and detecting arm blobs comprises at least one of the following:

performing a skeleton analysis to detect the arm blobs; and

performing a depth slicing to detect the arm blobs.

20. The system of claim 18, wherein the plurality of features comprises:

waist, head center, left shoulder, right shoulder, left elbow, right elbow, left wrist and right wrist.