Human pose estimation and tracking using label assignment

• US 8,351,646 B2
• Filed: 10/09/2007
• Issued: 01/08/2013
• Est. Priority Date: 12/21/2006
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Fujimura, Kikuo, Zhu, Youding
• Primary Examiner(s): Mehta, Bhavesh
• Assistant Examiner(s): ANSARI, TAHMINA N
• Application Number:
  US11/869,435
• Publication Number:
  US 20080152191A1
• Time in Patent Office: 1,918 Days
• Field of Search: None
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 40/107 Static hand or arm

Abstract

A method and apparatus for estimating poses of a subject by grouping data points generated by a depth image into groups representing labeled parts of the subject, and then fitting a model representing the subject to the data points using the grouping of the data points. The grouping of the data points is performed by grouping the data points to segments based on proximity of the data points, and then using constraint conditions to assign the segments to the labeled parts. The model is fitted to the data points by using the grouping of the data points to the labeled parts.

Claims

1. A method for estimating and tracking poses of a subject, the method comprising:

generating data points of the subject from a first depth image including the subject by a depth camera, the first depth image representing distances from the depth camera to different parts of the subject;

grouping the data points into a plurality of segments of the data points based on, at least, depth profiles of the data points and spatial positions of the data points;

grouping the segments of the data points into labeled parts of the subject based on, at least, spatial relationships between the segments and constraint conditions by;

calculating a cost of assigning the segments to the labeled parts; and

assigning the segments to the labeled parts where the cost is minimized; and

fitting a model representing the subject to the data points using the groups to generate a pose vector representing the pose of the subject estimated from the data points.

2. The method of claim 1, wherein the step of fitting the model comprises:

determining vertices of a previous model visible to the depth image camera, the previous model generated from a second depth image preceding the first depth image;

obtaining the data points of the first depth image that correspond to the vertices of the model generated from the second depth image; and

estimating a pose vector of the subject in the first depth image based on the vertices of the model and the corresponding data points.

3. The method of claim 2, wherein the step of estimating the pose vector of the model comprises performing inverse kinematics.

4. The method of claim 2, wherein the step of estimating the pose vector of the model comprises:

generating a stacked matrix

5. The method of claim 1, wherein the constraint conditions comprises:

a fitted part constraint condition representing that first data points outside a shape fitted to a first group not be grouped to the first group;

a depth constraint condition representing that second data points having a depth profile be grouped to a second group; and

a color constraint condition representing that third data points be grouped to a third group responsive to pixels of a color image corresponding to the third data points having a predetermined color.

6. The method of claim 1, wherein the segements of the data points are grouped into labeled parts further by:

modifying assignment of the segments to the labeled parts.

7. The method of claim 1, wherein the grouping of the segments is performed using an optimization algorithm.

8. The method of claim 1, further comprising the step of initializing parameters of the model using a predetermined pose of the subject.

9. The method of claim 8, wherein the step of initializing the parameters of the model comprises:

detecting the predetermined pose of the subject;

detecting locations of joints of the subject from the predetermined pose of the subject; and

adjusting parameters of the model to fit the locations of the joints.

10. The method of claim 1, wherein the pose vector of the model fitted for the first depth image is used for grouping the segments of the data points into the labeled parts in a next depth image subsequent to the first depth image.

11. A non-transitory computer readable medium structured to store instructions executable by a processor, the instructions, when executed cause the processor to:

generate data points of the subject from a first depth image including the subject by a depth camera, the first depth image representing distances from the depth camera to different parts of the subject;

group the data points into a plurality of segments of the data points based on, at least, depth profiles of the data points and spatial positions of the data points;

group the segments of the data points into labeled parts of the subject based on, at least, spatial relationships between the segments and constraint conditions by instructions to;

calculate a cost of assigning the segments to the labeled parts; and

assign the segments to the labeled parts where the cost is minimized; and

fit a model representing the subject to the data points using the groups to generate a pose vector representing the pose of the subject estimated from the data points.

12. The computer readable medium of claim 11, wherein instructions for fitting the model comprises instructions causing the processor to:

determine vertices of the model visible to the depth image camera, the model generated from a second depth image preceding the first depth image;

obtain the data points of the first depth image that correspond to the vertices of the model in the second depth image; and

estimate a pose vector of the subject in the first depth image based on the vertices of the model and the corresponding data points.

13. The computer readable medium of claim 12, wherein instructions for estimating the pose vector of the model comprises instructions causing the processor to:

generate a stacked matrix

14. The computer readable medium of claim 11, further storing instructions causing the processor to initialize parameters of the model using a predetermined pose of the subject.

15. The computer readable medium of claim 11, further storing instructions causing the processor to use the pose vector of the model generated from the first depth image to group the segments of the data points into the labeled parts of a next depth image subsequent to the first depth image into the groups.

16. An apparatus for estimating and tracking poses of a subject, comprising:

a depth camera configured generate a first depth image by capturing data points of the subject, the first depth image representing distances from the depth camera to different parts of the subject;

a labeling module coupled to the depth camera, the labeling module configured to group the data points into a plurality of segments of the data points based on, at least, depth profiles of the data points and spatial positions of the data points, the labeling module further configured to group the segments of the data points into the labeled parts of the subject based on, at least, spatial relationships between the segments and constraint conditions, the labeling module comprising;

a cost calculation module configured to calculate a cost of assigning the segments to the labeled parts; and

a section adjustment module coupled to the cost assignment module and configured to assign the segments to the labeled parts where the cost is minimized; and

a model fitting module coupled to the depth camera and the labeling module, the model fitting module configured to fit a model representing the subject to the data points.

17. The apparatus of claim 16, wherein the labeling module is configured to determine the assignment using an optimization algorithm.

18. The apparatus of claim 16, wherein the model fitting module is configured to estimate the pose vector of the model representing the pose of the subject from the vertices of the model and the data points corresponding to the vertices of the model.