Human Pose Estimation and Tracking Using Label Assignment

• US 20080152191A1
• Filed: 10/09/2007
• Published: 06/26/2008
• Est. Priority Date: 12/21/2006
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Fujimura, Kikuo, Zhu, Youding
• Granted Patent: US 8,351,646 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 40/107 Static hand or arm

Abstract

A method and apparatus for estimating poses of a subject by grouping data points generated by a depth image into groups representing labeled parts of the subject, and then fitting a model representing the subject to the data points using the grouping of the data points. The grouping of the data points is performed by grouping the data points to segments based on proximity of the data points, and then using constraint conditions to assign the segments to the labeled parts. The model is fitted to the data points by using the grouping of the data points to the labeled parts.

Claims

1. A method for estimating and tracking poses of a subject, the method comprising:

generating data points of the subject from a first image including the subject;

grouping the data points into groups of the data points representing labeled parts of the subject based on, at least, proximity of the data points;

fitting a model representing the subject to the data points using the groups to generate a pose vector representing the pose of the subject estimated from the data points; and

outputting the pose vector.

2. The method of claim 1, wherein the first image comprises a depth image of the subject captured using a depth image capturing device, the depth image representing a distance between the depth capturing device and parts of the subject.

3. The method of claim 2, wherein the step of fitting the model comprises:

determining vertices of a previous model visible to the depth image capturing device, the previous model generated from a second image preceding the first image;

obtaining the data points of the first image that correspond to the vertices of the model generated from the second image; and

estimating a pose vector of the subject in the first image based on the vertices of the model and the corresponding data points.

4. The method of claim 3, wherein the step of estimating the pose vector of the model comprises performing inverse kinematics.

5. The method of claim 3, wherein the step of estimating the pose vector of the model comprises:

generating a stacked matrix

$J=\left[\begin{array}{c}{J}_1\\ J_2\\ \mathrm{\cdots <br><br>}\\ J_m\end{array}\right]$

by stacking Jacobian matrix

$J_i=[<br><br>\begin{array}{cccccccc}1& 0& 0& 0& 0& 0& \mathrm{\cdots <br><br>}& 0\\ 0& 1& 0& \frac{\partial <br><br>p_i}{\partial <br><br>{\mathrm{\theta <br><br>}}_3}& \frac{\partial <br><br>p_i}{\partial <br><br>{\mathrm{\theta <br><br>}}_4}& \frac{\partial <br><br>p_i}{\partial <br><br>{\mathrm{\theta <br><br>}}_5}& \mathrm{\cdots <br><br>}& \frac{\partial <br><br>p_i}{\partial <br><br>{\mathrm{\theta <br><br>}}_n}\\ 0& 0& 1& 0& 0& 0& \mathrm{\cdots <br><br>}& 0\end{array}<br><br>],<br><br>\mathrm{where}<br><br>\left(p_i^x,p_i^y,p_i^z\right)$

represents a coordinate of an i^th data point, and θ

_i represents an i^th joint angle of the model,

generating a difference matrix

$\mathrm{\Delta <br><br>}<br><br><br><br>E=\left[\begin{array}{c}\mathrm{\Delta <br><br>}<br><br><br><br>e_1\\ \mathrm{\Delta <br><br>}<br><br><br><br>e_2\\ \mathrm{\cdots <br><br>}\\ \mathrm{\Delta <br><br>}<br><br><br><br>e_m\end{array}\right],\mathrm{where}<br><br><br><br>\mathrm{\Delta <br><br>}<br><br><br><br>e_i=\left[\begin{array}{c}{p}_i^x-v_i^x\\ p_i^y-v_i^y\\ p_i^z-v_i^z\end{array}\right]$

and (v_i^x, v_i^y, v_i^z) represents a coordinate of an i^th vertex of the model; and

reiteratively solving JΔ

q=Δ

P until Δ

q is below a threshold where q=(θ

₀,θ

₁, . . . θ

_n)^T represents a pose vector and P=(p₁,p₂, . . . ,p_m) represents the data points corresponding to the vertices.

6. The method of claim 1, wherein the step of grouping the data points comprises:

grouping the data points in proximity into segments; and

grouping the segments into the groups using constraint conditions.

7. The method of claim 6, wherein the constraint conditions comprises:

a fitted part constraint condition representing that first data points outside a shape fitted to a first group not be grouped to the first group;

a depth constraint condition representing that second data points having a depth profile be grouped to a second group; and

a color constraint condition representing that third data points be grouped to a third group responsive to pixels of a color image corresponding to the third data points having a predetermined color.

8. The method of claim 6, wherein the step of grouping the segments comprises:

modifying assignment of the segments to the groups; and

calculating a cost of assigning the segments to the groups based on at least distances between the segments and centers of the groups to which the segments are assigned; and

determining an assignment of the segments to the groups where the cost is minimized.

9. The method of claim 6, wherein the step of grouping the segments uses an optimization algorithm.

10. The method of claim 1, further comprising the step of initializing parameters of the model using a predetermined pose of the subject.

11. The method of claim 10, wherein the step of initializing the parameters of the model comprises:

detecting the predetermined pose of the subject;

detecting locations of joints of the subject from the predetermined pose of the subject; and

adjusting parameters of the model to fit the locations of the joints.

12. The method of claim 1, wherein a pose vector of the model fitted for the first image is used for grouping the data points in a next image subsequent to the first image and for fitting the model to the data points of the next image.

13. A computer program product comprising a computer readable medium structured to store instructions executable by a processor, the instructions, when executed cause the processor to:

generate data points of the subject from a first image including the subject;

group the data points into groups of the data points representing labeled parts of the subject based on, at least, proximity of the data points;

fit a model representing the subject to the data points using the groups to generate a pose vector representing a pose of the subject estimated from the data points; and

output the pose vector.

14. The computer program product of claim 13, wherein the first image comprises a depth image of the subject captured using a depth image capturing device, the depth image representing a distance between the depth capturing device and parts of the subject.

15. The computer program product of claim 14, wherein fitting the model comprises causing the processor to:

determine vertices of the model visible to the depth image capturing device, the model generated from a second image preceding the first image;

obtain the data points of the first image that correspond to the vertices of the model in the second image; and

estimate a pose vector of the subject in the first image based on the vertices of the model and the corresponding data points.

16. The computer program product of claim 15, wherein estimating the pose vector of the model comprises causing the processor to:

generate a stacked matrix

$J=\left[\begin{array}{c}{J}_1\\ J_2\\ \mathrm{\cdots <br><br>}\\ J_m\end{array}\right]$

by stacking Jacobian matrix

$J_i=[<br><br>\begin{array}{cccccccc}1& 0& 0& 0& 0& 0& \mathrm{\cdots <br><br>}& 0\\ 0& 1& 0& \frac{\partial <br><br>p_i}{\partial <br><br>{\mathrm{\theta <br><br>}}_3}& \frac{\partial <br><br>p_i}{\partial <br><br>{\mathrm{\theta <br><br>}}_4}& \frac{\partial <br><br>p_i}{\partial <br><br>{\mathrm{\theta <br><br>}}_5}& \mathrm{\cdots <br><br>}& \frac{\partial <br><br>p_i}{\partial <br><br>{\mathrm{\theta <br><br>}}_n}\\ 0& 0& 1& 0& 0& 0& \mathrm{\cdots <br><br>}& 0\end{array}<br><br>],<br><br>\mathrm{where}<br><br>\left(p_i^x,p_i^y,p_i^z\right)$

represents a coordinate of an i^th data point, and θ

_i represents an i^th joint angle of the model,

generate a difference matrix

$\mathrm{\Delta <br><br>}<br><br><br><br>E=\left[\begin{array}{c}\mathrm{\Delta <br><br>}<br><br><br><br>e_1\\ \mathrm{\Delta <br><br>}<br><br><br><br>e_2\\ \mathrm{\cdots <br><br>}\\ \mathrm{\Delta <br><br>}<br><br><br><br>e_m\end{array}\right],\mathrm{where}<br><br><br><br>\mathrm{\Delta <br><br>}<br><br><br><br>e_i=\left[\begin{array}{c}{p}_i^x-v_i^x\\ p_i^y-v_i^y\\ p_i^z-v_i^z\end{array}\right]$

and (v_i^x, v_i^y, v_i^z) represents a coordinate of an i^th vertex of the model; and

reiteratively solve JΔ

q=Δ

P until Δ

q is below a threshold where q=(θ

₀,θ

₁, . . . θ

_n)^T represents a pose vector and P=(p₁,p₂, . . . ,p_m) represents the data points corresponding to the vertices.

17. The computer program product of claim 13, wherein grouping the data points causes the processor to:

group the data points in proximity into segments; and

group the segments into the groups using constraint conditions.

18. The computer program product of claim 13, wherein the computer program product further causes the processor to initialize parameters of the model using a predetermined pose of the subject.

19. The computer program product of claim 13, wherein the computer program product further causes the processor to use a pose vector of the model generated from the first image to group the data points of a next image subsequent to the first image into the groups, and to fit a next model to the data points of the next image.

20. An apparatus for estimating and tracking poses of a subject, comprising:

an image capturing module for generating data points from an image of the subject;

a labeling module coupled to the image capturing module for grouping the data points into groups of the data points representing labeled parts of the subject based on, at least, proximity of the data points; and

a model fitting module coupled to the image capturing module and the labeling module, the model fitting module fitting a model representing the subject to the data points.

21. The apparatus of claim 20, wherein the image comprises a depth image representing distances between an imaging capturing device of the image capturing module and parts of the subject.

22. The apparatus of claim 20, the labeling module comprises:

a first grouping module for grouping the data points in proximity to segments; and

a second grouping module for grouping the segments into the groups using constraint conditions.

23. The apparatus of claim 22, wherein the second grouping module comprises:

a cost calculation module for calculating a cost of assigning the segments to the groups based at least on distances between the segments and centers of the groups to which the segments are assigned; and

a section adjustment module coupled to the cost assignment module for determining the assignment of segments where the cost is minimized.

24. The apparatus of claim 22, wherein the second grouping module determines the assignment using an optimization algorithm.

25. The apparatus of claim 20, wherein the model fitting module estimates a pose vector of the model representing the pose of the subject from the vertices of the model and the data points corresponding to the vertices of the model.