Method for figure tracking using 2-D registration

• US 6,240,198 B1
• Filed: 04/13/1998
• Issued: 05/29/2001
• Est. Priority Date: 04/13/1998
• Status: Expired due to Fees

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Compaq Computer Corporation (HP Inc.)
• Inventors: Rehg, James Matthew, Morris, Daniel D.
• Primary Examiner(s): Bella, Matthew C.
• Assistant Examiner(s): CHOOBIN, BARRY
• Application Number:
  US09/059,478
• Time in Patent Office: 1,142 Days
• Field of Search: 345/473, 345/474, 382/103, 382/107, 395/152, 703/6
• US Class Current: 382/103
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

In a computerized method, a moving articulated figure is tracked in a sequence of 2-D images measured by a monocular camera. The images are individually registered with each other using a 2-D scaled prismatic model of the figure. The 2-D model includes a plurality of links connected by revolute joints to form is a branched, linear-chain of connected links. The registering produces a state trajectory for the figure in the sequence of images. During a reconstructing step, a 3-D model is fitted to the state trajectory to estimate kinematic parameters, and the estimated kinematic parameters are refined using an expectation maximization technique.

Claims

1. A computerized method for tracking a moving articulated figure in a sequence of 2-D images, comprising the steps of:

representing the figure in the sequence of images with a 2-D scaled prismatic model (SPM), the SPM comprising

a plurality of prismatic links connected by revolute joints having a connectivity identical to connectivity of the figure, each link having at least two degrees of freedom including, in the image plane, rotation in the image plane about an axis at a base joint of the link, and scaling along the link'"'"'s prismatic axis, and

a plurality of templates, a template associated with each link, each template including a plurality of image measurements representing a portion of the articulated moving figure, each template rotating and uniformly scaling responsive to rotation and scaling of the associated link;

registering, for each image, the 2-D scaled prismatic model responsive to image pixel values; and

producing from the registered models a state trajectory of the figure in the sequence of images, the state trajectory comprising a sequence of SPM state vectors, each SPM state vector comprising SPM state parameters that describe rotation and scaling of the SPM links for the figure in a corresponding image.

2. The method of claim 1 including capturing the sequence of images with a single monocular camera.

3. The method of claim 1 wherein the 2-D scaled prismatic model acts in a plane parallel to the image plane of eacb image, the 2-D scaled prismatic model describing all possible image projections of an equivalent 3-D kinematic model under orthographic projection onto the image plane.

4. The method of claim 1 wherein the figure is a human body, and the links represent the head, torso and limbs of the human body, and the joints represent the connectivity of the head, torso, and limbs.

5. The method of claim 1 wherein each template has a width w and a height h and an intrinsic coordinate system (u, v).

6. The method of claim 5 wherein transfornation of the pixels of tacit template is expressed as a matrix:

$T<br><br>\left(q\right)=\left[\begin{array}{ccc}\cos <br><br>\left(\mathrm{\theta <br><br>}\right)& -\sin <br><br>\left(\mathrm{\theta <br><br>}\right)& 0\\ \sin <br><br>\left(\mathrm{\theta <br><br>}\right)& \cos <br><br>\left(\mathrm{\theta <br><br>}\right)& 0\\ 0& 0& 1\end{array}\right]<br><br>\left[\begin{array}{ccc}1& 0& d\\ 0& 1& 0\\ 0& 0& 1\end{array}\right]=\left[\begin{array}{ccc}\cos <br><br>\left(\mathrm{\theta <br><br>}\right)& -\sin <br><br>\left(\mathrm{\theta <br><br>}\right)& \cos <br><br>\left(\mathrm{\theta <br><br>}\right)<br><br>d\\ \sin <br><br>\left(\mathrm{\theta <br><br>}\right)& \cos <br><br>\left(\mathrm{\theta <br><br>}\right)& \sin <br><br>\left(\mathrm{\theta <br><br>}\right)<br><br>d\\ 0& 0& 1\end{array}\right]$

where θ

is the aagle of rotation around the base joint of the associated link, and d is the translation along the prismmtic axis of the link.

7. The method of claim 6 including forward mapping the pixels of each template having intrinsic coordinates u and v to image plane coordinates x and y using:

$\left[\begin{array}{c}x\\ y\end{array}\right]=\left[\begin{array}{c}u\\ \left(v/h\right)<br><br>d\end{array}\right]=\left[\begin{array}{c}u\\ b<br><br>\text{\hspace{1em}<br><br>}<br><br>d\end{array}\right]$

where b=v/h gives a position of a particular pixel relative to the scaling of the associated link.

8. The method of claim 1 wherein the 3-D kinematic model is a branched, linear-chain of connected links.

9. The method of claim 8 wherein a singularity occurs if and only if the length of a particular joint lengths is zero.

10. The method of claim 9 wherein a direction of the singularity is always perpendicular to a direction in which the singularity was entered.

11. A computer program product for tracking a moving articulated figure in a sequence of 2-D images, the computer program product comprising usable medium having computer readable code thereon, including program code which:

represents the figure in the sequence of images with a 2-D scaled prismatic model (SPHD comprising

a plurality of prismatic links connected bv revolute joints having a connectivity identical to connectivity of the figure, each link having at least two degrees of freedom including, in the image plane, rotation in the image plane about an axis at a base joint of the link, and scaling along the link'"'"'s prismatic axis, and

a plurality of templates, a template associated with each link, each template including a plurality of image measurements representing a portion of the articulated moving figure, each template rotating and uniformly scaling responsive to rotation and scaling of the associated link;

projects, for each image, a pose of the figure onto an image plane, each pose represented by the 2-D scaled prismatic model, the projected poses producing a state trajectory of the figure in the sequence of images, the state trajectory comprising a sequence of SPM state vectors, each SPM state vector comprising SPM state parameters that describe rotation and scaling of the SPM links for the figure in a corresponding image.

12. A system for tracking a moving articulated figure in a sequence of 2-D images, comprising the steps of:

means for representing the figure in the sequence of images as a 2-D scaled prismatic model (SPM) comprising

a plurality of prismatic links connected by revolute joints having a connectivity identical to connectivity of the figure, each link having at least two degrees of freedom including, in the image plane, rotation in the image plane about an axis at a base joint of the link, and scaling along the link'"'"'s prismatic axis, and

a plurality of templates, a template associated with each link, each template including a plurality of image measurements representing a portion of the articulated moving figure, each template rotating and uniformly scaling responsive to rotation and scaling of the associated link;

means for projecting, for each image, a pose of the figure onto an image plane, each pose represented by the 2-D scaled prismatic model, the projected poses producing a state trajectory of the figure in the sequence of images, the state trajectory comprising a sequence of SPM state vectors, each SPM state vector comprising SPM state parameters that describe rotation and scaling of the SPM links for the figure in a corresponding image.