System and Method for Detecting and Tracking Features in Images

• US 20080187174A1
• Filed: 12/12/2007
• Published: 08/07/2008
• Est. Priority Date: 12/12/2006
• Status: Active Grant

• Current Assignee: Rutgers University
• Original Assignee: Rutgers University
• Inventors: Metaxas, Dimitris, Kanaujia, Atul
• Granted Patent: US 8,121,347 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06F 18/2135 based on approximation criteria, e.g. principal component analysis
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 10/7553 based on shape, e.g. active shape models [ASM]
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 10/774 Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A system and method for tracking features, e.g., facial features, is provided, which allows for the tracking of features which move in a series of images and whose shape changes nonlinearly due to perspective projection and complex 3D movements. A training set of images is processed to produce clustered shape subspaces corresponding to the set of images, such that non-linear shape manifolds in the images are represented as piecewise, overlapping linear surfaces that are clustered according to similarities in perspectives. A landmark-based training algorithm (e.g., ASM) is applied to the clustered shape subspaces to train a model of the clustered shape subspaces and to create training data. A subsequent image is processed using the training data to identify features in the target image by creating an initial shape, superimposing the initial shape on the target image, and then iteratively deforming the shape in accordance with the model until a final shape is produced corresponding to a feature in the target image.

Claims

1. A method for detecting and tracking a feature in an image, comprising the steps of:

obtaining a set of training images clustered into clustered shape subspaces representative of at least one non-linear shape manifold in the training images;

receiving an image in which identification of a feature in the image is desired and movement of the feature has occurred with respect to a previous image;

creating an initial shape for identifying the feature in the image;

searching through the clustered shape subspaces to find a potential matching shape which potentially matches the feature;

deforming the initial shape into the potential matching shape; and

continuing searching and deformation until a final shape indicative of the feature is obtained.

2. The method of claim 1, further comprising the step of training a shape model using the clustered shape subspaces.

3. The method of claim 2, wherein the step of training the shape model further comprises aligning two-dimensional planar shapes in a global tangent space to cluster the clustered shape subspaces.

4. The method of claim 3, wherein the step of training the shape model further comprises aligning the two-dimensional planar shapes using Procrustes analysis to produce a plurality of aligned shapes.

5. The method of claim 4, wherein the step of training the shape model farther comprises clustering the plurality of aligned shapes into a plurality of clusters ensuing an Expectation-Maximization algorithm.

6. The method of claim 5, wherein the step of training the shape model further comprises generating linear subspaces from the two-dimensional planar shapes.

7. The method of claim 6, wherein the step of training the shape model further comprises learning locally linear point distribution models using Principal Component Analysis within each of the clustered shape subspaces.

8. The method of claim 7, wherein the step of training the shape model further comprises learning intensity profile models for each landmark for each of the clustered shape subspaces.

9. The method of claim 8, wherein the step of training the shape model further comprises deriving multivariate Gaussian mixture density distributions from the intensity profile models.

10. The method of claim 2, wherein the step of deforming the initial shape comprises deforming the initial shape using the shape model.

11. The method of claim 1, wherein the step of searching further comprises assigning an initial cluster to an average shape based upon a mahalanobis distance between the average shape and cluster centers in a global tangent space of the clustered shape subspaces.

12. The method of claim 11, wherein the step of searching further comprises searching through intensity profile models for an intensity profile that maximizes mixture density probability to create an intermediate shape.

13. The method of claim 11, wherein the step of deforming the initial shape further comprises constraining the deformation of the shape to match at least one of the clustered shape subspaces to produce an intermediate shape.

14. The method of claim 13, wherein the step of searching comprises assigning a new cluster shape space to the intermediate shape by finding a nearest cluster based on a mahalanobis distance.

15. The method of claim 14, wherein the step of searching comprises projecting the intermediate shape onto a local tangent space at a cluster mean shape.

16. The method of claim 1, further comprising tracking the feature across a plurality of subsequent images wherein the feature moves relative to frames of the subsequent images.

17. The method of claim 16, wherein the step of tracking the feature further comprises tracking the feature across the clustered shape subspaces in consecutive frames using a sum of squared intensity difference (SSID) tracker.

18. The method of claim 17, wherein the step of tracking the feature further comprises re-initializing points which have lost track by searching along a normal and maximizing profile mixture densities.

19. The method of claim 18, wherein the step of tracking the feature further comprises constraining a subsequent shape indicative of the feature to lie on a shape manifold within the subsequent images.

20. The method of claim 1, further comprising estimating a pose angle of a subject in the target image by learning Mixture of Experts models for the shapes and using the feature shapes to predict pitch and yaw of the head.

21. A method for detecting movement of a subject'"'"'s head in a series of images, comprising the steps of:

identifying a feature in a set of images corresponding to a subject'"'"'s nosetip;

determining coordinates of the feature for each frame of the set of images;

determining whether at least one value of the coordinates differs across the set of images; and

if at least one value differs, indicating that the subject'"'"'s head has moved.

22. The method of claim 21, further comprising indicating that a subject has nodded if at least one value of the coordinates differs across the set of images.

23. The method of claim 21, further comprising applying discriminative appearance models to the set of images to compensate for variations in illumination of the subject or glasses worn by the subject.

24. A method for detecting blinking of a subject in a series of images, comprising the steps of.

detecting a feature in a set of images corresponding to a subject'"'"'s eye;

comparing the feature to a template of a closed eye;

determining whether the detected feature matches template; and

if the detected feature matches the template, indicating that a subject blinked.

25. The method of claim 24, further comprising learning the template by analyzing a plurality of initial frames of a video of a subject.