System and method for detecting and tracking facial features in images

• US 9,928,405 B2
• Filed: 05/27/2016
• Issued: 03/27/2018
• Est. Priority Date: 01/13/2014
• Status: Active Grant

• Current Assignee: Carnegie Mellon University
• Original Assignee: Carnegie Mellon University
• Inventors: De la Torre, Fernando, Xiong, Xuehan
• Primary Examiner(s): Yentrapati, Avinash
• Application Number:
  US15/166,317
• Publication Number:
  US 20160275339A1
• Time in Patent Office: 669 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

The present invention relates to a system for detecting and tracking facial features in images and can be used in conjunction with a camera. Given a camera, the system will detect facial landmarks in images. The present invention includes software for real time, accurate facial feature detection and tracking in unconstrained images and videos. The present invention is better, more robust and faster than existing approaches and can be implemented very efficiently allowing real-time processing, even on low-power devices, such as mobile phones.

Claims

1. A system for detecting and tracking features in a series of images of an object, comprising:

a camera;

a microprocessor, in communication with said camera

software, executed by said processor, said software performing the functions of detection and tracking,

wherein said function of detection includes the steps of;

(a) detecting the presence of said object in an image from said series of images and placing said object in a bounding box;

(b) initializing a set of landmarks in the bounding box; and

(c) applying multiple iterations of a regression function learned during a training phase to update said landmarks by mapping image features of said object to said landmarks; and

wherein said function of tracking includes the steps of;

(d) applying a classifier to determine how closely said features map to said landmarks in said object;

(e) repeating steps (c)-(d) for the next image in said series of images if said classifier determines that the mapping of features to landmarks exceeds a predetermined threshold; and

(f) repeating steps (a)-(d) for the next image in said series of images if said classifier determines that the mapping of features to landmarks does not exceed said predetermined threshold.

2. The system of claim 1 wherein said training phase includes

(a) training said classifier to classify landmarks as correctly or incorrectly placed;

and

(b) training a set of regressors to update said landmarks.

3. The system of claim 2 wherein said classifier is a function that maps image features to discrete classes.

4. The system of claim 3 wherein said discrete classes includes correctly predicted landmarks, and incorrectly predicted landmarks.

5. The system of claim 4 wherein said classifier is trained with images where the landmarks in the images are well place to learn correctly predicted landmarks and images where the landmarks are incorrectly placed to learn incorrectly-placed landmarks.

6. The system of claim 4 wherein said classifier indicates whether said landmarks are well-placed in said image or are not well-placed in said image.

7. The system of claim 1 wherein said set of regressors is trained using the Supervised Descent Method to update the landmarks.

8. The system of claim 2 wherein said features of said object are expressed as a vector of histograms of gradients.

9. The system of claim 7 wherein said regressors are based on a set of training data.

10. The system of claim 9 wherein said training data consists of a plurality of images having landmarks manually applied thereto.

11. The system of claim 1 wherein said object is a face.

12. The system of claim 1 wherein said function of detection further includes the initial steps of

(a) detecting said object within said image;

(b) normalizing said object to the scale and orientation of said images in said training data; and

(c) selecting one or more models which have been trained for different views of said object.

13. The system of claim 12 wherein each of said models has its own classifier and set of regressors.

14. The system of claim 1 wherein said step of initializing said set of landmarks is accomplished by placing an image of an average object in the center of said normalized object and transferring landmarks from said average object to said normalized object.

15. The system of claim 2 wherein said landmarks are manually placed on said objects in said images during said training phase.