Method for estimating locations of facial landmarks in an image of a face using globally aligned regression

• US 9,633,250 B2
• Filed: 09/21/2015
• Issued: 04/25/2017
• Est. Priority Date: 09/21/2015
• Status: Active Grant

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Tambe, Salil, Tuzel, Oncel, Marks, Tim
• Primary Examiner(s): Dunphy, David F
• Application Number:
  US14/859,469
• Publication Number:
  US 20170083751A1
• Time in Patent Office: 582 Days
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 3/02 Affine transformations for image registration G06T3/147; for image mosaicing G06T3/4038
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/443 by matching or filtering
• CPC Class Codes: G06V 10/754 involving a deformation of the sample pattern or of the reference pattern; Elastic matching
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

A method for face alignment operates on a face image and a set of intitial landmark locations by first aligning globally the initial locations to a set of landmark locations of a face with a prototype shape to obtain global alignment parameters, and then warping the initial locations and the image from a coordinate frame of the image to a coordinate frame of the prototype shape according to the global alignment parameters to obtain warped landmark locations and a warped face image. Features are extracted from the warped face image at the warped landmark locations, and a regression function is applied to the features to obtain updated landmark locations in the coordinate frame of the prototype shape. Finally, the updated landmark locations in the coordinate frame of the prototype shape are warped to the coordinate frame of the image, to obtain updated landmark locations.

Claims

1. A method for face alignment, comprising steps:

acquiring an image of a face and a set of intitial landmark locations associated with the image;

aligning globally the set of initial landmark locations to a set of landmark locations of a face with a prototype shape to obtain global alignment parameters;

warping the set of initial landmark locations and the image from a coordinate frame of the image to a coordinate frame of the prototype shape according to the global alignment parameters, to obtain warped landmark locations and a warped face image;

extracting features from the warped face image at the warped landmark locations;

applying a regression function to the features to obtain updated landmark locations in the coordinate frame of the prototype shape; and

warping the updated landmark locations in the coordinate frame of the prototype shape to the coordinate frame of the image, to obtain updated landmark locations in the image, wherein the steps are performed in a processor.

2. The method of claim 1, further comprising:

applying the steps of claim 1 for K iterations, where K≧

1, wherein the image is used for all iterations, and wherein at each iteration k>

1, the set of initial landmark locations are the updated landmark locations in the image used during iteration k−

1.

3. The method of claim 2, wherein the expert regression function is different at each iteration k.

4. The method of claim 3, wherein the regression function is learned using a set of training data that include a set of N images of faces and ground truth locations of the facial landmarks in each of these N images,

wherein the training data are augmented by repeating every training image M times to produce a set of augmented training data,

wherein for each image I_i in the set of augmented training data, for i=1, . . . , MN, image I_i has ground truth landmark locations {circumflex over (x)}_i that are displaced by offset Δ

{circumflex over (x)}_i,

and wherein the regression function is learned by minimizing an L₂-loss function

5. The method of claim 4, wherein the offsets Δ

{circumflex over (x)}_i are randomly determined.

6. The method of claim 4,

wherein at iteration k=1 the offsets Δ

{circumflex over (x)}_i are randomly determined, and

wherein at each iteration k≧

2, the offsets Δ

{circumflex over (x)}_i are residuals after a previous iteration.

7. The method of claim 1, further comprising a set of L Globally Aligned Supervised Descent Method regression experts (experts),

wherein the image is used by all experts,

wherein each expert is associated with a different prototype shape and a different regression function,

wherein a gating function produces an assignment weight for each of the L experts,

wherein, for each of the L experts, the steps of claim 1 are performed, yielding L sets of updated facial landmark locations,

and wherein the L sets of updated facial landmark locations are combined as a weighted average, using the assignment weights, to obtain the updated facial landmark locations.

8. The method of claim 7, further comprising:

applying the method of claim 4 for K iterations, where K≧

1, wherein the image is used for all iterations, and wherein at each iteration k>

1, the initial landmark locations are the updated landmark locations obtained at iteration k−

1.

9. The method of claim 8, wherein for each expert the regression function is different at each iteration.

10. The method of claim 7, wherein the prototype shapes of the L experts are determined by solving an optimization problem to obtain an affine-invariant clustering of the locations of the landmark in a set of training data.

11. The method of claim 7, wherein the gating function α

¹ of each expert E^l, where l∈

{1, . . . , L}, is a softmax transformation of a global alignment error ε

^l(x_k) between landmark locations x_k and the landmark locations of the prototype shape x^l of expert E^l;

12. The method of claim 11, wherein the class of global transformations is a class of all possible 2D affine transformations.

13. The method of claim 7,

wherein the regression function of each expert is learned using a set of training data that includes a set of N training face images and ground truth locations of facial landmarks in each of the N training face images,

wherein the training data are augmented by repeating every training face image M times to produce a set of augmented training data,

wherein for each image I_i in the set of augmented training data, for i=1, . . . , MN, image I_i has ground truth landmark locations {circumflex over (x)}_i that are displaced by offset Δ

{circumflex over (x)}_i

wherein for each expert the regression function is learned by minimizing an L₂-loss function;

14. The method of claim 13, wherein at iteration k=1 the offsets Δ

{circumflex over (x)}_i are randomly determined, and wherein at each iteration k≧

2, the offsets Δ

{circumflex over (x)}_i are residuals after a previous iteration.

15. The method of claim 1, wherein the method is used for emotion analysis.

16. The method of claim 1, wherein the method is used for super-resolution imaging.

17. The method of claim 1, wherein the method is used by advanced driver assistance systems.

18. The method of claim 1, wherein the method is for video conferencing.

19. The method of claim 1, wherein the image is acquired by a camera.

20. The method of claim 1, wherein the initial landmark locations are determined using a face detection algorithm.

21. The method of claim 1, wherein the initial landmark locations are determined using a facial part detection algorithm.

22. The method of claim 1, wherein the global alignment parameters represent a 2D affine transformation.

23. The method of claim 1, wherein the prototype shape comprises a mean location of each landmark over a set of training data.

24. The method of claim 1, wherein the extracting uses scale invariant feature transform (SIFT) features.

25. The method of claim 1, wherein the extracting uses Histogram of Oriented Gradients (HOG) features.

26. The method of claim 1, wherein the features include additional features encoding deviations of the warped landmark locations from landmark locations of the prototype shape λ

(x′

_k−

x), wherein λ

is a weight that controls a strictness of the constraint on deviations of the locations from the prototype shape, x′

_k is a vector of the warped landmark locations, and x is a vector of the landmark locations of the prototype shape, and k indexes iteration of the method.

27. The method of claim 1, wherein the regression function is a linear function.

28. The method of claim 1, wherein the regression function is a regression forest.

29. The method of claim 1, wherein the regression function is learned using a set of training data, which includes a set of N images of faces and ground truth locations of the facial landmarks in each of the N images.