Combinatorial shape regression for face alignment in images

• US 10,528,839 B2
• Filed: 06/26/2015
• Issued: 01/07/2020
• Est. Priority Date: 06/26/2015
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Yao, Anbang, Chen, Yurong
• Primary Examiner(s): Mehta, Bhavesh M
• Assistant Examiner(s): Lemieux, Ian L
• Application Number:
  US15/573,631
• Publication Number:
  US 20180137383A1
• Time in Patent Office: 1,656 Days
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 10/7557 based on appearance, e.g. active appearance models [AAM]
• CPC Class Codes: G06V 40/164 using holistic features
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/169 Holistic features and representations, i.e. based on the facial image taken as a whole
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

Combinatorial shape regression is described as a technique for face alignment and facial landmark detection in images. As described stages of regression may be built for multiple ferns for a facial landmark detection system. In one example a regression is performed on a training set of images using face shapes, using facial component groups, and using individual face point pairs to learn shape increments for each respective image in the set of images. A fern is built based on this regression. Additional regressions are performed for building additional ferns. The ferns are then combined to build the facial landmark detection system.

Claims

1. A method of building stages of regression for multiple ferns for a facial landmark detection system, the method comprising:

performing a regression on a training set of images using face shapes in the images, facial component groups in the images, and individual face point pairs in the images, to learn shape increments for each respective image in the training set of images, wherein performing the regression using individual face point pairs comprises introducing geometric-invariant constraints while progressively learning shape increments;

building a fern based on the regression;

performing additional regressions on the training set of images using face shapes in the images, facial component groups, and individual face point pairs;

building additional ferns based on the additional regressions; and

combining the ferns to build the facial landmark detection system.

2. The method of claim 1, wherein performing the regression comprises learning variations in an appearance of facial landmarks.

3. The method of claim 1, wherein performing the regression comprises learning variations in a relationship of facial landmarks.

4. The method of claim 1, wherein performing the regression using face shapes in the images comprises:

calculating shape differences for each image of the set of images as a difference between a normalized ground truth face shape and each face shape of each image;

determining a set of candidate point pair indexed image features having the closest correlation to the ground truth face shape; and

using a determined top candidate set from the set of candidate point pair indexed images features to build ferns.

5. The method of claim 1, wherein performing the regression comprises first performing a regression using face shapes, then after performing the regression using face shapes, performing a regression using facial component groups, then after performing the regression using facial component groups, performing a regression using individual face point pairs.

6. The method of claim 5, wherein performing the regression using individual face point pairs comprises considering geometric relationships among landmark point-pairs.

7. The method of claim 1, wherein performing the regression using facial component groups comprises dividing facial landmarks into sets of semantic groups according to facial components, wherein each component is separately identified.

8. The method of claim 7, wherein performing the regression using facial component groups comprises learning over each semantic groups separately from each other semantic group.

9. The method of claim 1, wherein performing the regression using individual face point pairs comprises learning over each pair of face points separately from each other pair of face points.

10. The method of claim 9, wherein performing the regression using individual face point pairs comprises drawing a line between two face points of a pair and determining an index mark on the drawn line.

11. The method of claim 1, further comprising normalizing face shape sizes in the images of the training set of images.

12. A method of locating facial landmark positions in an image comprising:

receiving an initial image having an initial face shape;

applying a fern regression through multiple stages of multiple trained features, the fern regression comprising

applying a first plurality of regression-based ferns to update a face shape in the image;

after applying the first plurality of regression-based ferns, applying a second plurality of regression-based ferns to update groups of facial components in the face shape;

after applying the second plurality of regression-based ferns, applying a third plurality of regression-based ferns to update individual face point pairs in the face shape; and

after applying the third plurality of regression-based ferns, applying a fourth plurality of regression-based ferns to update fine features of the face shape; and

identifying facial landmark positions based on the applied fern regression.

13. The method of claim 12, wherein for each stage of the fern regression the positions in the image are incremented to new positions to provide a new face shape after each stage.

14. A computing system comprising:

a memory configured to store a set of training images;

an image sensor configured to receive an image containing a face; and

an imaging processor configured to receive the set of training images and to build stages of regression for multiple ferns for a facial landmark detection system by performing a regression on a training set of images using face shapes in the images, facial component groups in the images, and individual face point pairs in the images to learn shape increments for each respective image in the set of images, building a fern based on the regression, performing additional regressions on the training set of images using face shapes in the images, facial component groups, and individual face point pairs, building additional ferns based on the additional regressions, and combining the ferns to build the facial landmark detection system, wherein performing the regression using the individual face point pairs comprises introducing geometric-invariant constraints while progressively learning shape increments,

the imaging processor configured to store the ferns in a memory accessible to the facial landmark detection system.

15. The method of claim 14, wherein performing the regression using face shapes in the images comprises:

calculating shape differences for each image of the set of images as a difference between a normalized ground truth face shape and each face shape of each image;

determining a set of candidate point pair indexed image features having the closest correlation to the ground truth face shape; and

using a determined top candidate set from the set of candidate point pair indexed images features to build ferns.

16. The method of claim 14, wherein performing the regression comprises first performing a regression using face shapes, then after performing the regression using face shapes, performing a regression using facial component groups, then after performing the regression using facial component groups, performing a regression using individual face point pairs.

17. The method of claim 14, wherein performing the regression using individual face point pairs comprises learning over each pair of face points separately from each other pair of face points, drawing a line between two face points of a pair, and determining an index mark on the drawn line.