Face alignment with shape regression

• US 10,019,622 B2
• Filed: 10/23/2014
• Issued: 07/10/2018
• Est. Priority Date: 08/22/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Cao, Xudong, Wei, Yichen, Sun, Jian, Ren, Shaoqing
• Primary Examiner(s): Kholdebarin, Iman K
• Application Number:
  US14/522,366
• Publication Number:
  US 20160055368A1
• Time in Patent Office: 1,356 Days
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06F 18/2431 Multiple classes
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 10/754 involving a deformation of the sample pattern or of the reference pattern; Elastic matching
• CPC Class Codes: G06V 10/7553 based on shape, e.g. active shape models [ASM]
• CPC Class Codes: G06V 10/757 Matching configurations of points or features
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

The subject matter described herein relates to face alignment via shape regression. A method, computer storage medium, and system are provided. In one embodiment, the method comprises receiving an image including a face; and performing shape regression to estimate a facial shape in the image. For each stage in the shape regression, a local feature is extracted from a local region around each facial landmark in the image independently; and a joint projection is performed based on local features of multiple facial landmarks to predict a facial shape increment. Then, a facial shape of a current stage is generated based on the predicted facial shape increment and a facial shape of a previous stage.

Claims

1. A method for face alignment implemented at least in part by a computer comprising:

receiving an image including a face; and

estimating, by a processor, a facial shape in the image with shape regression, wherein the facial shape is represented by a multiple of facial landmarks, the estimating including, for each stage in the shape regression;

extracting a local feature from a local region around each facial landmark in the image independently, the extracting comprising mapping a local region around a facial landmark into a local feature according to a local feature mapping function associated with the facial landmark, the local feature mapping function encoding the local region into a local binary feature;

organizing two or more of the local features based on correlations among the facial landmarks to predict a facial shape increment;

determining whether the predicted facial shape increment is less than a threshold; and

based on a determination that the predicted facial shape increment is greater than the threshold;

extracting a set of second local features around each facial landmark in the image independently, the extracting comprising mapping a second local region around a second facial landmark into a second local feature according to the local feature mapping function associated with the second facial landmark, the local feature mapping function encoding the second local region into the local binary feature;

organizing two or more of the second local features based on correlations among the second facial landmarks to predict a second facial shape increment; and

based on a determination that the second predicted facial shape increment is less than the threshold, generating a final facial shape.

2. The method of claim 1, wherein organizing two or more of the local features comprises:

concatenating the local features of the multiple facial landmarks into a global feature; and

performing a global projection on the global feature according to a global projection matrix.

3. The method of claim 1, further comprising training each local feature mapping function by regressing each associated facial landmark independently in a local region of the facial landmark.

4. The method of claim 2, further comprising training the global projection matrix by a linear regression based on trained local feature mapping functions on a whole region of the face.

5. The method of claim 3, wherein a random forest is used as the local feature mapping function.

6. The method of claim 1, wherein a size of a local region of a facial landmark decreases as the stage of the shape regression increases.

18. The method of claim 3, wherein a target of the regression is a distance between a truth offset of a facial landmark and an estimated offset of the facial landmark.

7. A computer storage device having computer-executable instructions, which when executed perform actions for face alignment comprising:

receiving an image including a face; and

estimating a facial shape in the image with shape regression, wherein the facial shape is represented by a multiple of facial landmarks, the estimating including, for each stage in the shape regression;

extracting a local feature from a local region around each facial landmark in the image independently, the extracting comprising mapping a local region around a facial landmark into a local feature according to a local feature mapping function associated with the facial landmark, the local feature mapping function encoding the local region into a local binary feature;

organizing two or more of the local features based on correlations among the facial landmarks to predict a facial shape increment;

determining whether the predicted facial shape increment is less than a threshold; and

based on a determination that the predicted facial shape increment is greater than the threshold;

extracting a set of second local features around each facial landmark in the image independently, the extracting comprising mapping a second local region around a second facial landmark into a second local feature according to the local feature mapping function associated with the second facial landmark, the local feature mapping function encoding the second local region into the local binary feature;

organizing two or more of the second local features based on correlations among the second facial landmarks to predict a second facial shape increment; and

based on a determination that the second predicted facial shape increment is less than the threshold, generating a final facial shape.

8. The computer storage device of claim 7, wherein organizing two or more of the local features comprises:

concatenating the local features of the multiple facial landmarks into a global feature; and

performing a global projection on the global feature according to a global projection matrix.

9. The computer storage device of claim 7, the actions further comprising training each local feature mapping function by regressing each associated facial landmark independently in a local region of the facial landmark.

10. The computer storage device of claim 8, the actions further comprising training the global projection matrix by a linear regression based on trained local feature mapping functions on a whole region of the face.

11. The computer storage device of claim 9, wherein a random forest is used as the local feature mapping function.

12. The computer storage device of claim 7, wherein a size of a local region of a facial landmark decreases as the stage of the shape regression increases.

19. The computer storage device of claim 9, wherein a target of the regression is a distance between a truth offset of a facial landmark and an estimated offset of the facial landmark.

13. In a computing environment, a system for face alignment comprising:

a processor;

a shape regression mechanism, including a local mapping mechanism configured to, for each stage of the shape regression;

extract a local feature from a local region around each facial landmark in a facial image independently; and

map a local region around a facial landmark into a local feature according to a local feature mapping function associated with the facial landmark where the local feature mapping function encodes the local region into a local binary feature; and

a joint projection mechanism configured to, for each stage of the shape regression, organize two or more of the local features based on correlations among the facial landmarks to predict a facial shape increment; and

an output mechanism configured to cause the processor to, based on a determination that the predicted facial shape increment is greater than the threshold;

extract a set of second local features around each facial landmark in the facial image independently, the extracting comprising mapping a second local region around a second facial landmark into a second local feature according to the local feature mapping function associated with the second facial landmark, the local feature mapping function encoding the second local region into the local binary feature;

organize two or more of the second local features based on correlations among the second facial landmarks to predict a second facial shape increment; and

based on a determination that the second predicted facial shape increment is less than the threshold, generate a final facial shape.

14. The system of claim 13, wherein the joint projection mechanism is configured to organize the local features by:

concatenating the local features of the multiple facial landmarks into a global feature; and

performing a global projection on the global feature according to a global projection matrix.

15. The system of claim 13, further comprising a training mechanism configured to train each local feature mapping function by regressing each associated facial landmark independently in a local region of the facial landmark.

16. The system of claim 15, the training mechanism is further configured to train a global projection matrix by a linear regression based on the trained local feature mapping functions on a whole region of the face.

17. The system of claim 13, wherein a size of a local region of a facial landmark decreases as the stage of the shape regression increases.

20. The system of claim 13, wherein each local feature mapping function is trained by regressing each associated facial landmark independently in a local region of the facial landmark, wherein a target of the regression is a distance between a truth offset of a facial landmark and an estimated offset of the facial landmark.