FACE ALIGNMENT WITH SHAPE REGRESSION

• US 20160055368A1
• Filed: 10/23/2014
• Published: 02/25/2016
• Est. Priority Date: 08/22/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Cao, Xudong, Wei, Yichen, Sun, Jian, Ren, Shaoqing
• Granted Patent: US 10,019,622 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06F 18/2431 Multiple classes
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 10/754 involving a deformation of the sample pattern or of the reference pattern; Elastic matching
• CPC Class Codes: G06V 10/7553 based on shape, e.g. active shape models [ASM]
• CPC Class Codes: G06V 10/757 Matching configurations of points or features
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

The subject matter described herein relates to face alignment via shape regression. A method, computer storage medium, and system are provided. In one embodiment, the method comprises receiving an image including a face; and performing shape regression to estimate a facial shape in the image. For each stage in the shape regression, a local feature is extracted from a local region around each facial landmark in the image independently; and a joint projection is performed based on local features of multiple facial landmarks to predict a facial shape increment. Then, a facial shape of a current stage is generated based on the predicted facial shape increment and a facial shape of a previous stage.

Claims

1. A method for face alignment implemented at least in part by a computer comprising:

receiving an image including a face; and

estimating, by a processor, a facial shape in the image with shape regression, wherein the facial shape is represented by a multiple of facial landmarks, the estimating including, for each stage in the shape regression,

extracting a local feature from a local region around each facial landmark in the image independently;

organizing two or more of the local features based on correlations among the facial landmarks to predict a facial shape increment; and

generating a facial shape of a current stage based on the predicted facial shape increment and a facial shape of a previous stage.

2. The method of claim 1, wherein extracting a local feature from a local region comprises:

mapping a local region around a facial landmark into a local feature according to a local feature mapping function associated with the facial landmark.

3. The method of claim 1, wherein organizing two or more of the local features comprises:

concatenating the local features of the multiple facial landmarks into a global feature; and

performing a global projection on the global feature according to a global projection matrix.

4. The method of claim 2, further comprising training each local feature mapping function by regressing each associated facial landmark independently in a local region of the facial landmark.

5. The method of claim 3, further comprising training the global projection matrix by a linear regression based on trained local feature mapping functions on a whole region of the face.

6. The method of claim 4, wherein a random forest is used as the local feature mapping function.

7. The method of claim 1, wherein a size of a local region of a facial landmark decreases as the stage of the shape regression increases.

8. A computer storage medium having computer-executable instructions, which when executed perform actions for face alignment comprising:

receiving an image including a face; and

estimating a facial shape in the image with shape regression, wherein the facial shape is represented by a multiple of facial landmarks, the estimating including, for each stage in the shape regression,

extracting a local feature from a local region around each facial landmark in the image independently;

organizing two or more of the local features based on correlations among the facial landmarks to predict a facial shape increment; and

generating a facial shape of a current stage based on the predicted facial shape increment and a facial shape of a previous stage.

9. The computer storage medium of claim 8, wherein extracting a local feature from a local region comprises:

mapping a local region around a facial landmark into a local feature according to a local feature mapping function associated with the facial landmark.

10. The computer storage medium of claim 8, wherein organizing two or more of the local features comprises:

concatenating the local features of the multiple facial landmarks into a global feature; and

performing a global projection on the global feature according to a global projection matrix.

11. The computer storage medium of claim 9, the actions further comprising training each local feature mapping function by regressing each associated facial landmark independently in a local region of the facial landmark.

12. The computer storage medium of claim 10, the actions further comprising training the global projection matrix by a linear regression based on trained local feature mapping functions on a whole region of the face.

13. The computer storage medium of claim 11, wherein a random forest is used as the local feature mapping function.

14. The computer storage medium of claim 8, wherein a size of a local region of a facial landmark decreases as the stage of the shape regression increases.

15. In a computing environment, a system for face alignment comprising:

a shape regression mechanism, including a local mapping mechanism configured to, for each stage of the shape regression, extract a local feature from a local region around each facial landmark in a facial image independently, and a joint projection mechanism configured to, for each stage of the shape regression, organize two or more of the local features based on correlations among the facial landmarks to predict a facial shape increment, and an output mechanism configured to generate a facial shape of a current stage based on the predicted facial shape increment and a facial shape of a previous stage.

16. The system of claim 15, wherein the local mapping mechanism is configured to map a local region around a facial landmark into a local feature according to a local feature mapping function associated with the facial landmark.

17. The system of claim 15, wherein the joint projection mechanism is configured to organize the local features by:

concatenating the local features of the multiple facial landmarks into a global feature; and

performing a global projection on the global feature according to a global projection matrix.

18. The system of claim 15, further comprising a training mechanism configured to train each local feature mapping function by regressing each associated facial landmark independently in a local region of the facial landmark.

19. The system of claim 18, the training mechanism is further configured to train a global projection matrix by a linear regression based on the trained local feature mapping functions on a whole region of the face.

20. The system of claim 15, wherein a size of a local region of a facial landmark decreases as the stage of the shape regression increases.