Facial Landmark Localization Using Coarse-to-Fine Cascaded Neural Networks

• US 20150347822A1
• Filed: 05/29/2014
• Published: 12/03/2015
• Est. Priority Date: 05/29/2014
• Status: Active Grant

• Current Assignee: Beijing Kuangshi Technology Co., Ltd.
• Original Assignee: Beijing Kuangshi Technology Co., Ltd.
• Inventors: Fan, Haoqiang, Cao, Zhimin, Jiang, Yuning, Yin, Qi, Zhou, Erjin
• Granted Patent: US 9,400,922 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

The present invention overcomes the limitations of the prior art by performing facial landmark localization in a coarse-to-fine manner with a cascade of neural network levels, and enforcing geometric constraints for each of the neural network levels. In one approach, the neural network levels may be implemented with deep convolutional neural network. One aspect concerns a system for localizing landmarks on face images. The system includes an input for receiving a face image, and an output for presenting landmarks identified by the system. Neural network levels are coupled in a cascade from the input to the output for the system. Each neural network level produces an estimate of landmarks. The estimate of landmarks is more refined than an estimate of landmark of a previous neural network level.

Claims

1. A system for localizing landmarks on face images, the system comprising:

an input for receiving a face image;

an output for presenting landmarks identified by the system; and

a plurality of neural network levels coupled in a cascade from the input to the output;

wherein each neural network level produces an estimate of landmarks that is more refined than an estimate of landmarks of a previous neural network level.

2. The system of claim 1 wherein, for each neural network level, inputs to the neural network level are a same size or larger than outputs for the neural network level.

3. The system of claim 2 wherein, for each neural network level, an input to the neural network level is a clip of the face image with a first bounding box, an output for the neural network level is a clip of the face image with a second bounding box, and the first bounding box is a same size or larger than the second bounding box.

4. The system of claim 3 wherein the second bounding box is a subset of the first bounding box.

5. The system of claim 1 wherein one of the neural network levels receives a face image as input and produces a cropped face image as output, the cropped face image estimating a location of a face within the face image.

6. The system of claim 1 wherein one of the neural network levels receives a face image as input and produces a landmarked face image as output, the landmarked face image containing an initial prediction of landmarks within the face image.

7. The system of claim 1 wherein one of the neural network levels receives a landmarked face image as input, the landmarked face image containing an initial prediction of landmarks within the face image, and produces a plurality of landmarked component images as output, each landmarked component image containing a refined estimate of an individual landmark.

8. The system of claim 1 wherein one of the neural network levels receives a landmarked component image as input and produces a rotated version of the landmarked component image as output.

9. The system of claim 1 wherein the landmarks include inner points defining landmarks within a face of the face image.

10. The system of claim 9 wherein the inner points include inner points defining eyes, mouth and nose.

11. The system of claim 1 wherein the landmarks include outer points defining a contour of a face of the face image.

12. The system of claim 1 wherein the number of neural network levels in cascade is different for different landmarks.

13. The system of claim 12 wherein the landmarks include inner points defining landmarks within a face of the face image and outer points defining a contour of the face, and a number of neural network levels in cascade for the inner points is greater than a number of neural network levels in cascade for the outer points.

14. The system of claim 1 wherein the neural network levels are convolutional neural networks that include convolution, non-linearity and down-sampling.

15. The system of claim 14 wherein each neural network level includes a convolutional neural network with at least two convolutional layers.

16. The system of claim 14 wherein the down-sampling is not more than 2×

down-sampling.

17. The system of claim 1 wherein the system contains between three to five neural network levels in cascade.

18. The system of claim 1 wherein the a plurality of neural network levels comprises:

at least three cascaded neural network levels for predicting inner points defining landmarks within a face of the face image, the three cascaded neural network levels including the following in order from input to output;

an first bounding box estimator that receives the face image as input and produces a first cropped face image as output, the first cropped face image estimating a location of the face within the face image for purposes of estimating inner points;

a first initial prediction module that receives the first cropped face image as input and produces a first landmarked face image as output, the first landmarked face image containing an initial prediction of inner points within the face image; and

for each of the landmarks to be predicted, a component refinement module that receives the first landmarked face image as input and produces a landmarked component image as output, the landmarked component image containing a refined estimate of inner points defining the landmark; and

two cascaded neural network levels for predicting outer points defining a contour of the face of the face image, the two cascaded neural network levels including the following in order from input to output;

a second bounding box estimator that receives the face image as input and produces a second cropped face image as output, the second cropped face image estimating a location of the face within the face image for purposes of estimating outer points; and

a second initial prediction module that receives the second cropped face image as input and produces a second landmarked face image as output, the second landmarked face image containing a prediction of outer points within the face image.

19. A method for localizing landmarks on face images, the method comprising:

receiving a face image;

producing an estimate of landmarks of a neural network that is more refined than an estimate of landmarks of a previous neural network level in a cascaded neural network; and

presenting landmarks identified based on the estimate of landmarks,

wherein the method is performed by one or more processors.

20. A non-transitory computer readable medium configured to store program code, the program code comprising instructions for localizing landmarks on face images, the instructions when executed by a processor cause the processor to:

receive a face image;

produce an estimate of landmarks of a neural network that is more refined than an estimate of landmarks of a previous neural network level in a cascaded neural network; and

present landmarks identified based on the estimate of landmarks.