SYSTEMS AND METHODS FOR FACIAL REPRESENTATION

• US 20150125049A1
• Filed: 10/31/2014
• Published: 05/07/2015
• Est. Priority Date: 11/04/2013
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Taigman, Yaniv, Yang, Ming, Ranzato, Marc'Aurelio
• Granted Patent: US 10,095,917 B2
• Time in Patent Office: Days
• US Class Current: 382/118
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2210/44 Morphing
• CPC Class Codes: G06T 3/02 Affine transformations for image registration G06T3/147; for image mosaicing G06T3/4038
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/173 face re-identification, e.g. recognising unknown faces across different face tracks

Abstract

Systems, methods, and non-transitory computer readable media can align face images, classify face images, and verify face images by employing a deep neural network (DNN). A 3D-aligned face image can be generated from a 2D face image. An identity of the 2D face image can be classified based on provision of the 3D-aligned face image to the DNN. The identity of the 2D face image can comprise a feature vector.

Claims

1. A computer-implemented method, comprising:

generating, by a computing system, a 3D-aligned face image from a 2D face image; and

classifying, by the computing system, an identity of the 2D face image based on provision of the 3D-aligned face image to a deep neural network (DNN), the identity of the 2D face image comprising a feature vector.

2. The computer-implemented method of claim 1, wherein the generating the 3D-aligned face image from the 2D face image comprises:

detecting a set of fiducial points of the 2D face image; and

warping the 2D face image to a 3D shape using the set of fiducial points.

3. The computer-implemented method of claim 2, wherein the warping the 2D face image to the 3D shape comprises:

placing a set of anchor points onto the 3D shape, each anchor point corresponding to a fiducial point of the set of fiducial points of the 2D face image;

back-projecting a location of each fiducial point of the set of fiducial points of the 2D face image to the 3D shape;

projecting the location of each fiducial point of the set of fiducial points of the 2D face image to a frontal view of the 3D shape; and

transforming the 2D face image to the 3D-aligned face image based on the set of fiducial points and the set of anchor points.

4. The computer-implemented method of claim 3, wherein the set of fiducial points of the 2D face image define a set of triangles, and the transforming the 2D face image to the 3D-aligned face image comprises performing affine transformation of each triangle of the set of triangles.

5. The computer-implemented method of claim 1, further comprising:

identifying a face portion of an image by detecting a second set of fiducial points of the image;

generating the 2D face image by transforming the face portion of the image based on the second set of fiducial points.

6. The computer-implemented method of claim 5, wherein the generating the 2D face image comprises determining a set of anchor locations and warping the face portion of the image based on the set of anchor locations.

7. The computer-implemented method of claim 1, wherein the DNN comprises a set of layers comprising a convolutional-pooling layer, a set of locally-connected layers, and a set of fully-connected layers.

8. The computer-implemented method of claim 7, wherein the convolutional-pooling layer comprises a first convolutional layer, a max-pooling layer, and a second convolutional layer, and the convolutional-pooling layer is configured to extract a set of features of the 3D-aligned face image.

9. The computer-implemented method of claim 7, wherein each locally-connected layer of the set of locally-connected layers is configured to extract a set of features of the 3D-aligned face image.

10. The computer-implemented method of claim 7, wherein each fully-connected layer of the set of fully-connected layers is configured to determine correlations between a set of features of the 3D-aligned face image.

11. The computer-implemented method of claim 7, wherein an output of a fully-connected layer of the set of fully-connected layers is the feature vector.

12. The computer-implemented method of claim 7, wherein the DNN is configured to classify the 2D face image based on the feature vector, the feature vector comprising a set of features of the 3D-aligned face image, each feature of the set of features normalized to a predetermined range.

13. The computer-implemented method of claim 1, wherein the DNN is configured to define a set of filters for each pixel of the 3D-aligned face image.

14. The computer-implemented method of claim 13, further comprising training the DNN to define the set of filters with a dataset.

15. The computer-implemented method of claim 1, further comprising identifying an individual of the 2D face image by querying the identity of the 2D face image in an image store, the image store comprising a set of face images, each face image of the set of face images corresponding to an individual.

16. The computer-implemented method of claim 1, further comprising:

classifying a second identity of a second 2D face image; and

comparing the identity to the second identity to determine whether the identity and the second identity correspond to an individual.

17. The computer-implemented method of claim 16, wherein the comparing the identity to the second identity comprises determining a weighted-χ

² distance between the identity and the second identity.

18. The computer-implemented method of claim 16, wherein the identity comprises a first set of features and the second identity comprises a second set of features, and the comparing the identity to the second identity comprises:

determining a set of feature differences between the first set of features and the second set of features; and

mapping the set of feature differences to a predetermined value by using a fully-connected layer.

19. A system comprising:

at least one processor; and

a memory storing instructions that, when executed by the at least one processor, cause the system to perform;

generating a 3D-aligned face image from a 2D face image; and

classifying an identity of the 2D face image based on the provision of the 3D-aligned face image to a deep neural network (DNN), the identity of the 2D face image comprising a feature vector.

20. A non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing system, cause the computing system to perform:

generating a 3D-aligned face image from a 2D face image; and

classifying an identity of the 2D face image based on the provision of the 3D-aligned face image to a deep neural network (DNN), the identity of the 2D face image comprising a feature vector.