Neural networks for facial modeling

• US 10,198,626 B2
• Filed: 10/19/2016
• Issued: 02/05/2019
• Est. Priority Date: 10/19/2016
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Li, Jia, Lv, Xutao, Wang, Xiaoyu, Xiong, Xuehan, Yang, Jianchao
• Primary Examiner(s): Osinski, Michael S
• Application Number:
  US15/297,789
• Publication Number:
  US 20180107866A1
• Time in Patent Office: 839 Days
• Field of Search: None
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20112 Image segmentation details
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/178 estimating age from face image; using age information for improving recognition

Abstract

Systems, devices, media, and methods are presented for modeling facial representations using image segmentation with a client device. The systems and methods receive an image depicting a face, detect at least a portion of the face within the image, and identify a set of facial features within the portion of the face. The systems and methods generate a descriptor function representing the set of facial features, fit object functions of the descriptor function, identify an identification probability for each facial feature, and assign an identification to each facial feature.

Claims

1. A method, comprising:

receiving, by one or more processors, one or more images depicting at least a portion of one or more faces;

detecting, by the one or more processors, the portion of the one or more faces depicted within the one or more images;

in response to detecting each portion of the one or more faces, identifying a set of facial features depicted on the portion of the one or more faces depicted within the one or more image;

based on the identified set of facial features, generating a descriptor function representing the set of facial features, the descriptor function including a set of object functions and each object function representing a facial feature of the set of facial features;

in response to generating the descriptor function, fitting each of the object functions of the set of object functions;

defining a probability boundary that includes minimum and maximum values corresponding to first and second identifying aspects of a face;

identifying a numerical probability for each facial feature of the set of facial features, each numerical probability identifying a probability that a facial feature corresponds to a specified characteristic of a set of feature characteristics, wherein a first numerical probability for a first facial feature is identified between the defined probability boundary such that a closer proximity of the first numerical probability to the minimum value of the probability boundary indicates that the first facial feature corresponds to the first identifying aspect of the face and a closer proximity of the first numerical probability to the maximum value of the probability boundary indicates that the first facial feature corresponds to the second identifying aspect of the face; and

assigning an identification to each facial feature based on the numerical probabilities identified for each facial feature.

2. The method of claim 1 further comprising:

accessing, by the one or more processors, a reference image from a facial reference database, the reference image having a set of reference facial landmarks; and

in response to identifying the set of facial features for each face of the one or more faces, mapping each face and the set of facial features of each face to the reference image.

3. The method of claim 2, wherein the reference image is a composite face representing a normalized face generated from a plurality of faces.

4. The method of claim 1, wherein fitting each of the object functions of the set of object functions for the descriptor function further comprises:

modifying one or more of the object functions within the descriptor function using stochastic gradient descent updating.

5. The method of claim 4, wherein the set of object functions comprises a plurality of object functions and modifying the one or more of the object functions further comprises:

selecting a first object function as a regularizing function; and

modifying one or more remaining object functions of the plurality of object functions with respect to the regularizing function.

6. The method of claim 1, wherein the numerical probability is an identification probability value bounded between a first value and a second value, the first value representing a male gender and the second value representing a female gender, and further comprising:

identifying a gender threshold value between the first value and the second value; and

selecting a gender for a face based on the identification probability value exceeding the gender threshold in a direction of one of the first value or the second value.

7. The method of claim 1, wherein a first specified object function of the set of objective functions is a convolution layer describing at least one facial feature of the set of facial features, and further comprising:

dividing the one or more images into a set of pixel regions;

performing the convolution layer for each pixel region of the set of pixel regions by executing the first specified object function on each pixel region; and

in response to performing the convolution layer on each pixel region, generating, for each pixel region, a value representing existence of a visual feature within the pixel region.

8. The method of claim 7, wherein the one or more images is represented as a two-dimensional matrix and further comprising:

generating a result matrix including the values representing the existence of the visual feature within the pixel region.

9. The method of claim 8, wherein a second specified object function of the set of object functions is a pooling layer configured to reduce the size of the result matrix, and further comprising:

identifying a maximum value of one or more sub-region of each pixel region;

selecting the maximum value for each sub-region as a representation of the pixel region in which the sub-region is included in the result matrix; and

generating a pooled matrix including a maximum value of each pixel region.

10. The method of claim 9, wherein a third specified object function of the set of object functions is a connected layer representing connections between the result matrix and the pooled matrix, and further comprising:

generating an inner product of the pooled matrix and a layer parameter; and

generating the numerical probability based on the inner product, the pooled matrix, and the result matrix.

11. A system, comprising:

one or more processors; and

a processor-readable storage device coupled to the one or more processors and storing processor-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising;

receiving, by the one or more processors, one or more images depicting at least a portion of one or more faces;

detecting, by the one or more processors, the portion of the one or more faces depicted within the one or more images;

in response to detecting each portion of the one or more faces, identifying a set of facial features depicted on the portion of the one or more faces depicted within the one or more images;

based on the identified set of facial features, generating a descriptor function representing the set of facial features, the descriptor function including a set of object functions and each object function representing a facial feature of the set of facial features;

in response to generating the descriptor function, fitting each of the object functions of the set of object functions;

defining a probability boundary that includes minimum and maximum values corresponding to first and second identifying aspects of a face;

identifying a numerical probability for each facial feature of the set of facial features, each numerical probability identifying a probability that a facial feature corresponds to a specified characteristic of a set of feature characteristics, wherein a first numerical probability for a first facial feature is identified between the defined probability boundary such that a closer proximity of the first numerical probability to the minimum value of the probability boundary indicates that the first facial feature corresponds to the first identifying aspect of the face and a closer proximity of the first numerical probability to the maximum value of the probability boundary indicates that the first facial feature corresponds to the second identifying aspect of the face; and

and

assigning an identification to each facial feature based on the numerical probabilities identified for each facial feature.

12. The system of claim 11, wherein fitting each of the object functions of the set of object functions for the descriptor function further comprises:

modifying one or more of the object functions within the descriptor function using stochastic gradient descent updating.

13. The system of claim 12, wherein the set of object functions comprises a plurality of object functions and modifying the one or more of the object functions further comprises:

selecting a first object function as a regularizing function; and

modifying one or more remaining object functions of the plurality of object functions with respect to the regularizing function.

14. The system of claim 11, a first specified object function of the set of object functions is a convolution layer describing at least one facial feature of the set of facial features, and the operations further comprising:

dividing the one or more images into a set of pixel regions;

performing the convolution layer for each pixel region of the set of pixel regions by executing the first specified object function on each pixel region;

in response to performing the convolution layer on each pixel region, generating, tier each pixel region, a value representing existence of a visual feature within the pixel region; and

generating a result matrix including the values representing the existence of the visual feature within the pixel region based on the one or more images being represented as a two-dimensional matrix.

15. The system of claim 14, wherein a second specified object function of the set of object functions is a pooling layer configured to reduce the size of the result matrix, and the operations further comprising:

identifying a maximum value of one or more sub-region of each pixel region;

selecting the maximum value for each sub-region as a representation of the pixel region in which the sub-region is included in the result matrix; and

generating a pooled matrix including a maximum value of each pixel region.

16. The system of claim 15, wherein a third specified object function of the set of object functions is a connected layer representing the connections between the result matrix and the pooled matrix, and further comprising:

generating an inner product of the pooled matrix and a layer parameter; and

generating the numerical probability based on the inner product, the pooled matrix, and the result matrix.

17. A processor-readable storage device storing processor-executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising:

receiving, by the one or more processors, one or more images depicting at least a portion of one or more faces;

detecting, by the one or more processors, the portion of the one or more faces depicted within the one or more images;

in response to detecting each portion of the one or more faces, identifying a set of facial features depicted on the portion of the face depicted within the one or more images;

based on the identified set of facial features, generating a descriptor function representing the set of facial features, the descriptor function including a set of object functions and each object function representing a facial feature of the set of facial features;

in response to generating the descriptor function, fitting each f the object functions of the set of object functions;

defining a probability boundary that includes minimum and maximum values corresponding to first and second identifying aspects of a face;

identifying a numerical probability for each facial feature of the set of facial features, each numerical probability identifying a probability that a facial feature corresponds to a specified characteristic of a set of feature characteristics, wherein a first numerical probability for a first facial feature is identified between the defined probability boundary such that a closer proximity of the first numerical probability to the minimum value of the probability boundary indicates that the first facial feature corresponds to the first identifying aspect of the face and a closer proximity of the first numerical probability to the maximum value of the probability boundary indicates that the first facial feature corresponds to the second identifying aspect of the face; and

and

assigning an identification to each facial feature based on the numerical probabilities identified for each facial feature.

18. The processor-readable storage device of claim 17, wherein a first specified object function of the set of object functions filters is a convolution layer describing at least one facial feature of the set of facial features, and the operations further comprise:

dividing the one or more images into a set of pixel regions;

performing the convolution layer for each pixel region of the set of pixel regions by executing the first specified object function on each pixel region;

in response to performing the convolution layer on each pixel region, generating, for each pixel region, a value representing existence of a visual feature within the pixel region; and

generating a result matrix including the values representing the existence of the visual feature within the pixel region based on the one or more images being represented as a two-dimensional matrix.

19. The processor-readable storage device of claim 18, wherein a second specified object function of the set of object functions is a pooling layer configured to reduce the size of the result matrix, and the operations further comprise:

identifying a maximum value of one or more sub-region of each pixel region;

selecting the maximum value for each sub-region as a representation of the pixel region in which the sub-region is included in the result matrix; and

generating a pooled matrix including a maximum value of each pixel region.

20. The processor-readable storage device of claim 19, wherein a third specified object function of the set of object functions is a connected layer representing the connections between the result matrix and the pooled matrix, and the operations further comprise:

generating an inner product of the pooled matrix and a layer parameter; and

generating the numerical probability based on the inner product, the pooled matrix, and the result matrix.