Neural networks for facial modeling

• US 10,395,100 B1
• Filed: 12/19/2018
• Issued: 08/27/2019
• Est. Priority Date: 10/19/2016
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Li, Jia, Lv, Xutao, Wang, Xiaoyu, Xiong, Xuehan, Yang, Jianchao
• Primary Examiner(s): Osinski, Michael S
• Application Number:
  US16/226,084
• Time in Patent Office: 251 Days
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20112 Image segmentation details
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/178 estimating age from face image; using age information for improving recognition

Abstract

Systems, devices, media, and methods are presented for modeling facial representations using image segmentation with a client device. The systems and methods receive an image depicting a face, detect at least a portion of the face within the image, and identify a set of facial features within the portion of the face. The systems and methods generate a descriptor function representing the set of facial features, fit object functions of the descriptor function, identify an identification probability for each facial feature, and assign an identification to each facial feature.

Claims

1. A method comprising:

identifying, by one or more processors, a feature of an object depicted within an image;

retrieving a probability boundary that includes first and second values corresponding to first and second identifying aspects of a given object;

computing a numerical probability for the feature of the object, depicted within the image, that indicates a probability that the feature corresponds to a specified identifying aspect of a set of identifying aspects of the given object;

determining whether the numerical probability of the feature of the object is closer in proximity to the first or second value of the probability boundary to indicate whether the feature corresponds to the first or second identifying aspect of the given object; and

assigning an identification to the feature of the object, depicted within the image, based on the determination of whether the numerical probability of the feature of the object is closer in proximity to the first or second value of the probability boundary.

2. The method of claim 1 further comprising:

accessing, by the one or more processors, a reference image from an object reference database, the reference image having a set of reference object landmarks; and

in response to identifying the feature, mapping the object and the feature of the object to the reference image.

3. The method of claim 1 further comprising:

generating a descriptor function representing feature of the object, the descriptor function including a set of object functions and each object function representing a different feature; and

in response to generating the descriptor function, fitting the object functions of the set of object functions.

4. The method of claim 3, wherein fitting the object functions comprises modifying one or more of the object functions within the descriptor function using stochastic gradient descent updating.

5. The method of claim 4, wherein the set of object functions comprises a plurality of object functions and modifying the one or more of the object functions further comprises:

selecting a first object function as a regularizing function; and

modifying one or more remaining object functions of the plurality of object functions with respect to the regularizing function.

6. The method of claim 1, wherein the first value represents a male gender and the second value represents a female gender, further comprising:

identifying a gender threshold value between the first value and the second value; and

selecting a gender based on the identification probability value exceeding the gender threshold in a direction of one of the first value or the second value.

7. The method of claim 1 further comprising:

providing a convolution layer describing at least one feature of a set of features;

dividing the image into a set of pixel regions;

performing the convolution layer for each pixel region of the set of pixel regions by executing a specified object function on each pixel region; and

in response to performing the convolution layer on each pixel region, generating, for each pixel region, a value representing existence of a visual feature within the pixel region.

8. The method of claim 7, wherein the image is represented as a two-dimensional matrix and further comprising:

generating a result matrix including the values representing the existence of the visual feature within the pixel region.

9. The method of claim 8, wherein the specified object function is a pooling layer configured to reduce the size of the result matrix, and further comprising:

identifying a maximum value of one or more sub-region of each pixel region;

selecting the maximum value for each sub-region as a representation of the pixel region in which the sub-region is included in the result matrix; and

generating a pooled matrix including a maximum value of each pixel region.

10. The method of claim 9, wherein another specified object is a connected layer representing connections between the result matrix and the pooled matrix, and further comprising:

generating an inner product of the pooled matrix and a layer parameter; and

generating the numerical probability based on the inner product, the pooled matrix, and the result matrix.

11. A system, comprising:

one or more processors; and

a processor-readable storage device coupled to the one or more processors and storing processor-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising;

identifying, by the one or more processors, a feature of an object depicted within an image;

retrieving a probability boundary that includes first and second values corresponding to first and second identifying aspects of a given object;

computing a numerical probability for the feature of the object, depicted within the image, that indicates a probability that the feature corresponds to a specified identifying aspect of a set of identifying aspects of the given object;

determining whether the numerical probability of the feature of the object is closer in proximity to the first or second value of the probability boundary to indicate whether the feature corresponds to the first or second identifying aspect of the given object; and

assigning an identification to the feature of the object, depicted within the image, based on the determination of whether the numerical probability of the feature of the object is closer in proximity to the first or second value of the probability boundary.

12. The system of claim 11, wherein the operations further comprise:

generating a descriptor function representing feature of the object, the descriptor function including a set of object functions and each object function representing a different feature; and

in response to generating the descriptor function, fitting the object functions of the set of object functions.

13. The system of claim 12, wherein:

fitting each of the object functions of the set of object functions for the descriptor function further comprises modifying one or more of the object functions within the descriptor function using stochastic gradient descent;

the operations further comprise operations for updating the set of object functions by;

selecting a first object function as a regularizing function; and

modifying one or more remaining object functions of the plurality of object functions with respect to the regularizing function.

14. The system of claim 11, wherein the operations further comprise:

providing a convolution layer describing at least one feature of a set of features;

dividing the image into a set of pixel regions;

performing the convolution layer for each pixel region of the set of pixel regions by executing a specified object function on each pixel region;

in response to performing the convolution layer on each pixel region, generating, for each pixel region, a value representing existence of a visual feature within the pixel region; and

generating a result matrix including the values representing the existence of the visual feature within the pixel region based on the image being represented as a two-dimensional matrix.

15. The system of claim 14, wherein the specified object function is a pooling layer configured to reduce the size of the result matrix, and the operations further comprising:

identifying a maximum value of one or more sub-region of each pixel region;

selecting the maximum value for each sub-region as a representation of the pixel region in which the sub-region is included in the result matrix; and

generating a pooled matrix including a maximum value of each pixel region.

16. The system of claim 15, wherein another specified object function is a connected layer representing the connections between the result matrix and the pooled matrix, and further comprising:

generating an inner product of the pooled matrix and a layer parameter; and

generating the numerical probability based on the inner product, the pooled matrix, and the result matrix.

17. A processor-readable storage device storing processor-executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising:

identifying, by one or more processors, a feature of an object depicted within an image;

retrieving a probability boundary that includes first and second values corresponding to first and second identifying aspects of a given object;

computing a numerical probability for the feature of the object, depicted within the image, that indicates a probability that the feature corresponds to a specified identifying aspect of a set of identifying aspects of the given object;

determining whether the numerical probability of the feature of the object is closer in proximity to the first or second value of the probability boundary to indicate whether the feature corresponds to the first or second identifying aspect of the given object; and

assigning an identification to the feature of the object, depicted within the image, based on the determination of whether the numerical probability of the feature of the object is closer in proximity to the first or second value of the probability boundary.

18. The processor-readable storage device of claim 17, wherein the operations further comprise:

providing a convolution layer describing at least one feature of a set of features;

dividing the image into a set of pixel regions;

performing the convolution layer for each pixel region of the set of pixel regions by executing a specified object function on each pixel region;

in response to performing the convolution layer on each pixel region, generating, for each pixel region, a value representing existence of a visual feature within the pixel region; and

generating a result matrix including the values representing the existence of the visual feature within the pixel region based on the one or more images being represented as a two-dimensional matrix.

19. The processor-readable storage device of claim 18, wherein the specified object function is a pooling layer configured to reduce the size of the result matrix, and the operations further comprise:

identifying a maximum value of one or more sub-region of each pixel region;

selecting the maximum value for each sub-region as a representation of the pixel region in which the sub-region is included in the result matrix; and

generating a pooled matrix including a maximum value of each pixel region.

20. The processor-readable storage device of claim 19, wherein another specified object function is a connected layer representing the connections between the result matrix and the pooled matrix, and the operations further comprise:

generating an inner product of the pooled matrix and a layer parameter; and

generating the numerical probability based on the inner product, the pooled matrix, and the result matrix.