Hybrid deep learning method for recognizing facial expressions

• US 10,679,041 B2
• Filed: 06/20/2018
• Issued: 06/09/2020
• Est. Priority Date: 04/25/2018
• Status: Active Grant

• Current Assignee: Shutterfly, LLC
• Original Assignee: Shutterfly, LLC
• Inventors: Cyrus, Leo
• Primary Examiner(s): Dunphy, David F
• Application Number:
  US16/012,989
• Publication Number:
  US 20190332854A1
• Time in Patent Office: 720 Days
• CPC Class Codes: G06F 18/213 Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
• CPC Class Codes: G06F 18/254 of classification results, e.g. of results related to same input data
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/809 of classification results, e.g. where the classifiers operate on the same input data
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/175 Static expression

Abstract

A computer implemented method for recognizing facial expressions by applying feature learning and feature engineering to face images. The method includes conducting feature learning on a face image comprising feeding the face image into a first convolution neural network to obtain a first decision, conducting feature engineering on a face image, comprising the steps of automatically detecting facial landmarks in the face image, transforming the facial features into a two-dimensional matrix, and feeding the two-dimensional matrix into a second convolution neural network to obtain a second decision, computing a hybrid decision based on the first decision and the second decision, and recognizing a facial expression in the face image in accordance to the hybrid decision.

Claims

1. A computer-implemented method for recognizing facial expressions by applying a hybrid of feature learning and feature engineering to face images, comprising:

conducting feature learning on a face image by one or more computer processors, comprising;

feeding the face image into a first convolution neural network to obtain a first decision;

conducting feature engineering on a face image by the one or more computer processors, comprising;

automatically detecting facial landmarks in the face image;

describing each of the facial landmarks by a set of facial features;

transforming the facial features into a two-dimensional matrix comprising coordinates of the facial features; and

feeding the two-dimensional matrix into a second convolution neural network to obtain a second decision;

computing a hybrid decision based on the first decision and the second decision; and

recognizing a facial expression in the face image in accordance to the hybrid decision.

2. The computer-implemented method of claim 1, further comprising:

multiplying the first decision by a first weight to produce a first weighted decision; and

multiplying the second decision by a second weight to produce a second weighted decision,

wherein the hybrid decision is computed based on the first weighted decision and the second weighted decision.

3. The computer-implemented method of claim 2, wherein the hybrid decision is an average, a sum, or a root-mean square function of the first weighted decision and the second weighted decision.

4. The computer-implemented method of claim 2, updating the first weight and the second weight in the hybrid decision by backpropagation.

5. The computer-implemented method of claim 1, wherein the two-dimensional matrix comprises multiple pairs of coordinates each associated with one of the facial features.

6. The computer-implemented method of claim 1, further comprising:

automatically detecting a face in a digital image by the one or more computer processors;

extracting a face portion surrounding the face from the digital image by the one or more computer processors; and

normalizing the face portion to obtain the face image.

7. The computer-implemented method of claim 1, wherein the facial landmarks respectively describe at least a portion of an eye, an eye brow, a mouth, a chin, an edge of a face, or a nose in the face image.

8. The computer-implemented method of claim 1, wherein the facial landmarks include a plurality of groups each of which describes at least a portion of a facial feature.

9. The computer-implemented method of claim 8, wherein the facial landmarks include an eye, an eye brow, a mouth, a chin, an edge of a face, or a nose in the face image.

10. The computer-implemented method of claim 1, wherein each of the facial features is described by a pair of coordinates.

11. The computer-implemented method of claim 1, wherein the step of transforming the facial features into a two-dimensional matrix comprises:

expressing each of the facial features by a pair of coordinates; and

forming a first matrix using the facial features and their respective coordinates.

12. The computer-implemented method of claim 11, further comprising:

concatenating copied of the first matrix one or more times to produce the two-dimensional matrix.

13. The computer-implemented method of claim 1, wherein the two-dimensional matrix is a square matrix.