METHOD AND DEVICE FOR RECOGNIZING FACIAL EXPRESSIONS

• US 20190188458A1
• Filed: 12/20/2017
• Published: 06/20/2019
• Est. Priority Date: 12/15/2017
• Status: Active Grant

• Current Assignee: Industrial Technology Research Institute
• Original Assignee: Industrial Technology Research Institute
• Inventors: KANG, Hao-Ping, WU, Jia-Hua, YANG, Yao-Chin, KAO, Chih-Chung
• Granted Patent: US 10,445,564 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/761 Proximity, similarity or dissimilarity measures
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/175 Static expression

Abstract

A method for recognizing face expressions is provided. The method includes: recognizing expression categories of expressions in a plurality of face images and obtaining recognition results between each expression category and another expression category; obtaining similarities between each expression category and another expression category according to the recognition results; classifying the expression categories into a plurality of expression groups according to the similarities; training a first recognition model to classify the expressions in the face images into the expression groups; and training a second recognition model for each of the expression groups to classify the face images in each of the expression group into one of the expression categories.

Claims

1. A method for recognizing face expressions, comprising:

recognizing expression categories of expressions in a plurality of face images and obtaining recognition results between each expression category and another expression category;

obtaining similarities between each expression category and another expression category according to the recognition results;

classifying the expression categories into a plurality of expression groups according to the similarities;

training a first recognition model to classify the expressions in the face images into the expression groups; and

training a second recognition model for each of the expression groups to classify the face images in each of the expression groups into one of the expression categories.

2. The method for recognizing face expressions as claimed in claim 1, wherein the similarity between a first expression category and a second expression category is obtained according to a cross entropy between the first expression category and the second expression category.

3. The method for recognizing face expressions as claimed in claim 1, wherein the similarity is expressed as follows:

S(p,q)=H(p,q)+H(q,p)

wherein S(p, q) is the similarity between an expression category p and an expression category q, H(p, q) and H(q, p) are the cross entropies between the expression category p and the expression category q, and the cross entropy H(p, q) is expressed as follows;

4. The method for recognizing face expressions as claimed in claim 1, wherein the step of classifying the expression categories into the expression groups according to the similarities further comprises:

merging two of the expression categories with the highest similarity into an expression group;

obtaining the similarities between each unmerged expression category and the expression group; and

merging one of the unmerged expression categories and the expression group with the highest similarity or merging two of the unmerged expression categories with the highest similarity into another expression group.

5. The method for recognizing face expressions as claimed in claim 4, wherein the step of classifying the expression categories into the expression groups according to the similarities further comprises:

obtaining the similarity between the expression group and the other expression group; and

merging the expression group and the other expression group with the highest similarity.

6. The method for recognizing face expressions as claimed in claim 4, wherein the similarities between each of the unmerged expression categories and the expression group is obtained according to the cross entropies between each of the unmerged expression categories and the expression group.

7. The method for recognizing face expressions as claimed in claim 4, further comprising:

receiving an input image including a face; and

recognizing an expression category of an expression in the input image by using the first recognition model and the second recognition model.

8. The method for recognizing face expressions as claimed in claim 1, wherein the expression categories comprise anger, disgust, fear, happiness, sadness, and surprise.

9. The method for recognizing face expressions as claimed in claim 1, wherein the first recognition model and the second recognition model are based on a convolutional neural network (CNN) model.

10. A device for recognizing face expressions, comprising:

one or more processors; and

one or more computer storage media for storing one or more computer-readable instructions, wherein the processor is configured to drive the computer storage media to execute the following tasks;

recognizing expression categories of expressions in a plurality of face images and obtaining recognition results between each expression category and another expression category;

obtaining similarities between each expression category and another expression category according to the recognition results;

classifying the expression categories into a plurality of expression groups according to the similarities;

training a first recognition model to classify the expressions in the face images into the expression groups; and

training a second recognition model for each of the expression groups to classify the face images in each of the expression groups into one of the expression categories.

11. The device for recognizing face expressions as claimed in claim 10, wherein the similarity between a first expression category and a second expression category is obtained according to a cross entropy between the first expression category and the second expression category.

12. The device for recognizing face expressions as claimed in claim 10, wherein the similarity is expressed as follows:

S(p,q)=H(p,q)+H(q,p)

wherein S(p, q) is the similarity between an expression category p and an expression category q, H(p, q) and H(q, p) are the cross entropies between the expression category p and the expression category q, and the cross entropy H(p, q) is expressed as follows;

13. The device for recognizing face expressions as claimed in claim 10, wherein the processor classifying the expression categories into the expression groups according to the similarities further execute the following tasks:

merging two of the expression categories with the highest similarity into an expression group;

obtaining the similarities between each unmerged expression category and the expression group; and

merging one of the unmerged expression categories and the expression group with the highest similarity or merging two of the unmerged expression categories with the highest similarity into another expression group.

14. The device for recognizing face expressions as claimed in claim 13, wherein the processor classifying the expression categories into the expression groups according to the similarities further executes the following tasks:

obtaining the similarity between the expression group and the other expression group; and

merging the expression group and the other expression group with the highest similarity.

15. The device for recognizing face expressions as claimed in claim 13, wherein the similarities between each of the unmerged expression categories and the expression group is obtained according to the cross entropies between each of the unmerged expression categories and the expression group.

16. The device for recognizing face expressions as claimed in claim 13, the processor further execute the following tasks:

receiving an input image including a face; and

recognizing an expression category of an expression in the input image by using the first recognition model and the second recognition model.

17. The device for recognizing face expressions as claimed in claim 10, wherein the expression categories comprise anger, disgust, fear, happiness, sadness, and surprise.

18. The device for recognizing face expressions as claimed in claim 10, wherein the first recognition model and the second recognition model are based on a convolutional neural network (CNN) model.