Emotion classification based on expression variations associated with same or similar emotions

• US 10,489,690 B2
• Filed: 10/24/2017
• Issued: 11/26/2019
• Est. Priority Date: 10/24/2017
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Jin, Feng, Liu, Wen, Qin, Yong, Shi, Qin, Wang, Peng, Zhang, Shi Lei
• Primary Examiner(s): Tsai, Tsung Yin
• Application Number:
  US15/791,821
• Publication Number:
  US 20190122071A1
• Time in Patent Office: 763 Days
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 18/2431 Multiple classes
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06V 10/763 Non-hierarchical techniques, e.g. based on statistics of modelling distributions
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/174 Facial expression recognition

Abstract

Techniques are described that facilitate automatically distinguishing between different expressions of a same or similar emotion. In one embodiment, a computer-implemented is provided that comprises partitioning, by a device operatively coupled to a processor, a data set comprising facial expression data into different clusters of the facial expression data based on one or more distinguishing features respectively associated with the different clusters, wherein the facial expression data reflects facial expressions respectively expressed by people. The computer-implemented method can further comprise performing, by the device, a multi-task learning process to determine a final number of the different clusters for the data set using a multi-task learning process that is dependent on an output of an emotion classification model that classifies emotion types respectively associated with the facial expressions.

Claims

1. A system, comprising:

a memory that stores computer executable components;

a processor that executes the computer executable components stored in the memory, wherein the computer executable components comprise;

a clustering component that partitions a data set comprising facial expression data into different clusters of the facial expression data based on one or more distinguishing features respectively associated with the different clusters, wherein the facial expression data reflects facial expressions respectively expressed by people, and wherein the clustering component iteratively partitions the data set into the different clusters such that respective iterations result in an incrementally increased number of the different clusters; and

a multi-task learning component that determines a final number of the different clusters for the data set using a multi-task learning process that is dependent on an output of an emotion classification model that classifies emotion types respectively associated with the facial expressions, wherein the multi-task learning process comprises iteratively applying the emotion classification model to the different clusters generated at the respective iterations, and determining the final number of the different clusters based on a number of clusters associated with an iteration of the respective iterations associated with a drop in a classification rate by the emotion classification model.

2. The system of claim 1, wherein the people are respectively associated with different demographic profiles.

3. The system of claim 1, wherein the facial expression data comprises respective images of faces of the people with the facial expressions.

4. The system of claim 1, wherein the multi-task learning component further determines facial feature representations for the different clusters.

5. The system of claim 4, further comprising:

a model optimization component that trains the emotion classification model to determine the emotion types corresponding to the facial feature representations using the facial expression data and information defining the emotion types that respectively correspond to the facial feature representations, resulting in an updated emotion classification model configured to distinguish between different facial expressions of a same emotion type associated with different demographic profiles.

6. The system of claim 4, wherein the facial feature representations respectively comprise two or more different facial feature representations for a same emotion type of the emotion types.

7. The system of claim 6, wherein the two or more different facial feature representations are associated with different demographic profiles.

8. The system of claim 6, wherein the two or more different facial feature representations comprise a first facial feature representation associated with a first set of features and a second facial feature representation associated with a second set of features, and wherein the computer executable components further comprise:

a feature extraction component that determines a first subset of the first set of features that are different from the second set of features and a second subset of the second set of features that are different from the first set of features.

9. A computer implemented method, comprising:

partitioning, by a device operatively coupled to a processor, a data set comprising facial expression data into different clusters of the facial expression data based on one or more distinguishing features respectively associated with the different clusters, wherein the facial expression data reflects facial expressions respectively expressed by people, and wherein the clustering comprises iteratively partitioning the data set into the different clusters such that respective iterations result in an incrementally increased number of the different clusters; and

performing, by the device, a multi-task learning process to determine a final number of the different clusters for the data set, wherein the multi-task learning process is dependent on an output of an emotion classification model that classifies emotion types respectively associated with the facial expressions, and wherein the performing the multi-task learning process comprises iteratively applying the emotion classification model to the different clusters generated at the respective iterations, and determining the final number of the different clusters based on a number of clusters associated with an iteration of the respective iterations associated with a drop in a classification rate by the emotion classification model.

10. The computer implemented method of claim 9, wherein the people are respectively associated with different demographic profiles, and wherein the facial expression data comprises respective images of faces of the people with the facial expressions.

11. The computer implemented method of claim 9, wherein the performing the multi-task learning process further comprises determining facial feature representations for the different clusters.

12. The computer implemented method of claim 11, wherein the facial feature representations respectively comprise two or more different facial feature representations for a same emotion type of the emotion types.

13. The computer implemented method of claim 12, wherein the two or more different facial feature representations are associated with different demographic profiles.

14. The computer implemented method of claim 11, further comprising:

training, by the device, the emotion classification model to determine the emotion types corresponding to the facial feature representations using the facial expression data and information defining the emotion types that respectively correspond to the facial feature representations; and

generating, by the device and based on the training, a new emotion classification model configured to distinguish between different facial expressions of a same emotion type associated with different demographic profiles.

15. A computer program product facilitating automatically distinguishing between different facial expressions of a same emotion type associated with different demographic profiles, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processing component to cause the processing component to:

cluster a data set comprising facial expression data into different clusters of the facial expression data based on one or more distinguishing features respectively associated with the different clusters, wherein the facial expression data reflects facial expressions respectively expressed by people, and wherein the clustering of the data set comprises iteratively partitioning the data set into the different clusters such that respective iterations result in an incrementally increased number of the different clusters; and

determine a final number of the different clusters for the data set and facial feature representations for the different clusters using a multi-task learning process that is dependent on an output of an emotion classification model that classifies emotion types respectively associated with the facial expressions, and wherein the using the multi-task learning process comprises iteratively applying the emotion classification model to the different clusters generated at the respective iterations, and determining the final number of the different clusters based on a number of clusters associated with an iteration of the respective iterations associated with a drop in a classification rate by the emotion classification model.

16. The computer program product of claim 15, wherein the program instructions executable by the processing component to further cause the processing component to:

train the emotion classification model to determine the emotion types corresponding to the facial feature representations using the facial expression data and information defining the emotion types that respectively correspond to the facial feature representations; and

generate a new emotion classification model configured to distinguish between the different facial expressions of the same emotion type.

17. The computer program product of claim 15, wherein the people are respectively associated with different demographic profiles, and wherein the facial expression data comprises respective images of faces of the people with the facial expressions.

18. The computer program product of claim 15, wherein the performing the multi-task learning process further comprises determining facial feature representations for the different clusters.

19. The computer program product of claim 18, wherein the facial feature representations respectively comprise two or more different facial feature representations for a same emotion type of the emotion types.

20. The computer program product of claim 19, wherein the two or more different facial feature representations are associated with different demographic profiles.