MULTI-MODAL SENSOR BASED EMOTION RECOGNITION AND EMOTIONAL INTERFACE

• US 20140112556A1
• Filed: 10/19/2012
• Published: 04/24/2014
• Est. Priority Date: 10/19/2012
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Kalinli-Akbacak, Ozlem
• Granted Patent: US 9,031,293 B2
• Time in Patent Office: Days
• US Class Current: 382/128
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices
• CPC Class Codes: G10L 25/63 for estimating an emotional state

Abstract

Features, including one or more acoustic features, visual features, linguistic features, and physical features may be extracted from signals obtained by one or more sensors with a processor. The acoustic, visual, linguistic, and physical features may be analyzed with one or more machine learning algorithms and an emotional state of a user may be extracted from analysis of the features. It is emphasized that this abstract is provided to comply with the rules requiring an abstract that will allow a searcher or other reader to quickly ascertain the subject matter of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims.

Claims

1. A method for determining an emotional state of a user, comprising:

extracting features including one or more acoustic features, visual features, linguistic features, physical features from signals obtained by one or more sensors with a processor;

analyzing the features including the acoustic features, visual features, linguistic features, and physical features with one or more machine learning algorithms implemented on a processor; and

extracting an emotional state of the user from analysis of the features including analysis of the acoustic features, visual features, linguistic features, and physical features with the one or more machine learning algorithms.

2. The method of claim 1, wherein analyzing the acoustic features, visual features, linguistic features, and physical features includes use of a machine learning algorithm that analyzes a combination of two or more different types of features from the group of the acoustic features, visual features, linguistic features, and physical features.

3. The method of claim 2, further comprising augmenting and reducing the dimension of the acoustic features, visual features, linguistic features, and physical features before using the machine learning algorithm to analyze the combination of two or more different types of features.

4. The method of claim 1, wherein analyzing the acoustic features, visual features, linguistic features, and physical features includes use of separate machine learning algorithms for the acoustic, visual, linguistic, and physical features.

5. The method of claim 4, wherein a first machine learning algorithm provides feedback to a second machine learning algorithm in a serial fashion.

6. The method of claim 4, wherein analyzing the acoustic features, visual features, linguistic features, and physical features includes reducing a dimension of the acoustic, visual, linguistic, and physical features to produce corresponding reduced dimension acoustic, visual, linguistic, and physical features and using corresponding separate machine learning algorithms to analyze the reduced dimension acoustic, visual, linguistic, and physical features.

7. The method of claim 1, wherein analyzing the acoustic features, visual features, linguistic features, and physical features includes fusing estimated emotional states obtained in parallel from separate machine learning algorithms for the acoustic, visual, linguistic, and physical features.

8. The method of claim 1, further comprising changing a state of a computer system in response to the emotional state of the user extracted from analysis of the acoustic features, visual features, linguistic, and physical features.

9. The method of claim 8, further comprising performing automated speech recognition of one or more utterances of the user with the computer system.

10. The method of claim 8, wherein changing the state of the computer system includes using the emotional state of the user extracted from analysis of the acoustic features, visual features, linguistic features, and physical features to adjust one or more models used in the automated speech recognition.

11. The method of claim 1, wherein the acoustic features include features extracted from sound signals corresponding to the user'"'"'s speech.

12. The method of claim 1, wherein the visual features include facial expressions, head poses, body gestures, and eye movements.

13. The method of claim 1, wherein the linguistic features include semantics, syntax, and lexical features from both user and non-user.

14. The method of claim 1, wherein the features further include one or more context features.

15. The method of claim 14, wherein the one or more context features include one or more features from game, movie, video, music, environmental characteristics, non-user characters, sound signals not corresponding to the user'"'"'s speech, or visual signals not corresponding to the user.

16. The method of claim 1, wherein the physical features include heart rate, blood pressure, respiration rate, skin moisture, grip an object, depth and pace of breath, serotonin level, epinephrine level, skin moisture level, skin temperature, pressure in hands/fingers/wrist, level of saliva, level of hormones in enzymes in saliva, level of enzymes in saliva, or skin conductance.

17. An apparatus for determining an emotional state of a user, comprising:

a processor;

instructions executable by the processor, wherein the instructions are configured, when executed to extract one or more acoustic features, visual features, linguistic features, and physical features of the user from signals obtained by one or more sensors, analyze the acoustic features, visual features, linguistic features, and physical features with one or more machine learning algorithms; and

extract an emotional state of the user from analysis of the acoustic features, visual features, linguistic features, and physical features with the one or more machine learning algorithms.

18. The apparatus of claim 17, wherein the instructions are configured to analyze the acoustic features, visual features, linguistic features, and physical features by implementing a machine learning algorithm that analyzes a combination of two or more different types of features from the group of the acoustic features, visual features, linguistic features, and physical features.

19. The apparatus of claim 18, wherein the instructions are further configured to augment and reduce a dimension of the acoustic features, visual features, linguistic features, and physical features before using the machine learning algorithm to analyze the combination of two or more different types of features.

20. The apparatus of claim 17, wherein the instructions are configured to analyze the acoustic features, visual features, linguistic features, and physical features through use of separate machine learning algorithms for the acoustic, visual, linguistic, and physical features.

21. The apparatus of claim 20, wherein a first machine learning algorithm provides feedback to a second machine learning algorithm in a serial fashion.

22. The apparatus of claim 20, wherein the instructions are configured to analyze the acoustic features, visual features, linguistic features, and physical features by reducing a dimension of the acoustic, visual, linguistic, and physical features to produce corresponding reduced dimension acoustic, visual, linguistic, and physical features and by using corresponding separate machine learning algorithms to analyze the reduced dimension acoustic, visual, linguistic, and physical features.

23. The apparatus of claim 17, wherein the instructions are further configured to change a state of a computer system in response to the emotional state of the user extracted from analysis of the acoustic features, visual features, linguistic features, and physical.

24. The apparatus of claim 23, wherein the instructions are further configured to perform automated speech recognition of one or more utterances of the user with the computer system.

25. The apparatus of claim 23, wherein the instructions are further configured to change the state of the computer system by using the emotional state of the user extracted from analysis of the acoustic features, visual features, linguistic features, physical features to adjust a model used in the automated speech recognition.

26. The apparatus of claim 17, wherein the features further include one or more context features.

27. The apparatus of claim 26, wherein the one or more context features include one or more features from game, movie, video, music, environmental characteristics, non-user characters, sound signals not corresponding to the user'"'"'s speech, or visual signals not corresponding to the user.

28. A non-transitory computer-readable medium having computer executable instructions embodied therein, wherein the instructions are configured to implement a method for determining an emotional state of a user, when executed, the method comprising:

extracting one or more acoustic features, visual features, linguistic features, and physical features from signals obtained by one or more sensors;

analyzing the acoustic features, visual features, linguistic features, and physical features with one or more machine learning algorithms; and

extracting an emotional state of the user from analysis of the acoustic features, visual features, linguistic features, and physical features with the one or more machine learning algorithms.