Emotion Recognition System

• US 20080052080A1
• Filed: 11/30/2006
• Published: 02/28/2008
• Est. Priority Date: 11/30/2005
• Status: Active Grant

• Current Assignee: University of Southern California
• Original Assignee: University of Southern California
• Inventors: Narayanan, Shrikanth
• Granted Patent: US 8,209,182 B2
• Time in Patent Office: Days
• US Class Current: 704/270
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06V 40/175 Static expression
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices

Abstract

An emotion recognition system for assessing human emotional behavior from communication by a speaker includes a processing system configured to receive signals representative of the verbal and/or non-verbal communication. The processing system derives signal features from the received signals. The processing system is further configured to implement at least one intermediate mapping between the signal features and one or more elements of an emotional ontology in order to perform an emotion recognition decision. The emotional ontology provides a gradient representation of the human emotional behavior.

Claims

1. An emotion recognition system for assessing human emotional behavior from communication by a speaker, the system comprising:

a processing system configured to receive signals representative of the communication, and to derive signal features from the received signals, the processing system further configured to implement at least one intermediate mapping between the signal features and one or more elements of an emotional ontology in order to perform an emotion recognition decision, the emotional ontology providing a gradient representation of the human emotional behavior.

2. The system of claim 1, wherein the elements of the emotional ontology comprises at least one of:

expressive variables;

clusters of expressive variables; and

relations among the clusters and the variables.

3. The system of claim 1, wherein the signal features comprise verbal or non-verbal cues of the human expressive behavior.

4. The system of claim 1, wherein the signal features include at least one of:

acoustic information about the signals;

lexical information about the signals; and

discourse information about the signals.

5. The system of claim 1, wherein the received signals relate to one or more of:

vocal expression by the speaker;

heart rate of the speaker;

facial expression of the speaker;

gesture of the speaker; and

skin temperature of the speaker.

6. The system of claim 5, wherein the relations comprise probabilities that model a conditional dependency between the variables.

7. The system of claim 5, wherein the relations between the variables comprise a weighted graph including one or more weights, each weight denoting a strength of the relationships between the variables.

8. The system of claim 5, wherein the emotional ontology comprises one or more emotion categories, and wherein the expressive variables comprise words and phrases that provide linguistic descriptions of the human emotional behavior.

9. The system of claim 8, wherein the intermediate mapping comprises computation of emotional salience, and wherein the emotional salience is defined as mutual information between a word and an emotion category.

10. The system of claim 9, wherein a mathematical representation of the emotional salience comprises:

$\mathrm{sal}<br><br>\left(v_n\right)=I<br><br>\left(E;<br><br>W=v_n\right)=\underset{j=1}{\overset{k}{\sum <br><br>}}<br><br>P<br><br>\left(e_j|v_n\right)<br><br>i<br><br>\left(v_n,e_j\right),$

where v_n denotes an n-th word in an utterance W={v₁, v₂, . . . , v_n, . . . , v_N};

E denotes a set of emotion categories {e₁, e₂, . . . , e_j, . . . , e_k};

e_j denotes a j-th emotion category in the set E;

P(e_j|v_n) denotes a posterior probability that an utterance containing word v_n implies emotion category e_j;

P(e_j) denotes a prior probability of the emotion category e_j, and i(v_n, e_j) denotes mutual information between the word v_n and the emotion category e_j and is represented mathematically by;

$i<br><br>\left(v_n,e_j\right)=\log <br><br>\frac{P<br><br>\left(e_j|v_n\right)}{P<br><br>\left(e_j\right)}.$

11. The system of claim 1, wherein the signal features comprise features that are adapted to be processed at a plurality of time scales;

and wherein the plurality of time scales comprise one or more of;

linguistic time scales; and

signal change heuristics.

12. The system of claim 11, wherein the signal change heuristics comprise one or more of:

presence of speech;

absence of speech;

change of scene in which the speech communication takes place; and

change of audio quality of the speech communication.

13. The system of claim 11, wherein the plurality of time scales comprise linguistic time scales, and wherein the linguistic time scales comprise one or more of:

a phoneme;

a subphoneme;

a syllable;

a word;

a phrase;

a phrase boundary;

an utterance;

a sentence;

a discourse; and

a session.

14. The system of claim 1,

wherein the signal features comprise one or more of time domain features and transform domain features;

wherein the time domain features comprise one or more of;

amplitudes of the signals;

derivatives of the amplitudes;

energy contours of the signals;

and intensity contours of the signals; and

wherein the transform domain features comprise one or more of;

spectral domain features of the signals;

time-frequency transforms of the signals;

variants of the time-frequency transforms; and

derivatives of the spectral domain features,

wherein the derivatives of the spectral domain features comprise one or more of;

pitch;

formants; and

filterbank energies.

15. The system of claim 1, wherein the intermediate mapping comprises at least one of:

derivation of information relating to phenome, syllables, words, and phrases;

computation of emotional salience between a word and an emotion category;

detection of word class;

analysis of facial expression;

tracking of activation, valence, and dominance at multiple time resolution;

analysis of gestures;

modeling of signal behavior; and

classification of auditory information.

16. The system of claim 1, wherein the processing system comprises at least one of:

an automatic speech recognizer configured to derive from the received signals information relating to one or more of;

phonemes;

syllables;

words;

and phrases;

a speech rate estimator and a boundary detector configured to derive information relating to linguistic expressive markers;

a lexical analyzer configured to calculate emotional salience of one or more word sequences;

a parser configured to provide emotionally salient parse structures;

a speech act detector configured to provide communication act information;

a word class detector configured to detect word class;

an audio classifier configured to detect nonverbal cues of emotion;

an audio classifier configured to detect verbal disfluency as cues of emotion;

a facial expression analyzer configured to analyze facial expression of the speaker; and

a gesture analyzer configured to analyze gestures of the speaker.

17. The system of claim 8, wherein the emotion categories comprise non-negative and negative emotions.

18. A method of assessing human emotional behavior from communication by a speaker, the method comprising:

receiving signals representative of the communication;

deriving signal features from the received signals; and

implementing at least one intermediate mapping between the signal features and one or more elements of an emotional ontology to perform an emotion recognition decision, the emotion ontology providing a gradient representation of the human emotional behavior.

19. The method of claim 18, wherein the signal features comprise words that describe the human emotional behavior, and wherein the act of implementing at least one intermediate mapping comprises computing emotional salience between at least one of the words and an emotion category within the emotional ontology.