Emotion detection device and method for use in distributed systems

• US 8,214,214 B2
• Filed: 10/14/2009
• Issued: 07/03/2012
• Est. Priority Date: 12/03/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Phoenix Solutions Incorporated (CDC Corporation)
• Inventors: Bennett, Ian M.
• Primary Examiner(s): AZAD, ABUL K
• Application Number:
  US12/579,233
• Publication Number:
  US 20100036660A1
• Time in Patent Office: 993 Days
• Field of Search: 704231-357
• US Class Current: 704/254
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices

Abstract

A prosody analyzer enhances the interpretation of natural language utterances. The analyzer is distributed over a client/server architecture, so that the scope of emotion recognition processing tasks can be allocated on a dynamic basis based on processing resources, channel conditions, client loads etc. The partially processed prosodic data can be sent separately or combined with other speech data from the client device and streamed to a server for a real-time response. Training of the prosody analyzer with real world expected responses improves emotion modeling and the real-time identification of potential features such as emphasis, intent, attitude and semantic meaning in the speaker'"'"'s utterances.

Claims

1. In a method for performing real-time continuous speech recognition distributed across a client device and a server device, and which transfers speech data from an utterance to be recognized using a packet stream of extracted acoustic feature data including at least some cepstral coefficients, the improvement comprising:

extracting prosodic features at the client device from the utterance to generate extracted prosodic data;

transferring said extracted prosodic data with said extracted acoustic feature data to the server device;

recognizing words spoken continuously in said utterance at said server device based on said extracted acoustic feature data; and

recognizing an emotion state of a speaker of the utterance at said server device based on at least said extracted prosodic data;

wherein operations associated with recognition of prosodic features as well as words in the utterance are distributed across the client device and server device, such that a first set of prosodic recognition operations take place at the client device, and a second set of prosodic recognition operations take place at the server device to recognize said emotion state; and

wherein said prosodic data and acoustic feature data are transmitted using different priorities.

2. The method of claim 1, wherein said operations are distributed across the client device and server device on a case-by-case basis.

3. The method of claim 1 further including a parts-of-speech analyzer for identifying a first set of emotion cues based on evaluating a syntax structure of the utterance.

4. The method of claim 3, further including a real-time classifier for identifying the emotion state based on said first set of emotion cues and a second set of emotion cues derived from said extracted prosodic data.

5. The method of claim 1, wherein said prosodic features include data values which are related to one or more acoustic measures including one of pitch, duration, and energy.

6. The method of claim 1, wherein said emotion state includes at least one of stress and non-stress.

7. The method of claim 1, wherein said emotion state includes at least one of certainty, uncertainty, or doubt.

8. The method of claim 1, further including a step:

performing an operation on natural language presented in said speech utterance to determine a meaning of said speech utterance.

9. The method of claim 8, further including a step:

forming a database query based on said utterance to retrieve an answer for said query.

10. The method of claim 1, wherein a number of speech recognition operations and prosody recognition operations performed on the utterance can be adjusted automatically based on resource availability.

11. A method for performing real-time emotion detection comprising:

generating representative speech data from a speech utterance;

recognizing words in said utterance;

extracting syntactic cues from said words relating to an emotion state of a speaker of said speech utterance;

extracting prosodic features from the utterance to generate extracted prosodic data, such that a first set of prosodic recognition operations take place at a client device, and a second set of prosodic recognition operations take place at a server device to recognize said emotion state; and

classifying inputs based on said prosodic features and a parts-of speech analyzer relating to said speech utterance and processing the same along with said syntactic cues to output an emotion cue data value corresponding to said emotion state;

wherein said prosodic data and representative speech data are transmitted using different priorities.

12. The method of claim 11, wherein the representative speech data comprises data selected from the group consisting of:

digitized, compressed speech data;

digitized, uncompressed speech data;

extracted acoustic feature data; and

cepstral coefficients.

13. The method of claim 11, further including a step:

performing an operation on natural language presented in said speech utterance to determine a meaning of said speech utterance.

14. The method of claim 13, further including a step:

forming a database query based on said utterance to retrieve an answer for said query.

15. The system method of claim 11, wherein a number of speech recognition operations and prosody recognition operations performed on the utterance can be adjusted automatically based on resource availability.

16. A real-time emotion detector system, comprising:

a prosody analyzer adapted to extract selected acoustic features of a continuous speech utterance;

said prosody analyzer being distributed such that a first set of prosody extraction operations take place at a client device, and a second set of prosody extraction operations take place at a server;

a continuous speech recognizer for identifying words presented in said continuous speech utterance;

a parts-of-speech analyzer adapted to process said words and extract syntactic cues relating to an emotion state of a speaker of said speech utterance; and

a classifier adapted to receive inputs from said prosody analyzer and said parts-of-speech analyzer and process the same to output an emotion cue data value corresponding to said emotion state;

wherein said prosodic data and acoustic feature data are transmitted using different priorities.

17. The system of claim 16, wherein the classifier is a trained Classification and Regression Tree classifier.

18. The system of claim 16, wherein said classifier is trained with data obtained during an off-line training phase.

19. The system of claim 18, wherein said classifier uses a history file containing data values for emotion cues derived from a sample population of test subjects and using a set of sample utterances common to content associated with the real-time recognition system.

20. The system of claim 16, wherein said emotion cue data value is in the form of a data variable suitable for inclusion within a SQL construct.

21. In a system for performing real-time continuous speech recognition which is distributed across a client device and a server device, and which transfers speech data from an utterance to be recognized using a packet stream of extracted acoustic feature data including at least some cepstral coefficients, the improvement comprising:

a first routine executing on the client device configured to extract prosodic features from the utterance and to generate extracted prosodic data;

a second routine executing on the client device configured to transfer said extracted prosodic data with said extracted acoustic feature data to the server device;

a third routine executing on said server device and adapted to recognize words spoken continuously in said utterance based on said extracted acoustic feature data; and

a fourth routine executing on the server device configured to recognize an emotion state of a speaker of the utterance based on at least said extracted prosodic data;

wherein operations associated with recognition of prosodic features and words in the utterance are distributed across the client device and server device, such that a first set of operations take place at the client device, and a second set of operations take place at the server device to recognize said emotion state; and

wherein said prosodic data and acoustic feature data are transmitted using different priorities.

22. The system of claim 21, further including a fifth routine executing on the server device configured to extract syntax information from the utterance and generate a set of emotion cues which are used by said fourth routine in combination with said extracted prosodic data to determine said emotion state.

23. The system of claim 21, wherein said emotion state is used to formulate a response by an interactive agent in a real-time natural language processing system.

24. The system of claim 21, wherein said emotion state is used by an interactive agent to control dialog content and/or a dialog sequence with a user of a speech recognition system.

25. The system of claim 21, wherein said emotion state is used to control visual feedback presented to a user of the real-time speech recognition system.

26. The system of claim 21, wherein said emotion state is used to control non-verbal audio feedback presented to a user of the real-time speech recognition system.

27. The system of claim 26, wherein said non-verbal audio feedback is one of a selected set of audio recordings associated with different user emotion states.

28. The system of claim 21, wherein an amount of prosodic data to be transferred to said server device is determined on a case by case basis in accordance with one or more of the following parameters:

(a) computational capabilities of the respective devices;

(b) communications capability of a network coupling the respective devices;

(c) loading of said server device; and

(d) a performance requirement of a speech recognition task associated with a user query.

29. The system of claim 21, wherein both prosodic data and acoustic feature data are packaged within a common data stream as received at the server device.

30. The system of claim 21, wherein prosodic data and acoustic feature data are packaged within different data streams as received at the server device.

31. The system of claim 21, wherein said prosodic data is selected and configured to have a data content which is significantly less than said acoustic feature data.

32. The system of claim 21, wherein said prosodic data and acoustic feature data are configured with different payload formats within their respective packets by a transport routine.

33. The system of claim 21, wherein said emotion state is determined by evaluating both individual words and an entire sentence of words uttered by the user.

34. The system of claim 21, further including a calibration routine.

35. A real-time emotion detector system, comprising:

a prosody analyzer adapted to extract selected acoustic features of a continuous speech utterance;

said prosody analyzer being distributed such that a first set of prosody extraction operations take place at a client device, and a second set of prosody extraction operations take place at a server;

a continuous speech recognizer for identifying words presented in said continuous speech utterance;

a parts-of-speech analyzer adapted to process said words and extract syntactic cues relating to an emotion state of a speaker of said speech utterance; and

a classifier adapted to receive inputs from said prosody analyzer and said parts-of-speech analyzer and process the same to output an emotion cue data value corresponding to said emotion state;

wherein said prosodic data and representative speech data are transmitted using different priorities.

36. The system of claim 35, wherein the representative speech data comprises data selected from the group consisting of:

digitized, compressed speech data;

digitized, uncompressed speech data;

extracted acoustic feature data; and

cepstral coefficients.

37. The system of claim 35, wherein the classifier is a trained Classification and Regression Tree classifier.

38. The system of claim 35, wherein said classifier is trained with data obtained during an off-line training phase.

39. The system of claim 38, wherein said classifier uses a history file containing data values for emotion cues derived from a sample population of test subjects and using a set of sample utterances common to content associated with the real-time recognition system.

40. The system of claim 35, wherein said emotion cue data value is in the form of a data variable suitable for inclusion within a SQL construct.