System and method for building emotional machines

• US 8,204,749 B2
• Filed: 03/21/2011
• Issued: 06/19/2012
• Est. Priority Date: 07/20/2005
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Hakkani-Tur, Dilek Z., Liscombe, Jackson J., Riccardi, Guiseppe
• Primary Examiner(s): YEN, ERIC L
• Application Number:
  US13/053,179
• Publication Number:
  US 20110172999A1
• Time in Patent Office: 456 Days
• Field of Search: 704/1, 704/9, 704/238, 704/239, 704/240, 704/257, 704/270, 704/270.1, 704/275
• US Class Current: 704/270
• CPC Class Codes: G06F 40/253 Grammatical analysis; Style critique
• CPC Class Codes: G10L 15/1807 using prosody or stress
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices

Abstract

A system, method and computer-readable medium for practicing a method of emotion detection during a natural language dialog between a human and a computing device are disclosed. The method includes receiving an utterance from a user in a natural language dialog, receiving contextual information regarding the natural language dialog which is related to changes of emotion over time in the dialog, and detecting an emotion of the user based on the received contextual information. Examples of contextual information include, for example, differential statistics, joint statistics and distance statistics.

Claims

1. A method for emotion detection, the method comprising:

receiving an utterance from a user as part of a natural language dialog between the user and a computing device to yield a received utterance;

receiving non-repetitive prompts generated by the computing device; and

detecting an emotion of the user based at least in part on the received utterance and the non-repetitive prompts.

2. The method of claim 1, wherein at least one prompt of the non-repetitive prompts comprises a greeting prompt, a re-prompt, a specification prompt, an acknowledgement prompt, and an informative prompt.

3. The method of claim 2, wherein detecting the emotion of the user further is based on prosodic and lexical features extracted from the received utterance.

4. The method of claim 1, further comprising receiving contextual information used for detecting the emotion of the user, the contextual information comprising at least one of differential statistics, joint statistics, distance statistics, and statistics collected during a prior dialog between the user and the computing device, comprising (1) differential information of prosodic features, (2) temporal variation of emotion indicators inter-utterance, and (3) temporal variation of emotion indicators intra-utterance.

5. The method of claim 4, wherein the differential statistics relate to a rate of change of prosodic values from one turn to another in the natural language dialog.

6. The method of claim 4, wherein the joint statistics relate to examining a discrete-value feature in a current utterance with reference to at least one previous value for the discrete-value feature.

7. The method of claim 6, wherein the discrete-value feature is one of a dialog act, a number of dialogs, a number of user turns, and a number of words per turn.

8. The method of claim 4, wherein the distance statistics relate to similarities in lexical features between a current utterances and at least one previous utterance.

9. A non-transitory computer-readable medium, storing instructions for controlling a computing device to detect emotion during a natural language dialog between a user and the computing device, the instructions comprising:

receiving an utterance from the user in a human-computing device natural language dialog;

receiving non-repetitive prompts in the natural-language dialog, wherein the non-repetitive prompts are prompts by the computing device; and

detecting an emotion of the user based on the utterance and at least in part on the non-repetitive prompts.

10. The non-transitory computer-readable medium of claim 9, wherein the non-repetitive prompts comprise at least one of:

greeting, re-prompt, specification, acknowledgement, and informative.

11. The non-transitory computer-readable medium of claim 10, wherein detecting the emotion of the user further is based on prosodic and lexical features extracted from the received utterance.

12. The non-transitory computer-readable medium of claim 9, further comprising receiving contextual information used for detecting the emotion of the user, the contextual information comprising at least one of differential statistics, joint statistics, distance statistics, and statistics collected during a prior dialog between the user and the computing device including (1) differential information of prosodic features, (2) temporal variation of emotion indicators inter-utterance, and (3) temporal variation of emotion indicators intra-utterance.

13. The non-transitory computer-readable medium of claim 12, wherein differential statistics relate to a rate of change of prosodic values from one turn to another in the natural language dialog.

14. The non-transitory computer-readable medium of claim 13, wherein the joint statistics relate to examining a discrete-value feature in the current utterance with reference to at least one previous value for the discrete-value feature.

15. The non-transitory computer-readable medium of claim 14, wherein the discrete-value feature is one of a dialog act, a number of dialogs, a number of user turns, and a number of words per turn.

16. The non-transitory computer-readable medium of claim 12, wherein the distance statistics relate to similarities in lexical features between the current utterances and at least one previous utterance.

17. A computing device that detects emotion during a natural language dialog between a user and the computing device, the computing device comprising:

a processor;

a first module configured to control the processor to receive an utterance from a user in a human-computing device natural language dialog;

a second module configured to control the processor to receive non-repetitive prompts in the natural language dialog, wherein the non-repetitive prompts are prompts by the computing device; and

a third module configured to control the processor to detect an emotion of the user based on the utterance and at least in part on the non-repetitive prompts.

18. The computing device of claim 17, wherein the non-repetitive prompts comprise at least one of:

greeting, re-prompt, specification, acknowledgement, and informative.

19. The computing device of claim 18, wherein the third module is further configured to detect the emotion of the user based on prosodic and lexical features extracted from the received utterance.

20. The computing device of claim 17, wherein the second module is further configured to receive contextual information used for detecting the emotion of the user, the contextual information comprising at least one of differential statistics, joint statistics, distance statistics, and statistics collected during a prior dialog between the user and the computing device including (1) differential information of prosodic features, (2) temporal variation of emotion indicators inter-utterance, and (3) temporal variation of emotion indicators intra-utterance.