REAL-TIME EMOTION TRACKING SYSTEM

• US 20150235655A1
• Filed: 05/04/2015
• Published: 08/20/2015
• Est. Priority Date: 12/12/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: DIMITRIADIS, Dimitrios, GILBERT, Mazin E., MISHRA, Taniya, SCHROETER, Horst J.
• Granted Patent: US 9,355,650 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G10L 17/04 Training, enrolment or model building
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices
• CPC Class Codes: G10L 25/48 specially adapted for particular use
• CPC Class Codes: G10L 25/63 for estimating an emotional state

Abstract

Devices, systems, methods, media, and programs for detecting an emotional state change in an audio signal are provided. A plurality of segments of the audio signal is received, with the plurality of segments being sequential. Each segment of the plurality of segments is analyzed, and, for each segment, an emotional state and a confidence score of the emotional state are determined. The emotional state and the confidence score of each segment are sequentially analyzed, and a current emotional state of the audio signal is tracked throughout each of the plurality of segments. For each segment, it is determined whether the current emotional state of the audio signal changes to another emotional state based on the emotional state and the confidence score of the segment.

Claims

1. A device for detecting an emotional state change in an audio signal, the device comprising:

a processor; and

a memory storing instructions that, when executed by the processor, cause the processor to perform operations including;

receiving a plurality of segments of the audio signal, the plurality of segments being sequential;

sequentially analyzing each segment of the plurality of segments and determining, for each segment, an emotional state from among a plurality of emotional states and a confidence score of the emotional state;

sequentially analyzing the emotional state and the confidence score of each segment and tracking a current emotional state of the audio signal throughout each of the plurality of segments; and

determining, for each segment, whether the current emotional state of the audio signal changes to an other emotional state of the plurality of emotional states based on the emotional state and the confidence score of the segment,

wherein the processor determines that the current emotional state of the audio signal changes to the other emotional state of the plurality of emotional states when the emotional state of a predetermined number of the plurality of segments is the other emotional state with the confidence score of the emotional state of each of the predetermined number of the plurality of segments being below a predetermined threshold.

2. The device according to claim 1, wherein

the processor further determines that the current emotional state of the audio signal changes to the other emotional state of the plurality of emotional states when the emotional state of one of the plurality of segments is the other emotional state with the confidence score of the emotional state of the one of the plurality of segments being greater than the predetermined threshold.

3. The device according to claim 2, wherein

the predetermined number of the plurality of segments is consecutive in the audio signal.

4. The device according to claim 2, wherein

the predetermined number of the plurality of segments is inconsecutive and within a predetermined time frame in the audio signal.

5. The device according to claim 2, wherein

the processor further determines that the current emotional state of the audio signal changes to the other emotional state of the plurality of emotional states when, for each of the predetermined number of the plurality of segments, the emotional state is not equal to the current emotional state with the confidence score of the emotional state being greater than the predetermined threshold.

6. The device according to claim 1, further comprising:

a database that stores the audio signal and tracked emotional states in association with a customer service agent.

7. The device according to claim 6, wherein

the processor is configured to identify transitions from a first emotional state to a second emotional state in the database.

8. The device according to claim 1, further comprising:

an automatic speech recognition module that analyzes the audio signal to determine the plurality of segments.

9. The device according to claim 8, wherein

each of the plurality of segments of the audio signal comprises a word of speech in the audio signal.

10. The device according to claim 1, wherein

each segment is analyzed in accordance with a plurality of analyses,

for each segment, a plurality of emotional states and a plurality of confidence scores of the plurality of emotional states are determined in accordance with the plurality of analyses, and

for each segment, the plurality of emotional states and the plurality of confidence scores of the plurality of emotional states are combined for determining the emotional state and the confidence score of the emotional state of the segment.

11. The device according to claim 1, the operations further including:

issuing an instruction for controlling a motorized vehicle in response to determining that the current emotional state of the audio signal changes to the other emotional state of the plurality of emotional states.

12. The device according to claim 1, the operations further including:

issuing an instruction for controlling an alarm system in response to determining that the current emotional state of the audio signal changes to the other emotional state of the plurality of emotional states.

13. A method for detecting an emotional state change in an audio signal, the method comprising:

receiving a plurality of segments of the audio signal, the plurality of segments being sequential;

sequentially analyzing, by a processor, each segment of the plurality of segments and determining, for each segment and by the processor, an emotional state from among a plurality of emotional states and a confidence score of the emotional state;

sequentially analyzing, by the processor, the emotional state and the confidence score of each segment and tracking a current emotional state of the audio signal throughout each of the plurality of segments; and

determining, for each segment and by the processor, whether the current emotional state of the audio signal changes to an other emotional state of the plurality of emotional states based on the emotional state and the confidence score of the segment,

wherein the processor determines that the current emotional state of the audio signal changes to the other emotional state of the plurality of emotional states when the emotional state of a predetermined number of the plurality of segments is the other emotional state with the confidence score of the emotional state of each of the predetermined number of the plurality of segments being below a predetermined threshold.

14. The method according to claim 13, wherein

the processor further determines that the current emotional state of the audio signal changes to the other emotional state of the plurality of emotional states when the emotional state of one of the plurality of segments is the other emotional state with the confidence score of the emotional state of the one of the plurality of segments being greater than the predetermined threshold.

15. The method according to claim 14, wherein

the predetermined number of the plurality of segments is consecutive in the audio signal.

16. The method according to claim 14, wherein

the processor further determines that the current emotional state of the audio signal changes to the other emotional state of the plurality of emotional states when, for each of the predetermined number of the plurality of segments, the emotional state is not equal to the current emotional state with the confidence score of the emotional state being greater than the predetermined threshold.

17. A non-transitory computer-readable medium having an executable computer program for detecting an emotional state change in an audio signal that, when executed by a processor, causes the processor to perform operations comprising:

receiving a plurality of segments of the audio signal, the plurality of segments being sequential;

sequentially analyzing each segment of the plurality of segments and determining, for each segment, an emotional state from among a plurality of emotional states and a confidence score of the emotional state;

sequentially analyzing the emotional state and the confidence score of each segment and tracking a current emotional state of the audio signal throughout each of the plurality of segments; and

determining, for each segment, whether the current emotional state of the audio signal changes to an other emotional state of the plurality of emotional states based on the emotional state and the confidence score of the segment,

wherein the processor determines that the current emotional state of the audio signal changes to the other emotional state of the plurality of emotional states when the emotional state of a predetermined number of the plurality of segments is the other emotional state with the confidence score of the emotional state of each of the predetermined number of the plurality of segments being below a predetermined threshold.

18. The non-transitory computer-readable medium according to claim 17, wherein

the processor further determines that the current emotional state of the audio signal changes to the other emotional state of the plurality of emotional states when the emotional state of one of the plurality of segments is the other emotional state with the confidence score of the emotional state of the one of the plurality of segments being greater than the predetermined threshold.

19. The non-transitory computer-readable medium according to claim 18, wherein

the predetermined number of the plurality of segments is consecutive in the audio signal.

20. The non-transitory computer-readable medium according to claim 18, wherein

the processor further determines that the current emotional state of the audio signal changes to the other emotional state of the plurality of emotional states when, for each of the predetermined number of the plurality of segments, the emotional state is not equal to the current emotional state with the confidence score of the emotional state being greater than the predetermined threshold.