Systems and Methods for Enhancing Responsiveness to Utterances Having Detectable Emotion

• US 20190325867A1
• Filed: 04/20/2018
• Published: 10/24/2019
• Est. Priority Date: 04/20/2018
• Status: Active Grant

• Current Assignee: Spotify AB (Spotify Technology SA)
• Original Assignee: Spotify AB (Spotify Technology SA)
• Inventors: BROMAND, Daniel, GUSTAFSSON, David, MITIC, Richard, MENNICKEN, Sarah
• Granted Patent: US 10,621,983 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/109 Font handling; Temporal or kinetic typography
• CPC Class Codes: G10L 13/02 Methods for producing synthetic speech; Speech synthesisers
• CPC Class Codes: G10L 13/0335 Pitch control
• CPC Class Codes: G10L 13/10 Prosody rules derived from text; Stress or intonation
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 25/63 for estimating an emotional state

Abstract

Methods, systems, and related products that provide emotion-sensitive responses to user'"'"'s commands and other utterances received at an utterance-based user interface. Acknowledgements of user'"'"'s utterances are adapted to the user and/or the user device, and emotions detected in the user'"'"'s utterance that have been mapped from one or more emotion features extracted from the utterance. In some examples, extraction of a user'"'"'s changing emotion during a sequence of interactions is used to generate a response to a user'"'"'s uttered command. In some examples, emotion processing and command processing of natural utterances are performed asynchronously.

Claims

1. A method comprising:

receiving, via a user device, a natural utterance, the natural utterance including at least one emotion feature;

extracting the at least one emotion feature from the natural utterance;

mapping each of the at least one emotion feature to an emotion;

inferring, based on the emotion, an inferred command of the natural utterance; and

responding to the natural utterance at least by one or both of;

     1) performing an action corresponding to the inferred command; and

     2) providing a verbalized acknowledgement of the natural utterance adapted to the emotion.

2. The method of claim 1, wherein the responding includes both the performing an action corresponding to the inferred command and the providing a verbalized acknowledgement of the natural utterance adapted to the emotion.

3. The method of claim 1, wherein the verbalized acknowledgement includes an indication of the action corresponding to the inferred command.

4. The method of claim 1, wherein the verbalized acknowledgement includes an indication that a command was not extractable from the natural utterance.

5. The method of claim 1, wherein the verbalized acknowledgement comprises a synthesized utterance.

6. The method of claim 5, wherein the synthesized utterance is adapted to the emotion with at least one audible stylization, the at least one audible stylization being one or more of a cadence, a volume, a pitch, a word, a string of words, a pace, and a tone.

7. The method of claim 1, wherein the verbalized acknowledgement comprises a displayed text.

8. The method of claim 7, wherein the displayed text is adapted to the emotion with at least one visual stylization, the at least one visual stylization being one or more of a font type, a font size, a font color, capital letters, a number of words, and a punctuation mark.

9. The method of claim 1, wherein the at least one emotion feature is one or more of a cadence, a volume, a pitch, a word, a string of words, a pace, and a tone of the natural utterance.

10. The method of claim 1, wherein the mapping the emotion to an inferred command is partially based on an identity of a user.

11. The method of claim 2, wherein a type of the verbalized acknowledgement is based on a type of the user device.

12. The method of claim 11,

wherein the user device comprises an automobile, and

wherein the type of the verbalized acknowledgement is a synthesized utterance.

13. The method of claim 1, further comprising:

extracting at least two emotion features from the natural utterance; and

detecting a conflict between the at least two extracted emotion features.

14. The method of claim 13, further comprising:

reconciling the conflict and, based on the reconciling, performing the inferring.

15. A non-transitory computer readable medium comprising:

an emotion processor having one or more sequences of emotion processor instructions that, when executed by one or more processors, causes the one or more processors to generate an output adapted to an emotion detected in a natural utterance;

and one or more sequences of instructions which, when executed by one or more processors, cause the one or processors to;

receive, via a user device, a natural utterance, the natural utterance including at least one emotion feature;

extract the at least one emotion feature from the natural utterance;

map each of the at least one emotion feature to an emotion;

infer, based on the emotion, an inferred command of the natural utterance; and

respond to the natural utterance at least by one or both of;

     1) performing an action corresponding to the inferred command; and

     2) providing a verbalized acknowledgement of the natural utterance adapted to the emotion.