Systems and methods for enhancing responsiveness to utterances having detectable emotion

• US 10,622,007 B2
• Filed: 04/20/2018
• Issued: 04/14/2020
• Est. Priority Date: 04/20/2018
• Status: Active Grant

• Current Assignee: Spotify AB (Spotify Technology SA)
• Original Assignee: Spotify AB (Spotify Technology SA)
• Inventors: Bromand, Daniel, Gustafsson, David, Mitic, Richard, Mennicken, Sarah
• Primary Examiner(s): Le, Thuykhanh
• Application Number:
  US15/958,340
• Publication Number:
  US 20190325895A1
• Time in Patent Office: 725 Days
• Field of Search: None
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 13/033 Voice editing, e.g. manipulating the voice of the synthesiser
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 25/63 for estimating an emotional state

Abstract

Methods, systems, and related products that provide emotion-sensitive responses to user'"'"'s commands and other utterances received at an utterance-based user interface. Acknowledgements of user'"'"'s utterances are adapted to the user and/or the user device, and emotions detected in the user'"'"'s utterance that have been mapped from one or more emotion features extracted from the utterance. In some examples, extraction of a user'"'"'s changing emotion during a sequence of interactions is used to generate a response to a user'"'"'s uttered command. In some examples, emotion processing and command processing of natural utterances are performed asynchronously.

Claims

1. A method comprising:

receiving, via a user device, a natural utterance;

converting the natural utterance into text;

extracting a textual emotion feature from the text;

extracting a non-textual emotion feature from the natural utterance, the textual emotion feature and the non-textual emotion feature corresponding, respectively, to conflicting first and second commands;

selecting either the non-textual emotion feature or the textual emotion feature, thereby providing a selected emotion feature;

mapping the selected emotion feature to an emotion; and

responding to the natural utterance by providing a verbalized acknowledgement of the natural utterance adapted to the emotion and performing an action corresponding to the first or the second command associated with the selected emotion feature.

2. The method of claim 1, wherein the verbalized acknowledgement comprises a synthesized utterance.

3. The method of claim 2, wherein the synthesized utterance is adapted to the emotion with at least one audible stylization, the at least one audible stylization being one or more of a cadence, a volume, a pitch, a word, a string of words, a pace, and a tone.

4. The method of claim 1, wherein the verbalized acknowledgement comprises a displayed text.

5. The method of claim 4, wherein the displayed text is adapted to the emotion with at least one visual stylization, the at least one visual stylization being one or more of a font type, a font size, a font color, capital letters, a number of words, and a punctuation mark.

6. The method of claim 1, wherein the non-textual emotion feature includes one or more of a cadence, a volume, a pitch, a pace, and a tone.

7. The method of claim 1, wherein the mapping is partially based on an identity of a user.

8. The method of claim 1, wherein a type of the verbalized acknowledgement is based on a type of the user device.

9. The method of claim 8,

wherein the user device comprises an automobile, and

wherein the type of the verbalized acknowledgement is a synthesized utterance.

10. The method of claim 1, wherein the verbalized acknowledgement comprises a reflector of the first or the second command associated with the selected emotion feature.

11. The method of claim 10, wherein the reflector comprises a conjugated form of a word in the first or the second command associated with the selected emotion feature.

12. The method of claim 1, wherein the textual emotion feature includes one or more of a word or a string of words from the text.

13. A non-transitory computer readable medium comprising:

an emotion processor having one or more sequences of emotion processor instructions that, when executed by one or more processors, causes the one or more processors to generate an output adapted to an emotion detected in a natural utterance;

and one or more sequences of instructions which, when executed by one or more processors, cause the one or processors to;

receive the natural utterance,

convert the natural utterance into text;

extract a textual emotion feature from the text;

extract a non-textual emotion feature from the natural utterance, the textual emotion feature and the non-textual emotion feature corresponding, respectively, to conflicting first and second commands;

select either the non-textual emotion feature or the textual emotion feature, thereby providing a selected emotion feature;

map the selected emotion feature to an emotion; and

respond to the natural utterance by providing a verbalized acknowledgement adapted to the emotion and performing an action corresponding to the first or the second command associated with the selected emotion feature.

14. A system comprising:

one or more processors configured to execute one or more sequences of computer-readable instructions that cause the one or more processors to;

convert a received natural utterance into text;

extract a textual emotion feature from the text;

extract a non-textual emotion feature from the natural utterance, the textual emotion feature and the non-textual emotion feature corresponding, respectively, to conflicting first and second commands;

select either the non-textual emotion feature or the textual emotion feature, thereby providing a selected emotion feature;

map the selected emotion feature to an emotion; and

respond to the natural utterance by providing a verbalized acknowledgement adapted to the emotion and performing an action corresponding to the first or the second command associated with the selected emotion feature.

15. The system of claim 14, wherein the verbalized acknowledgement comprises a synthesized utterance.

16. The system of claim 15, wherein the synthesized utterance is adapted to the emotion with at least one audible stylization, the at least one audible stylization being one or more of a cadence, a volume, a pitch, a word, a string of words, a pace, and a tone.

17. The system of claim 14, wherein the verbalized acknowledgement comprises a displayed text.

18. The system of claim 17, wherein the displayed text is adapted to the emotion with at least one visual stylization, the at least one visual stylization being one or more of a font type, a font size, a font color, capital letters, a number of words, and a punctuation mark.

19. The system of claim 14, wherein the non-textual emotion feature includes one or more of a cadence, a volume, a pitch, a pace, and a tone.

20. The system of claim 14, wherein the map the selected emotion feature to an emotion is partially based on an identity of a user.

21. The system of claim 14, wherein a type of the verbalized acknowledgement is based on a type of a user devices that receives the natural utterance.

22. The system of claim 21,

wherein the user device comprises an automobile, and

wherein the type of the verbalized acknowledgement is a synthesized utterance.

23. The system of claim 14, wherein the verbalized acknowledgement comprises a reflector of the first or the second command associated with the selected emotion feature.

24. The system of claim 23, wherein the reflector comprises a conjugated form of a word in the first or the second command associated with the selected emotion feature.

25. The system of claim 14, wherein the textual emotion feature includes one or more of a word or a string of words from the text.