Digital personal assistant interaction with impersonations and rich multimedia in responses

• US 9,514,748 B2
• Filed: 01/15/2014
• Issued: 12/06/2016
• Est. Priority Date: 01/15/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Reddy, Mouni, Howard, Robert J. III, Harrison, Deborah B., Malekzadeh, Sogol
• Primary Examiner(s): Hang, Vu B
• Application Number:
  US14/156,009
• Publication Number:
  US 20150199967A1
• Time in Patent Office: 1,056 Days
• Field of Search: 704/231, 704/232, 704/235, 704/237, 704/238, 704/239, 704/246, 704/249, 704/259, 704/270
• US Class Current: 1/1
• CPC Class Codes: G10L 13/02 Methods for producing synthetic speech; Speech synthesisers
• CPC Class Codes: G10L 13/08 Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
• CPC Class Codes: G10L 15/16 using artificial neural networks
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 25/30 using neural networks

Abstract

Systems, methods, apparatuses, and computer program products are described for implementing a digital personal assistant. The digital personal assistant is capable of determining that a user has asked a question or made a statement that is intended to engage with a persona of the digital personal assistant. In response to determining that the user has asked such a question or made such a statement, the digital personal assistant provides a response thereto by displaying or playing back a multimedia object associated with a popular culture reference within or by a user interface of the digital personal assistant. Additionally or alternatively, in response to determining that the user has asked such a question or made such a statement, the digital personal assistant provides the response thereto by generating or playing back speech that comprises an impersonation of a voice of a person associated with the popular culture reference.

Claims

1. A method for implementing a digital personal assistant, comprising:

receiving a digital representation of an utterance of a user of the digital personal assistant;

based on at least an analysis of the digital representation of the utterance, determining that the utterance comprises an utterance intended to engage with a persona of the digital personal assistant;

in response to at least the determining that the utterance comprises an utterance intended to engage with the persona of the digital personal assistant, identifying a plurality of eligible responses to the utterance, the plurality of eligible responses including;

at least one multimedia object associated with a popular culture reference; and

speech comprising an impersonation of a voice of a person associated with the popular culture reference; and

selecting one of the plurality of eligible responses and causing the selected response to be displayed when the selected response includes the at least one multimedia object or played back when the selected response includes the speech by the digital personal assistant.

2. The method of claim 1, wherein the selected response to the utterance includes text comprising a quotation associated with the popular culture reference.

3. The method of claim 1, wherein the speech comprises a quotation associated with the popular culture reference.

4. The method of claim 1, wherein the selected response includes a visual representation of the digital personal assistant that evokes the popular culture reference.

5. The method of claim 4, wherein the visual representation of the digital personal assistant comprises an animation of the digital personal assistant.

6. The method of claim 1, wherein the multimedia object comprises:

image or video content that is suitable for display within a user interface of the digital personal assistant or audio content that is suitable for playback by the user interface of the digital personal assistant.

7. The method of claim 1, wherein the selected response includes:

a link that is suitable for display within a user interface of the digital personal assistant and that can be activated by the user to access content associated with the popular culture reference.

8. The method of claim 1, wherein identifying the plurality of eligible responses to the utterance comprises:

matching the utterance to a particular utterance type within a hierarchical tree of utterance types, each utterance type in the hierarchical tree of utterance types having one or more responses associated therewith.

9. The method of claim 1, wherein identifying the plurality of eligible response to the utterance comprises:

determining that the utterance is associated with a trending topic; and

in response to determining that the utterance is associated with the trending topic, identifying one or more responses associated with the trending topic.

10. The method of claim 1, identifying the plurality of eligible responses to the utterance comprises:

determining that the utterance is an utterance for which there are one or more predefined responses intended to convey the persona of the digital personal assistant.

11. The method of claim 1, wherein causing the selected response to be displayed or played back by the digital personal assistant comprises:

sending an audio file that includes the speech or information that identifies the audio file to a computing device executing the digital personal assistant.

12. The method of claim 1, wherein causing the selected response to be displayed or played back by the digital personal assistant comprises:

providing text to a computing device executing the digital personal assistant, the text to be processed by a text-to-speech component of the digital personal assistant to generate the speech.

13. A computing device, comprising:

at least one processor; and

a memory that stores computer program logic for execution by the at least one processor, the computer program logic including one or more components configured to perform operations when executed by the at least one processor, the one or more components including;

a digital personal assistant configured to capture audio that represents an utterance of a user and to transmit the audio to a digital personal assistant backend, the utterance comprising an utterance intended to engage with a persona of the digital personal assistant;

the digital personal assistant being further configured to provide a response to the utterance based at least on information received from the digital personal assistant backend, providing the response including generating or playing back speech that comprises an impersonation of a voice of a person associated with a popular culture reference.

14. The computing device of claim 13, wherein the digital personal assistant is configured to play back an audio file that includes the speech.

15. The computing device of claim 13, wherein the digital personal assistant is configured to apply text-to-speech conversion to text to generate the speech.

16. The computing device of claim 13, wherein providing the response further includes at least one of:

displaying or playing back a multimedia object by a user interface of the digital personal assistant, the multimedia object being associated with the popular culture reference;

displaying text within the user interface of the digital personal assistant, the text comprising a quotation associated with the popular culture reference;

displaying a visual representation of the digital personal assistant that evokes the popular culture reference within the user interface of the digital personal assistant; and

displaying a link within the user interface of the digital personal assistant that can be activated by the user to access content associated with the popular culture reference.

17. A computing device, comprising:

at least one processor; and

a memory that stores computer program logic for execution by the at least one processor, the computer program logic including one or more components configured to perform operations when executed by the at least one processor, the one or more components including;

a digital personal assistant configured to capture audio that represents an utterance of a user and to transmit the audio to a digital personal assistant backend, the utterance comprising an utterance intended to engage with a persona of the digital personal assistant;

the digital personal assistant being further configured to;

based at least on information received from the digital personal assistant backend, provide a response to the utterance that is selected from among a plurality of eligible responses that includes at least one of text or audio and a multimedia object associated with a popular culture reference, providing the response including displaying the response for the text or the multimedia object or playing back the response for the audio by a user interface of the digital personal assistant.

18. The computing device of claim 17, wherein the multimedia object comprises an image, video content, or audio content.

19. The computing device of claim 17, wherein providing the response further includes at least one of:

generating or playing back speech that comprises an impersonation of a voice of a person associated with the popular culture reference;

displaying text within the user interface of the digital personal assistant, the text comprising a quotation associated with the popular culture reference;

displaying a visual representation of the digital personal assistant that evokes the popular culture reference within the user interface of the digital personal assistant; and

displaying a link within the user interface of the digital personal assistant that can be activated by the user to access content associated with the popular culture reference.