Leveraging user signals for improved interactions with digital personal assistant

• US 9,807,559 B2
• Filed: 06/25/2014
• Issued: 10/31/2017
• Est. Priority Date: 06/25/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Clark, Lee Dicks, Harrison, Deborah B., Hendrich, Susan, Gardner, David, Malekzadeh, Sogol, Maritan, Catherine L., Lim, Melissa, Czerwinski, Mary P., Gilad-Bachrach, Ran
• Primary Examiner(s): Arevalo, Joseph
• Application Number:
  US14/315,049
• Publication Number:
  US 20150382147A1
• Time in Patent Office: 1,224 Days
• Field of Search: 455411, 455446, 455436, 455407, 455406, 455405, 455517, 455 412, 704232
• CPC Class Codes: H04M 1/72454 according to context-related or environment-related conditions
• CPC Class Codes: H04M 3/42365 Presence services providing information on the willingness to communicate or the ability to communicate in terms of media capability or network connectivity
• CPC Class Codes: H04W 4/025 using location based information parameters
• CPC Class Codes: H04W 4/16 Communication-related supplementary services, e.g. call-transfer or call-hold
• CPC Class Codes: H04W 4/18 Information format or content conversion, e.g. adaptation by the network of the transmitted or received information for the purpose of wireless delivery to users or terminals
• CPC Class Codes: H04W 8/18 Processing of user or subscriber data, e.g. subscribed services, user preferences or user profiles; Transfer of user or subscriber data

Abstract

Systems, methods, apparatuses, and computer program products are described for implementing a digital personal assistant. The digital personal assistant is capable of determining that a user has asked a question or made a statement that is intended to engage with a persona of the digital personal assistant. In response to determining that the user has asked such a question or made such a statement, the digital personal assistant provides a response thereto by displaying or playing back a multimedia object associated with a popular culture reference within or by a user interface of the digital personal assistant. Additionally or alternatively, in response to determining that the user has asked such a question or made such a statement, the digital personal assistant provides the response thereto by generating or playing back speech that comprises an impersonation of a voice of a person associated with the popular culture reference.

Claims

1. A method performed by a digital personal assistant comprising a software agent implemented on at least one computing device, comprising:

obtaining one or more first signals related to an availability of a user;

obtaining one or more second signals related to a mental or emotional state of the user;

determining whether a particular time is an appropriate time to attempt to initiate a conversation between the software agent and the user based at least on the first signal(s) and the second signal(s); and

in response to a determination that the particular time is an appropriate time to attempt to initiate the conversation between the software agent and the user;

querying the user to determine if he or she is available to converse; and

in response to receiving a positive response to the query, selecting a conversation topic, and initiating a conversation between the software agent and the user about the selected conversation topic.

2. The method of claim 1, wherein the first signal(s) comprise one or more of:

calendar information associated with the user;

daily habits information associated with the user; and

information associated with a current activity of the user.

3. The method of claim 1, wherein the second signal(s) comprise one or more of:

facial expressions of the user;

voice characteristics of the user;

a location of the user;

a rate at which the user is turning on and off a mobile device;

keystroke and/or gesture metadata associated with the user;

written and/or spoken content of the user;

application interaction metadata associated with the user;

accelerometer, compass, and/or gyroscope output;

degree of exposure to light;

temperature;

weather conditions;

traffic conditions;

pollution and/or allergen levels;

activity level of the user;

heart rate and heart rate variability of the user;

electrodermal activity of the user;

device and/or network connection information for a device associated with the user; and

battery and/or charging information for a device associated with the user.

4. The method of claim 1, wherein the second signal(s) comprise one or more signals identified by a machine learner as being determinative of the mental or emotional state of the user.

5. The method of claim 1, wherein the machine learner is trained by one or more of a test population and the user.

6. The method of claim 1, wherein selecting the conversation topic comprises:

selecting the conversation topic based on one or more of a set of current events and a set of interests of the user.

7. The method of claim 6, wherein the set of current events are stored in one or more first databases, and wherein the set of interests of the user are stored in one or more second databases.

8. A system, comprising:

at least one processor; and

a memory that stores computer program logic for execution by the at least one processor, the computer program logic including one or more components configured to perform operations when executed by the at least one processor, the one or more components including;

a digital personal assistant comprising a software agent configured to obtain one or more signals related to an availability of a user, to obtain one or more second signals related to a mental or emotional state of the user, to determine whether a particular time is an appropriate time to attempt to initiate a conversation between the software agent and the user based at least on the first signal(s) and the second signal(s), and in response to a determination that the particular time is an appropriate time to attempt to initiate the conversation between the software agent and the user;

query the user to determine if he or she is available to converse; and

in response to receiving a positive response to the query, select a conversation topic, and initiate a conversation with the user about the selected conversation topic.

9. The system of claim 8, wherein the first signal(s) comprise one or more of:

calendar information associated with the user;

daily habits information associated with the user; and

information associated with a current activity of the user.

10. The system of claim 8, wherein the second signal(s) comprise one or more of:

facial expressions of the user;

voice characteristics of the user;

a location of the user;

a rate at which the user is turning on and off a mobile device;

keystroke and/or gesture metadata associated with the user;

written and/or spoken content of the user;

application interaction metadata associated with the user;

accelerometer, compass, and/or gyroscope output;

degree of exposure to light;

temperature;

weather conditions;

traffic conditions;

pollution and/or allergen levels;

activity level of the user;

heart rate and heart rate variability of the user;

electrodermal activity of the user;

device and/or network connection information for a device associated with the user; and

battery and/or charging information for a device associated with the user.

11. The system of claim 8, wherein the second signal(s) comprise one or more signals identified by a machine learner as being determinative of the mental or emotional state of the user.

12. The system of claim 8, wherein the machine learner is trained by one or more of a test population and the user.

13. The system of claim 8, wherein the software agent is configured to select the conversation topic based on one or more of a set of current events and a set of interests of the user.

14. The system of claim 13, wherein the set of current events are stored in one or more first databases, and wherein the set of interests of the user are stored in one or more second databases.

15. A computer program product comprising a computer-readable memory device having computer program logic recorded thereon that when executed by at least one processor causes the at least one processor to perform a method of operating a digital personal assistant comprising a software agent, the method comprising:

obtaining from a mobile device operated by a user one or more first signals related to an availability of the user;

obtaining from the mobile device one or more second signals related to a mental or emotional state of the user;

determining whether a particular time is an appropriate time to attempt to initiate a conversation between the software agent and the user based at least on the first signal(s) and the second signal(s); and

in response to a determination that the particular time is an appropriate time to attempt to initiate the conversation between the software agent and the user;

querying the user to determine if he or she is available to converse; and

in response to receiving a positive response to the query, selecting a conversation topic, and initiating a conversation between the software agent and the user about the selected conversation topic.

16. The computer program product of claim 15, wherein the first signal(s) comprise one or more of:

calendar information associated with the user;

daily habits information associated with the user; and

information associated with a current activity of the user.

17. The computer program product of claim 15, wherein the second signal(s) comprise one or more of:

facial expressions of the user;

voice characteristics of the user;

a location of the user;

a rate at which the user is turning on and off a mobile device;

keystroke and/or gesture metadata associated with the user;

written and/or spoken content of the user;

application interaction metadata associated with the user;

accelerometer, compass, and/or gyroscope output;

degree of exposure to light;

temperature;

weather conditions;

traffic conditions;

pollution and/or allergen levels;

activity level of the user;

heart rate and heart rate variability of the user;

electrodermal activity of the user;

device and/or network connection information for a device associated with the user; and

battery and/or charging information for a device associated with the user.

18. The computer program product of claim 15, wherein the second signal(s) comprise one or more signals identified by a machine learner as being determinative of the mental or emotional state of the user.

19. The computer program product of claim 15, wherein the machine learner is trained by one or more of a test population and the user.

20. The computer program product of claim 15, wherein selecting the conversation topic comprises selecting the conversation topic based on one or more of a set of current events and a set of interests of the user.