Scaling digital personal assistant agents across devices

• US 9,690,542 B2
• Filed: 12/22/2014
• Issued: 06/27/2017
• Est. Priority Date: 12/22/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Reddy, Mouni, Kannan, Vishwac Sena, Suver, Molly Rose, Uzelac, Aleksandar, Hwang, Daniel J., Stifelman, Lisa Joy
• Primary Examiner(s): Nabi, Reza
• Application Number:
  US14/580,093
• Publication Number:
  US 20160179464A1
• Time in Patent Office: 918 Days
• Field of Search: 715728
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

A method for providing digital personal assistant responses may include receiving, by a digital personal assistant associated with a plurality of reactive agents, a user input initiating a dialog with the digital personal assistant within the computing device. In response to receiving the input, an operation mode of the computing device may be detected from a plurality of available operation modes. One of the plurality of reactive agents can be selected based on the received input. A plurality of response strings associated with the selected reactive agent can be accessed. At least one of the plurality of response strings is selected based at least on the operation mode and at least one hardware characteristic of the computing device. The selected at least one of the plurality of response strings is providing during the dialog, as a response to the user input.

Claims

1. In a computing device with a digital personal assistant, a method for providing digital personal assistant responses, the method comprising:

receiving a user input initiating a dialog with the digital personal assistant within the computing device, the digital personal assistant associated with a plurality of reactive agents;

in response to receiving the input, detecting an operation mode of the computing device from a plurality of available operation modes;

selecting one of the plurality of reactive agents based on the received input;

accessing a plurality of response strings associated with the selected reactive agent;

selecting at least one of the plurality of response strings based at least on the operation mode and at least one hardware characteristic of the computing device;

providing during the dialog, the selected at least one of the plurality of response strings as a response to the user input;

detecting at least one peripheral device communicatively coupled to the computing device;

selecting at least a second one of the plurality of response strings based at least on the operation mode and at least one hardware characteristic of the at least one peripheral device; and

providing at the at least one peripheral device during the dialog, the selected at least a second one of the plurality of response strings.

2. The method according to claim 1, wherein the at least one hardware characteristic of the computing device comprises one or more of:

a display resolution, a display size, display availability, microphone availability, speaker availability, Bluetooth availability, and a device form factor associated with the computing device.

3. The method according to claim 1, wherein the operation mode is one of:

an audio only operation mode;

a semi-distracted operation mode;

a fully-engaged operation mode; and

a limited display operation mode.

4. The method according to claim 1, further comprising:

selecting a presentation mode for the at least one of the plurality of response strings based at least in part on the operation mode and the at least one hardware characteristic of the computing device; and

modifying the presentation mode selection based on at least one of;

the user'"'"'s distance from the computing device, a noise level in the vicinity of the computing device, and the user'"'"'s current availability for an active engagement with the computing device.

5. The method according to claim 4, wherein the presentation mode comprises at least one of:

an audio output of a text-to-speech representation of the selected at least one of the plurality of response strings;

a visual output of the selected at least one of the plurality of response strings; and

a tactile output associated with the selected at least one of the plurality of response strings.

6. The method according to claim 1, further comprising:

generating a user interface for presenting the selected at least one of the plurality of response strings, the user interface based at least in part on the at least one hardware characteristic of the computing device.

7. The method according to claim 1, wherein the user input is a voice input and the method further comprises:

detecting at least one requested functionality of the digital personal assistant based on the voice input; and

selecting the one of the plurality of reactive agents to perform the at least one requested functionality.

8. A computing device that includes a processor, a memory, and a microphone, the computing device being adapted to perform a method for providing digital personal assistant responses, the method comprising:

receiving, by a digital personal assistant, a digital voice input generated by a user, wherein the digital voice input is received via the microphone;

in response to receiving the digital voice input, detecting an operation mode and at least one hardware characteristic of the computing device;

selecting at least one of a plurality of response strings associated with a reactive agent for the digital personal assistant;

selecting a presentation mode for the at least one of the plurality of response strings, wherein the selecting of the at least one of the plurality of response strings and the presentation mode is based at least on the operation mode and the at least one hardware characteristic of the computing device;

providing the at least one of the plurality of response strings using the presentation mode as a response to the digital voice input;

detecting at least one peripheral device communicatively coupled to the computing device;

selecting at least a second one of the plurality of response strings based at least on the operation mode and at least one hardware characteristic of the at least one peripheral device; and

providing at the at least one peripheral device, the selected at least a second one of the plurality of response strings.

9. The computing device according to claim 8, wherein the at least one hardware characteristic of the computing device comprises one or more of:

a display resolution, a display size, display availability, microphone availability, speaker availability, Bluetooth availability, and a device form factor associated with the computing device.

10. The computing device according to claim 8, wherein the operation mode is an audio only operation mode and the method further comprises:

converting the at least one of the plurality of response strings to a voice response audio output using text-to-speech (TTS) conversion; and

providing the voice response audio output as the response to the digital voice input.

11. The computing device according to claim 8, wherein the operation mode is one of a semi-distracted operation mode and a fully-engaged operation mode, and the method further comprises:

converting the at least one of the plurality of response strings to a voice response audio output using text-to-speech (TTS) conversion; and

providing one or both of the voice response audio output and a display output of the at least one of the plurality of response strings as the response to the digital voice input.

12. The computing device according to claim 8, wherein the operation mode is a limited display operation mode and the method further comprises:

providing the at least one of the plurality of response strings on a display of the computing device, as the response to the digital voice input.

13. The computing device according to claim 8, wherein the method further comprises:

performing natural language processing using the digital voice input to determine a user voice request, wherein the user voice request is a request for initiating a first dialog with the digital personal assistant.

14. The computing device according to claim 13, wherein the method further comprises:

detecting another digital voice input for initiating a second dialog with the digital personal assistant;

upon initiating the second dialog, detecting a change in at least one of the operation mode and the at least one hardware characteristic of the computing device.

15. The computing device according to claim 14, wherein the method further comprises:

when at least one of the operation mode and the at least one hardware characteristic of the computing device has changed upon initiating the second dialog, updating a selection of the at least one of the plurality of response strings and the presentation mode based on the changed at least one of the operation mode and the at least one hardware characteristic of the computing device.

16. A computer-readable storage medium including instructions that upon execution cause a computer system to:

receive, by an application running on the computer system, a digital voice input generated by a user, wherein the digital voice input is received via the microphone;

perform natural language processing using the digital voice input to determine a user voice request, wherein;

the user voice request is associated with one or more functionalities performed by the application running on the computer system; and

the application comprises one or more reactive agents for performing the one or more functionalities;

select one of the one or more reactive agents for performing the one or more functionalities;

request by the application, information identifying an operation mode and at least one hardware characteristic of the computer system, the requesting using an API request;

in response to the API request, receive the requested information identifying the operation mode and the at least one hardware characteristic of the computer system;

select at least one of a plurality of response strings associated with the selected reactive agent;

select a presentation mode for the at least one of the plurality of response strings, wherein the selecting of the at least one of the plurality of response strings and the presentation mode is based at least on the operation mode and the at least one hardware characteristic of the computer system;

provide the at least one of the plurality of response strings using the presentation mode as a response to the user voice request;

detecting at least one peripheral device communicatively coupled to the computer system;

selecting at least a second one of the plurality of response strings based at least on the operation mode and at least one hardware characteristic of the at least one peripheral device; and

providing at the at least one peripheral device, the selected at least a second one of the plurality of response strings.

17. The computer-readable storage medium according to claim 16, wherein the instructions, upon execution further cause the computer system to:

provide the selected reactive agent with a plurality of native reactive agents within the digital personal assistant.

18. The computer-readable storage medium according to claim 16, wherein the operation mode is one of a semi-distracted operation mode and a fully-engaged operation mode, and wherein the instructions, upon execution further cause the computer system to:

convert the at least one of the plurality of response strings to a voice response audio output using text-to-speech (TTS) conversion; and

provide one or both of the voice response audio output and a display output of the at least one of the plurality of response strings as the response to the digital voice input.