INTELLIGENT DEVICE USER INTERACTIONS

• US 20180260680A1
• Filed: 05/15/2018
• Published: 09/13/2018
• Est. Priority Date: 02/14/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: FINKELSTEIN, Erich-Soren, CALDWELL, Gary L.
• Granted Patent: US 11,100,384 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/006 based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G10L 15/08 Speech classification or search
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/228 of application context

Abstract

Intelligent assistant devices and methods for interacting with a user are disclosed. In some examples, a method for interacting with a user comprises predicting suggested action(s) for the user and displaying the action(s) via a display of the device. While the suggested action(s) are displayed, audio input comprising a command followed by a keyword is received from the user. The audio input is processed locally on the intelligent assistance device to (1) determine that the keyword follows the command and (2) recognize that the command applies to the suggested action(s). Based on determining that the keyword follows the command and recognizing that the command applies to the suggested action(s), a user selection of the suggested action(s) is established. Based on establishing the user selection, the one or more suggested actions are executed.

Claims

1. At an intelligent assistance device, a method for interacting with a user, the method comprising:

predicting one or more suggested actions for the user;

displaying the one or more suggested actions via a display of the intelligent assistance device;

while the one or more suggested actions are displayed, receiving audio input from the user comprising a command followed by a keyword;

locally processing the audio input on the intelligent assistance device to (1) determine that the keyword follows the command and (2) recognize that the command applies to the one or more suggested actions;

based on (1) determining that the keyword follows the command and (2) recognizing that the command applies to the one or more suggested actions, establishing a user selection of the one or more suggested actions; and

based on establishing the user selection, executing the one or more suggested actions.

2. The method of claim 1, wherein establishing the user selection of the one or more suggested actions is also based on receiving the audio input from the user while the one or more suggested actions are displayed.

3. The method of claim 1, wherein the one or more suggested actions comprise a plurality of suggested actions, the method further comprising:

locally processing the audio input on the intelligent assistance device to recognize that the command applies to a subset of the plurality of suggested actions;

based on recognizing that the command applies to the subset of the suggested actions, establishing a user selection of the subset of the suggested actions; and

based on establishing the user selection of the subset of the suggested actions, executing the subset of the suggested actions.

4. The method of claim 1, further comprising:

determining that a distance of the user from the intelligent assistant device is a first distance;

based on determining that the distance is the first distance, displaying a depiction of the one or more suggested actions;

determining that the distance of the user from the intelligent assistant device is a second distance; and

based on determining that the distance is the second distance, modifying the depiction of the one or more suggested actions.

5. The method of claim 4, wherein modifying the depiction of the one or more suggested actions comprises enlarging or shrinking the depiction.

6. The method of claim 4, wherein the depiction of the one or more suggested actions at the first distance comprises one or more words, and modifying the depiction of the one or more suggested actions comprises changing the one or more words to non-textual imagery.

7. The method of claim 4, wherein the depiction comprises a plurality of words at the first distance, and modifying the depiction comprises displaying fewer than all of the plurality of words at the second distance.

8. The method of claim 7, wherein modifying the depiction further comprises enlarging the fewer than all of the plurality of words at the second distance as compared to the first distance.

9. The method of claim 1, further comprising, upon determining that the keyword follows the command and recognizing that the command applies to the one or more suggested actions, proceeding to execute the one or more suggested actions without waiting to receive additional user input.

10. The method of claim 1, further comprising, based on determining that the keyword follows the command, electing to process the audio input locally on the intelligent assistance device to recognize that the command applies to the one or more suggested actions.

11. An intelligent assistant device configured to respond to natural language inputs, comprising:

a display;

a plurality of sensors comprising one or more microphones;

a logic machine; and

a storage machine holding instructions executable by the logic machine to;

predict one or more suggested actions for a user;

display the one or more suggested actions via the display;

while the one or more suggested actions are displayed, receive audio input from the user comprising a command followed by a keyword;

locally process the audio input on the intelligent assistance device to (1) determine that the keyword follows the command and (2) recognize that the command applies to the one or more suggested actions;

based on (1) determining that the keyword follows the command and (2) recognizing that the command applies to the one or more suggested actions, establish a user selection of the one or more suggested actions; and

based on establishing the user selection, execute the one or more suggested actions.

12. The intelligent assistant device of claim 11, wherein establishing the user selection of the one or more suggested actions is also based on receiving the audio input from the user while the one or more suggested actions are displayed.

13. The intelligent assistant device of claim 11, wherein the one or more suggested actions comprise a plurality of suggested actions, and the instructions are executable to:

locally process the audio input on the intelligent assistance device to recognize that the command applies to a subset of the plurality of suggested actions;

based on recognizing that the command applies to the subset of the suggested actions, establish a user selection of the subset of the suggested actions; and

based on establishing the user selection of the subset of the suggested actions, execute the subset of the suggested actions.

14. The intelligent assistant device of claim 11, wherein the instructions are executable to:

determine that a distance of the user from the intelligent assistant device is a first distance;

based on determining that the distance is the first distance, display a depiction of the one or more suggested actions;

determine that the distance of the user from the intelligent assistant device is a second distance; and

based on determining that the distance is the second distance, modify the depiction of the one or more suggested actions.

15. The intelligent assistant device of claim 14, wherein modifying the depiction of the one or more suggested actions comprises enlarging or shrinking the depiction.

16. The intelligent assistant device of claim 14, wherein the depiction of the one or more suggested actions at the first distance comprises one or more words, and modifying the depiction of the one or more suggested actions comprises changing the one or more words to non-textual imagery.

17. The intelligent assistant device of claim 14, wherein the depiction comprises a plurality of words at the first distance, and modifying the depiction comprises displaying fewer than all of the plurality of words at the second distance.

18. The intelligent assistant device of claim 17, wherein modifying the depiction further comprises enlarging the fewer than all of the plurality of words at the second distance as compared to the first distance.

19. The intelligent assistant device of claim 11, wherein the instructions are executable to, upon determining that the keyword follows the command and recognizing that the command applies to the one or more suggested actions, proceed to execute the one or more suggested actions without waiting to receive additional user input.

20. An intelligent assistant device configured to respond to natural language inputs, comprising:

a display;

a plurality of sensors comprising one or more microphones;

at least one speaker;

a logic machine; and

a storage machine holding instructions executable by the logic machine to;

predict one or more suggested actions for a user;

display the one or more suggested actions for a user via the display;

while the one or more suggested actions are displayed, receive audio input from the user comprising a command followed by a keyword;

locally process the audio input on the intelligent assistance device to (1) determine that the keyword follows the command and (2) recognize that the command applies to the one or more suggested actions;

based on (1) determining that the keyword follows the command, (2) recognizing that the command applies to the one or more suggested actions, and (3) receiving the audio input from the user while the one or more suggested actions are displayed, establish a user selection of the one or more suggested actions; and

based on establishing the user selection, execute the one or more suggested actions.