Intent driven command processing

• US 10,235,130 B2
• Filed: 08/14/2015
• Issued: 03/19/2019
• Est. Priority Date: 11/06/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Bai, Annie Y., Seto, Julie C., Yang, Chenguang, Fullmer, James R., Emmons, Julianne M., Morley, Becky M.
• Primary Examiner(s): Vu, Toan H
• Application Number:
  US14/826,968
• Publication Number:
  US 20160132291A1
• Time in Patent Office: 1,313 Days
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06F 9/453 Help systems
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/228 of application context

Abstract

A computing device receives a voice command to perform an action within a document. An interpretation of the voice command is mapped to a set of commands. Disambiguation is automatically performed by conducting a user experience to receive additional information.

Claims

1. A computing system, comprising:

a processor; and

memory storing instructions executable by the processor, wherein the instructions, when executed, configure the computing system to provide;

a user interface mechanism configured to detect a user utterance;

a select component configured to detect user selection of a portion of a document being displayed;

a command set processing component configured to;

receive an interpretation of the utterance;

based on the interpretation of the utterance and based on the selected portion of the document,

aggregate a plurality of different commands into a group of commands; and

surface a user actuatable input mechanism that collectively represents the group of commands; and

an action generator configured to;

in response to a detected user actuation of the user actuatable input mechanism that collectively represents the group of commands, control an application corresponding to the document to perform, in bulk, the plurality of different commands on the selected portion of the document.

2. The computing system of claim 1 wherein the select component is configured to:

determine whether the user selection is ambiguous and, if so, control the user interface mechanism to conduct a user experience to disambiguate the user selection to identify the selected portion of the document.

3. The computing system of claim 2 wherein the select component is configured to:

conduct the user experience by controlling the user interface mechanism to display an estimate of the selected portion with a set of user input mechanisms that are actuatable to change the selected portion, and a confirmation user input mechanism that is actuatable to confirm the selected portion.

4. The computing system of claim 1 wherein the command set processing component comprises:

a command analysis component is configured to determine whether the group of commands includes a given command that the user is to be informed of and, if so, displays a message identifying the given command to the user for user confirmation, prior to automatically executing the group of commands.

5. The computing system of claim 4 wherein the action generator, in response to detecting user confirmation of the given command, is configured to automatically execute the group of commands on the selected portion of the document.

6. The computing system of claim 5 wherein the action generator is configured to control the user interface mechanism to display a result of executing the group of commands on the selected portion of the document along with an undo actuator that is actuatable to undo the group of commands.

7. The computing system of claim 6 wherein the action generator is configured to control the user interface mechanism to display a result of executing the group of commands on the selected portion of the document along with a set of tweaking actuators, each of which is actuatable to modify the group of commands and execute the modified group of commands, and to display a result of executing the modified group of commands on the selected portion of the document.

8. The computing system of claim 4 wherein the selected portion of the document includes a set of objects and wherein the command analysis component is configured to determine whether all objects in the set of objects are of a same object type.

9. The computing system of claim 8 wherein the action generator is configured to:

if all objects in the set of objects are of the same type, execute the commands on the objects and control the user interface mechanism to surface a set of command user input mechanisms each actuatable to perform a command related to the set of commands, and a set of selection user input mechanisms, each actuatable to select other objects in the document on which commands are to be executed.

10. The computing system of claim 8 wherein the command analysis component is configured to:

if all objects in the set of objects are not the same type, determine whether the group of commands can all be executed on the objects in the set of objects and, if so, execute the group of commands on the set of objects.

11. The computing system of claim 8 wherein the command analysis component is configured to:

if all objects in the set of objects are not the same type, determine whether the group of commands can all be executed on the objects in the set of objects and, if not, control the user interface mechanism to conduct a multi-step user experience by displaying user input mechanisms actuatable to select different groups of objects in the set of the objects and execute commands on the selected groups of objects.

12. A computer implemented method, comprising:

detecting a user utterance;

detecting user selection of a portion of a document being displayed, the selected portion including a set of objects;

receiving, from a linguistic processing system, an interpretation and a textual representation of the utterance;

identifying and aggregating a plurality of commands into a group of commands, based on the interpretation of the utterance and based on the selected portion of the document;

controlling a user interface component to surface a user actuatable input mechanism, representing the group of commands;

detecting user actuation of the user actuatable input mechanism;

determining whether all objects in the set of objects are of a same object type;

if all objects in the set of objects are of the same object type,

executing the commands on the objects; and

controlling the user interface mechanism to surface a set of command user input mechanisms, each actuatable to perform a command related to the set of commands, and a set of selection user input mechanisms, each actuatable to select other objects in the document on which commands are to be executed; and

if all objects in the set of objects are not the same type,

determining whether the group of commands can all be executed on the objects in the set of objects and, if so, executing the group of commands on the set of objects.

13. The computer implemented method of claim 12 wherein detecting user selection of a portion of the document comprises:

determining whether the user selection is ambiguous; and

if so, controlling the user interface component to display an estimate of the selected portion with a set of user input mechanisms that are actuatable to change the selected portion, and a confirmation input that is actuatable to confirm the selected portion.

14. The computer implemented method of claim 12 wherein identifying and aggregating the plurality of commands comprises:

determining whether the group of commands includes a given command that the user is to be informed of;

if so, displaying a message identifying the given command to the user for user confirmation, prior to automatically executing the group of commands; and

in response to detecting user confirmation of the given command, automatically executing the group of commands on the selected portion of the document.

15. The computer implemented method of claim 14 wherein automatically executing comprises:

controlling the user interface component to display a result of executing the group of commands on the selected portion of the document along with an undo actuator that is actuatable to undo the group of commands.

16. The computer implemented method of claim 14 wherein automatically executing comprises:

controlling the user interface component to display a result of executing the group of commands on the selected portion of the document along with a set of tweaking actuators, each of which is actuatable to modify the group of commands and execute the modified group of commands;

detecting user actuation of a given tweaking actuator;

automatically executing the modified group of commands on the selected portion of the document; and

controlling the user interface component to display a result of executing the modified group of commands on the selected portion of the document.

17. The computer implemented method of claim 12 wherein, if all objects in the set of objects are not the same type, then further comprising:

determining whether the group of commands can all be executed on the objects in the set of objects; and

if not, controlling the user interface mechanism to conduct a multi-step user experience by displaying user input mechanisms actuatable to select different groups of objects in the set of the objects and execute commands on the selected groups of objects.

18. A mobile device, comprising:

a processor; and

memory storing instructions executable by the processor, wherein the instructions, when executed, configure the mobile device to;

detect a user utterance;

detect user selection of a portion of a document being displayed;

determine whether the user selection is ambiguous and, if so, display an estimate of the selected portion with a set of user input mechanisms that are actuatable to change the selected portion, and a confirmation input mechanism that is actuatable to confirm the selected portion;

receive an interpretation of the utterance;

aggregate a plurality of commands into a group of commands, based on the interpretation of the utterance and based on the selected portion of the document;

determine whether the group of commands includes a given command that the user is to be informed of and, if so, display a message identifying the given command to the user for user confirmation;

after display of the message, and in response to detecting user confirmation of the given command,

automatically execute the group of commands by controlling an application corresponding to the document to perform the group of commands on the selected portion of the document; and

display a result of executing the group of commands on the selected portion of the document along with at least one of a set of tweaking actuators, each of which is actuatable to modify the group of commands and an undo actuator that is actuatable to undo the group of commands.

19. The mobile device of claim 18 wherein the instructions, when executed, configure the mobile device to:

determine whether the user selection is ambiguous and, if so, conduct a user experience to disambiguate the user selection to identify the selected portion of the document.

20. The mobile device of claim 19, wherein the instructions, when executed, configure the mobile device to:

conduct the user experience by displaying an estimate of the selected portion with a set of user input mechanisms that are actuatable to change the selected portion, and a confirmation user input mechanism that is actuatable to confirm the selected portion.