Context-based command surfacing

• US 10,203,933 B2
• Filed: 08/14/2015
• Issued: 02/12/2019
• Est. Priority Date: 11/06/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Yang, Chenguang, Bai, Annie Y., Seto, Julie C., Fullmer, James R., Emmons, Julianne M.
• Primary Examiner(s): Titcomb, William D
• Application Number:
  US14/826,935
• Publication Number:
  US 20160132342A1
• Time in Patent Office: 1,278 Days
• Field of Search: 715708
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06F 9/453 Help systems
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/228 of application context

Abstract

A computing device receives a trigger to surface commands. A possible set of commands is identified and broken into categories. A category is surfaced for user interaction.

Claims

1. A computing system, comprising:

at least one processor; and

memory storing instructions executable by the at least one processor, wherein the instructions, when executed. configure the computing system to provide;

a user interface component configured to generate a user interface display for an open application associated with the computing system, the user interface display including a user interface mechanism configured to receive user input that manipulates content in the application;

a context detector configured to detect a content-independent context of the application, the content-independent context being independent of the content in the application;

a command set identifier configured to identify a set of possible commands in the application, based at least in part on the content-independent context; and

a command analysis component configured to;

categorize the possible commands into a set of command categories, each command category including a subset of the possible commands;

select a command category from the set of command categories;

based on the selection, control the user interface component to generate a user actuatable element, in the user interface display, that represents each command in the selected command category and is configured for user interaction; and

an action generator configured to;

receive an indication of user actuation of a particular one of the user actuatable elements; and

perform the command associated with the particular user actuatable element.

2. The computing system of claim 1 wherein the command analysis component is configured to control the user interface component to surface the selected command category for touch interaction on a touch sensitive display device associated with the computing system.

3. The computing system of claim 1, wherein the instructions, when executed, configure the computing system to provide:

a natural language user input mechanism configured to receive an indication of a natural language user input and obtain a command intent based on the natural language user input, the command set identifier identifying the set of possible commands based on the command intent.

4. The computing system of claim 1 wherein the set of possible commands is identified based on both the content-independent context and a content-dependent context that is based on the content.

5. The computing system of claim 4 wherein the content-dependent context is indicative of a location of a user, in the application.

6. The computing system of claim 4, wherein the content-dependent context is based on items currently displayed by the application.

7. The computing system of claim 1 wherein the command analysis component is configured to categorize the possible commands into a verb category, a subject category, and an action category.

8. The computing system of claim 1, wherein the content-independent context is based on at least one of:

determining whether a user is using the application in a work context or a non-work context;

detecting a usage patternindicative of a pattern of usage of the application, by the user, over time;

or

detecting a type of user device being used by the user.

9. The computing system of claim 1 wherein the command analysis component is configured to:

prior to the command being performed, determine whether to display another one of the command categories, and if so, identify the other command category based on a dependency structure and control the user interface mechanism to surface a user actuatable element that represents each command in the other command category.

10. A computing device, comprising:

a user interface mechanism;

a context detector configured to detect a context of an open application on;

a command set identifier configured to identify a set of possible commands in the application, based on the detected context; and

a command analysis component configured to;

categorize the possible commands in the set into a set of command categories including a verb category, a subject category, and an action category, each command category including a subset of the set of possible commands;

select a command category from the set of command categories; and

based on the selection, control the user interface mechanism to surface a user actuatable element that represents each command in the selected command category and is configured for user interaction;

an action generator configured to;

receive an indication of user actuation of a particular one of the user actuatable elements; and

perform the command associated with the particular user actuatable element using a processor of the mobile device.

11. The computing device of claim 10 wherein the context is based on a usage history of the application by a particular user.

12. The computing device of claim 10 wherein the context is indicative of a usage mode being used by the user.

13. The computing device of claim 10 wherein the context detector is configured to detect, as the context, items currently displayed by the application on the user interface mechanism and a location of the user in the application.

14. The computing device of claim 13 wherein the command analysis component is configured to categorize the possible commands into categories according to a dependency structure and to control the user interface mechanism to surface the categories according to the dependency structure.

15. The computing device of claim 14 wherein the command analysis component is configured to:

prior to the command being performed,

determine whether to display another one of the command categories, and

if so, identify the other command category based on the dependency structure and control the user interface mechanism to surface a user actuatable element that represents each command in the other command category.

16. A method performed by a computing device, the method comprising:

generating a user interface display for an open application associated with the computing device, the user interface display including a user interface mechanism configured to receive user input that manipulates content in the open application;

detecting a content-independent context of the application on the computing device, the content-independent context being independent of the content in the application;

identifying a set of possible commands in the application, based at least in part on the content-independent context;

categorizing the possible commands into a set of command categories, each command category including one or more of the possible commands;

selecting a command category from the set of command categories;

based on the selection, generating a user actuatable element that represents each command in the selected command category and is configured for user interaction;

receiving an indication of user actuation of a particular one of the user actuatable elements; and

performing the command associated with the particular user actuatable element.

17. The computer implemented method of claim 16 wherein detecting a context comprises:

detecting items currently displayed by the open application on the user interface mechanism and detecting a location of the user in the application.

18. The computer implemented method of claim 16 categorizing the possible commands comprises:

categorizing the possible commands into categories according to a dependency structure and wherein controlling the user interface mechanism comprises controlling the user interface mechanism to surface the categories according to the dependency structure.

19. The computer implemented method of claim 18, and further comprising:

prior to the command being performed,

determining whether to display another one of the command categories, and

if so, identifying the other command category based on the dependency structure and controlling the user interface mechanism to surface a user actuatable element that represents each command in the other command category.

20. The computer implemented method of claim 16 wherein receiving an indication of user actuation comprises:

receiving an indication of a user touch gesture on the particular user actuatable element.