Task assistant having multiple visual displays

• US 10,795,528 B2
• Filed: 03/06/2013
• Issued: 10/06/2020
• Est. Priority Date: 03/06/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Dykstra-Erickson, Elizabeth Ann, Mauro, David Andrew, Attayadmawittaya, Paweena, Piercy, Aimee, Daniel, Susan Dawnstarr
• Primary Examiner(s): Dasgupta, Shourjo
• Application Number:
  US13/787,775
• Publication Number:
  US 20140258857A1
• Time in Patent Office: 2,771 Days
• Field of Search: None
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance

Abstract

A method of providing a task assistant to provide an interface to an application, the method comprising activating the task assistant, the activation having an associated visual display. The method in one embodiment includes receiving input from a user through multimodal input including a plurality of speech input, typing input, and touch input, interpreting the input, and providing a formatted query to the application, receiving data from the application in response to the query, and providing a response to the user through multimodal output including a plurality of: speech output, text output, non-speech audio output, haptic output, and visual non-text output, wherein the task assistant has a plurality of active states, each of the active states having an associated visual display.

Claims

1. A method comprising:

causing, by a computing device, display of a user interface associated with a task assistant;

receiving, by the computing device, based on one or more first user interactions with the user interface, a first user input;

determining, by the computing device and based on the first user input, a query;

generating, by the computing device and based on determining that a response page for the query does not exist, a template for enabling data entry associated with the query, wherein generating the template for enabling data entry associated with the query includes generating a temporary anchor to the template to navigate the user to the template when the response page for the query does not exist;

populating, by the computing device, the template with data corresponding to a user;

causing, by the computing device, display of the template;

receiving, by the computing device and based on one or more second user interactions with the template, a second user input that indicates a modification to the data corresponding to the user;

modifying, based on the second user input, the template, resulting in a modified template;

causing the modified template to be stored such that the modified template is retrievable based on a user request;

providing, by the computing device and to an application, data that is based on the query and the second user input;

receiving, by the computing device and from the application, response data; and

outputting, by the computing device and based on the response data, a response to the user.

2. The method of claim 1, wherein the response comprises a request to repeat a previous input.

3. The method of claim 1, wherein the response comprises a request for additional information to clarify the query.

4. The method of claim 1 further comprising retrieving the data corresponding to the user from account information corresponding to the user.

5. The method of claim 1 further comprising receiving, based on one or more third user interactions with the template, a request to store a bookmark corresponding to the template.

6. The method of claim 5 further comprising:

receiving a selection of the bookmark; and

causing, based on the selection of the bookmark, display the template.

7. The method of claim 5 further comprising:

generating an anchor point associated with the template; and

generating, based on the anchor point, the bookmark,

wherein causing display of the template is performed based on a navigation associated with the anchor point, and

wherein causing the modified template to be stored is performed based on the bookmark.

8. The method of claim 1, wherein the first user input includes voice data.

9. One or more non-transitory computer readable media comprising executable instructions that, when executed, cause an apparatus to perform operations comprising:

causing display of a user interface associated with a task assistant;

receiving, based on one or more first user interactions with the user interface, a first user input;

determining, based on the first user input, a query;

generating, based on the determining that a response page for the query does not exist, a template for enabling data entry associated with the query, wherein generating the template for enabling data entry associated with the query includes generating a temporary anchor to the template to navigate the user to the template when the response page for the query does not exist;

populating the template with data corresponding to a user;

causing display of the template;

receiving, based on one or more second user interactions with the template, a second user input that indicates a modification to the data corresponding to the user;

modifying, based on the second user input, the template, resulting in a modified template;

causing the modified template to be stored such that the modified template is retrievable based on a user request;

providing, to the application, data that is based on the query and the second user input;

receiving, from the application, response data; and

outputting, based on the response data, a response to the user, wherein the response includes one of voice output, text output, non-speech audio output, haptic output, and visual non-text output.

10. The one or more non-transitory computer readable media of claim 9, wherein the operations further comprise retrieving the data corresponding to the user from account information corresponding to the user.

11. The one or more non-transitory computer readable media of claim 9, wherein the operations further comprise receiving, based on one or more third user interactions with the template, a request to store a bookmark corresponding to the template.

12. The one or more non-transitory computer readable media of claim 11, wherein the operations further comprise:

receiving a selection of the bookmark; and

causing, based on the selection of the bookmark, display the template.

13. An apparatus comprising:

one or more processors; and

memory storing executable instructions that, when executed by the one or more processors, cause the apparatus to perform operations comprising;

causing display of a user interface associated with a task assistant;

receiving, based on one or more first user interactions with the user interface, a first user input;

determining, based on the first user input, a query;

generating, based on the determining that a response page for the query does not exist, a template for enabling data entry associated with the query, wherein generating the template for enabling data entry associated with the query includes generating a temporary anchor to the template to navigate the user to the template when the response page for the query does not exist;

populating the template with data corresponding to a user;

causing display of the template;

receiving, based on one or more second user interactions with the template, a second user input that indicates a modification to the data corresponding to the user;

modifying, based on the second user input, the template, resulting in a modified template;

causing the modified template to be stored such that the modified template is retrievable based on a user request;

providing, to the application, data that is based on the query and the second user input;

receiving, from the application, response data; and

outputting, based on the response data, a response to the user, wherein the response includes one of voice output, text output, non-speech audio output, haptic output, and visual non-text output.

14. The apparatus of claim 13, wherein the response comprises a request to repeat a previous input.

15. The apparatus of claim 13, wherein the response comprises a request for additional information to clarify the query.

16. The apparatus of claim 13, wherein the operations further comprise retrieving the data corresponding to the user from account information corresponding to the user.

17. The apparatus of claim 13, wherein the operations further comprise receiving based on one or more third user interactions with the template, a request to store a bookmark corresponding to the template.

18. The apparatus of claim 17, wherein the operations further comprise:

receiving a selection of the bookmark; and

causing, based on the selection of the bookmark, display the template.

19. The apparatus of claim 13, wherein the response comprises a request to repeat a previous input.

20. The apparatus of claim 13, wherein the response comprises a request for additional information to clarify the query.