Task assistant utilizing context for improved interaction

• US 10,223,411 B2
• Filed: 03/06/2013
• Issued: 03/05/2019
• Est. Priority Date: 03/06/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Mauro, David Andrew, Bouvier, Henri, Dykstra-Erickson, Elizabeth Ann, Gandrabur, Simona, Daniel, Susan Dawnstarr, Piercy, Aimee, Sharp, Robert Douglas
• Primary Examiner(s): Gortayo, Dangelino N
• Application Number:
  US13/787,763
• Publication Number:
  US 20140258324A1
• Time in Patent Office: 2,190 Days
• Field of Search: 707706, 707723, 707765, 707766, 707769, 707771, 707760, 704231, 704251, 704270, 704275
• CPC Class Codes: G06F 16/2423 Interactive query statement specification based on a database schema
• CPC Class Codes: G06F 16/3322 using system suggestions G06F16/3325 takes precedence
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 21/06 Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids G10L15/26 takes precedence

Abstract

A method of providing a task assistant is described. The task assistant is designed to receive input from a user through multimodal input including a plurality of speech input, typing input, and touch input, determine the meaning of the input, and determining whether there is a context based on prior interactions with the user. The method further to generate an interpreted input based on a combination of the input and the context, and providing a formatted query to an application. The method further to receive data from the application in response to the formatted query, and provide a response to the user through multimodal output including a plurality of: speech output, text output, non-speech audio output, haptic output, and visual non-text output. The method further to update the context based on the interpreted input.

Claims

1. A method comprising:

receiving input, at a first device, from a user through multi modal input including a plurality of speech input, typing input, and touch input;

determining a meaning of the input;

determining, based on prior interactions with the user, that there is context related to the input, wherein the context comprises information that can be used to interpret a request in the input;

generating, based on a combination of the input and the context, an interpreted input in a natural language format, wherein the interpreted input comprises a query;

translating the interpreted input from the natural language format to a format compatible with an application, to generate a formatted query;

providing the formatted query to the application;

receiving data from the application in response to the formatted query;

providing a response to the user through multi modal output including a plurality of;

speech output, text output, non-speech audio output, haptic output, and visual non-text output;

updating the context based on the interpreted input;

verifying, based on a connection type between the first device and a second device, that the first device is proximal to the second device; and

in response to the verifying that the first device is proximal to the second device, transmitting the context from the first device to the second device.

2. The method of claim 1, wherein the context comprises prior terms input by the user.

3. The method of claim 1, further comprising:

displaying the interpreted input; and

enabling a user to alter the interpreted input by unselecting one or more elements of the interpreted input.

4. The method of claim 1, further comprising:

determining that the interpreted input is ambiguous; and

in response to determining that the interpreted input is ambiguous, requesting clarification from the user.

5. The method of claim 1, wherein generating the interpreted input further comprises generating the interpreted input based on user data defining what information is available to the user.

6. The method of claim 1, wherein generating the interpreted input further comprises generating the interpreted input based on user history data of prior successful queries.

7. The method of claim 1, wherein transmitting the context from the first device to the second device comprises transmitting a log of successful queries from the first device to the second device.

8. The method of claim 7, wherein the first device and the second device may be selected from among:

a mobile device, a computer system, a kiosk, a web interface, an in-store system, a television, a digital watch, an automobile, or an interactive surface display.

9. The method of claim 1, wherein verifying that the first device is proximal to the second device comprises determining that the first device is in communication with the second device via a Bluetooth connection.

10. The method of claim 1, wherein the translating the interpreted input comprises translating the interpreted input to a command for the application.

11. The method of claim 10, wherein the formatted query comprises the command.

12. A method comprising:

receiving input, from a user at a first device, through multi modal input including a plurality of speech input, typing input, and touch input;

determining, based on prior interactions with the user, that there is context corresponding to the input, wherein the context comprises information that can be used to interpret a request in the input;

generating, based on a combination of the input and the context, an interpreted input in a natural language format, wherein the interpreted input comprises a query;

translating the interpreted input from the natural language format to a format compatible with an application, to generate a formatted query;

recording the formatted query in a query log;

passing the formatted query to the application;

receiving data from the application in response to the formatted query;

providing, based on the data, a response to the user through multi modal output including a plurality of;

speech output, text output, non-speech audio output, haptic output, and visual non-text output;

updating the context based on the interpreted input;

determining, based on a network connection type between the first device and a second device, that the first device is proximal to the second device; and

transferring, in response to the determination that the first device is proximal to the second device, the query log from the first device to the second device.

13. The method of claim 12, further comprising:

displaying the interpreted input; and

enabling a user to alter the interpreted input by unselecting one or more elements of the interpreted input.

14. The method of claim 12, further comprising:

determining that the interpreted input is ambiguous; and

requesting clarification from the user.

15. The method of claim 12, wherein transmitting the query log from the first device to the second device comprises transmitting the query log when the user moves from using the first device to using the second device.

16. The method of claim 15, wherein the first device and the second device may be selected from among:

a mobile device, a computer system, a kiosk, a web interface, an in-store system, a television, a digital watch, an automobile, and an interactive surface display.

17. A method comprising:

receiving input from a user, at a first device, through multi modal input including a plurality of speech input, typing input, and touch input;

determining, based on prior interactions with the user, that there is a first context related to the input, wherein the first context comprises information that can be used to interpret a request in the input;

generating, based on the input and the first context, an interpreted input in a natural language format, wherein the interpreted input comprises a query;

translating the interpreted input from the natural language format to a format compatible with an application, to generate a formatted query;

providing the formatted query to the application;

recording the formatted query in a query log;

receiving data from the application in response to the formatted query;

providing a response to the user through multi modal output including a plurality of;

speech output, text output, non-speech audio output, haptic output, and visual non-text output;

updating the first context based on the interpreted input;

determining that the user has logged in to an application at a second device;

verifying, by communicating over a local network, that the first device is proximal to the second device;

in response to the determining that the user has logged in to the application at the second device and the verifying that the first device is proximal to the second device, transmitting the query log from the first device to the second device; and

generating, at the second device and based on the query log, second context for a session of a task assistant.

18. The method of claim 17, wherein the first context comprises one or more of:

prior terms input by the user during a current session, user data defining what information is available to the user, and user history data including prior successful queries.

19. The method of claim 17, wherein transmitting the query log from the first device to the second device comprises transmitting the query log when the user moves from using the first device to the second device.

20. The method of claim 19, wherein the first device and the second device may be selected from among:

a mobile device, a computer system, a kiosk, a web interface, an in-store system, a television, a digital watch, an automobile, and an interactive surface display.