Building multimodal collaborative dialogs with task frames

• US 10,114,676 B2
• Filed: 05/05/2015
• Issued: 10/30/2018
• Est. Priority Date: 05/05/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Bikkula, Ravi, Panic, Danko, Crook, Paul, Khan, Omar Zia, Sarikaya, Ruhi, Suzuki, Hisami
• Primary Examiner(s): Zhao, Bing
• Application Number:
  US14/704,564
• Publication Number:
  US 20160328270A1
• Time in Patent Office: 1,274 Days
• Field of Search: None
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: G06F 9/5011 the resources being hardware resources other than CPUs, Servers and Terminals

Abstract

Methods and systems are provided for collaborative completion of tasks using task frames. Upon receiving a request to perform a task, a system utilizes task frames in completing the requested task. A task frame is a data structure that contains the parameters and status signals that represent a particular task and captures the combined system'"'"'s understanding of a current state of the task. Input is received at a client device and sent to a server, where the input is processed. Based on the processed input, a task frame is retrieved and filled. The filled task frame is sent to the client device, where the client device performs actions based on the task frame and updates the task frame parameters and the state of the task. The updated task frame is returned to the server. The shared task frame provides improvements to the overall task completion process.

Claims

1. A system comprising:

at least one processor; and

memory communicatively coupled to the at least one processor, encoding computer executable instructions that, when executed by the at least one processor perform a method, the method comprising;

receiving initial input at a client, wherein the input requests a digital assistant application to perform a task;

sending the initial input to a remote service;

receiving, by the client from the remote service, a predefined task frame to serve as a master reference for completing the task, wherein the task frame is a non-graphical-user-interface (GUI) data structure including a value for a status of the task and two or more required parameters to complete the task, wherein the task frame includes a name and a value for each of the two or more required parameters;

based on the task frame, determining, by the client, a next action to complete the task;

performing, by the client, the next action to complete the task;

based on performance of the task, updating, by the client, the task frame by updating one or more values for at least one of the two or more required parameters to create an updated task frame;

sending, by the client, the updated task frame to the remote service; and

receiving, by the client, a further updated version of the task frame from the remote service, wherein at least one of a value for a task frame parameter and a status of the task frame in the further updated task frame has been updated by the remote service.

2. The system of claim 1, wherein the method further comprises:

prompting a user for additional input based on the task frame;

receiving the additional input; and

wherein updating the task frame comprises;

updating one or more values for one or more task frame parameters based on the received additional input; and

updating the status of the task frame.

3. The system of claim 2, wherein the two or more task frame parameters each have a value indicating a source from which the parameter was received and a value indicating a state of the task frame parameter.

4. The system of claim 3, wherein the source of the initial input is of a first source type and the source of the additional input is of a second source type, the first and second source types comprise at least one of spoken, touch, gesture, and text.

5. The system of claim 3, wherein the method further comprises, for each of the updated one or more task frame parameters, updating the state of the task frame parameter.

6. The system of claim 1, wherein the status of the tasks indicates at least one of the following:

the task is in progress, the task is completed, and the task is canceled.

7. The system of claim 1, wherein the method further comprises processing the further updated task frame updated by the service to determine if additional input is required;

and updating the status of the task frame.

8. A computer-implemented method comprising:

receiving initial input at a client, wherein the input requests a digital assistant application to perform a task;

sending the initial input to a remote service;

receiving, by the client from the remote service, a predefined task frame to serve as a master reference for completing the task, wherein the task frame is a non-GUI data structure having one or more fields for at least one value for a status of the task and two or more required parameters to complete the task, wherein the task frame includes a name and a value for each of the two or more required parameters;

based on the task frame, determining a next action for completing the task;

performing the next action to complete the task;

based on performance of the task, updating the task frame by updating one or more values for at least one of the one or more required parameters to create an updated task frame;

sending the updated task frame to the remote service; and

receiving, by the client, a further updated version of the task frame from the remote service, wherein at least one of a value for a task frame parameter and a status of the task frame in the further updated task frame has been updated by the remote service.

9. The method of claim 8, wherein the method further comprises:

prompting a user for additional input;

receiving the additional input; and

wherein updating the task frame comprises;

updating one or more values for one or more task frame parameters based on the received additional input; and

updating the status of the task frame.

10. The method of claim 9, wherein the one or more task frame parameters each have a value indicating a source from which the parameter was received and a value indicating a state of the task frame parameter.

11. The method of claim 10, wherein the source of the initial input is of a first source type and the source of the additional input is of a second source type, the first and second source types comprise at least one of spoken, touch, gesture, and text.

12. The system of claim 10, wherein the method further comprises for each of the updated one or more task frame parameters, updating the state of the task frame parameter.

13. The method of claim 9, wherein the method further comprises:

processing the task frame updated by the service to determine if additional input is required; and

updating the status of the task frame.

14. The method of claim 8, wherein the task frame includes all required parameters necessary to complete the task.

15. A system comprising:

at least one processor; and

memory communicatively coupled to the at least one processor, encoding computer executable instructions that, when executed by the at least one processor perform a method, the method comprising;

receiving, at a service operating on one or more servers, input from a remote client;

processing the input to determine a requested task;

retrieving, from a plurality of predefined task frames, a predefined task frame to serve as a master reference for the requested task, wherein the predefined task frame is a non-GUI data structure that includes two or more parameters required for completing the requested task and a status for the two or more parameters, wherein the task frame includes a name and a value for each of the one or more required parameters;

processing the input to determine one or more values for the one or more task frame parameters required for completing the requested task;

updating the task frame with the determined parameter values; and

sending the task frame to the client; and

receiving an updated version of the task frame from the client, wherein at least one of a value for a task frame parameter and a status of the task frame in the updated task frame has been updated by the client.

16. The system of claim 15, the method further comprising:

receiving additional input from the client;

processing the additional input from the client to determine one or more values for one or more parameters of the updated task frame;

updating the one or more values for the one or more parameters of the updated task frame; and

sending the updated task frame back to the client.

17. The system of claim 15, wherein the method further comprises:

determining if additional user input is required to complete the task; and

updating the task frame to indicate that the client is to prompt a user for additional input.

18. The system of claim 15, wherein the parameters in the task frame are used as a reference for any components in the system utilized in completing the task.

19. The system of claim 16, wherein one or more task frame parameters each have a value indicating the source from which the parameter was received and a value indicating the state of the parameter.

20. The system of claim 16, wherein the task frame comprises all required parameters necessary to complete the task.