Invoking an automated assistant to perform multiple tasks through an individual command

• US 10,552,204 B2
• Filed: 07/07/2017
• Issued: 02/04/2020
• Est. Priority Date: 07/07/2017
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Ni, Yuzhao, Schairer, David
• Primary Examiner(s): Wozniak, James S
• Application Number:
  US15/644,157
• Publication Number:
  US 20190012198A1
• Time in Patent Office: 942 Days
• Field of Search: 704257, 704270, 7042701, 704275
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 9/4831 with variable priority
• CPC Class Codes: G06F 9/4843 by program, e.g. task dispatcher, supervisor, operating system
• CPC Class Codes: G06Q 10/02 Reservations, e.g. for tickets, services or events
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command

Abstract

Methods, apparatus, systems, and computer-readable media for engaging an automated assistant to perform multiple tasks through a multitask command. The multitask command can be a command that, when provided by a user, causes the automated assistant to invoke multiple different agent modules for performing tasks to complete the multitask command. During execution of the multitask command, a user can provide input that can be used by one or more agent modules to perform their respective tasks. Furthermore, feedback from one or more agent modules can be used by the automated assistant to dynamically alter tasks in order to more effectively use resources available during completion of the multitask command.

Claims

1. A method implemented by one or more processors, the method comprising:

determining a correlation between a first spoken utterance and a second spoken utterance during previous interactions between a user and an automated assistant via an automated assistant interface,

wherein determining the correlation is based on a temporal relationship between the first spoken utterance and the second spoken utterance in each of the previous interactions,

wherein the first spoken utterance causes the automated assistant to initialize execution of multiple tasks performed by separate agent modules that include a first agent module and a second agent module, and

wherein the second spoken utterance causes the automated assistant to initialize execution of an additional task performed by an additional agent module;

prompting, based on determining the correlation between the first spoken utterance and the second spoken utterance, the user regarding a preconfigured multitask command,

wherein the preconfigured multitask command corresponds to the multiple tasks and the additional task, and

wherein the prompting is performed without an explicit request, from the user to the automated assistant, to initialize generation of the preconfigured multitask command; and

subsequent to prompting the user regarding the preconfigured multitask command;

determining that content of a natural language input provided to the automated assistant interface identifies the preconfigured multitask command;

invoking the first agent module of the separate agent modules to perform a first task of the multiple tasks and the additional agent module to perform the additional task;

in response to invoking the first agent module and the additional agent module, receiving a first agent output from the first agent module and an additional agent output from the additional agent module;

invoking, with parameters based on the first agent output, the second agent module of the separate agent modules to perform a second task of the multiple tasks, wherein the second task involves processing the parameters,

wherein invoking the second agent module with the parameters based on the first agent output includes incorporating at least some of the first agent output into one or more of the parameters and providing, by the automated assistant, the parameters to the second agent module;

in response to invoking the second agent module, receiving a second agent output from the second agent module; and

providing responsive output to the user, via the automated assistant interface, based on the first agent output, the second agent output, and the additional agent output.

2. The method of claim 1, wherein determining that the content of the natural language input includes the preconfigured multitask command comprises:

comparing the content to entries in an index that correlates stored commands to the preconfigured multitask command.

3. The method of claim 1, wherein determining the correlation between the first spoken utterance and the second spoken utterance includes:

determining that the temporal relationship between the first spoken utterance and the second spoken utterance indicates that the first spoken utterance and the second spoken utterance were provided within a threshold period of time.

4. The method of claim 1, wherein the responsive output corresponds to a query for the user and further comprising:

receiving, at the automated assistant interface, an input from the user in response to the query; and

causing the second agent module to process the input from the user.

5. The method of claim 4, wherein the second agent module provides feedback to the automated assistant in response to processing the input from the user, and the method further comprises:

in response to the feedback satisfying one or more criteria;

invoking, with parameters based on the feedback, a third agent module to perform a third task of the multiple tasks.

6. The method of claim 1, wherein determining the correlation between the first spoken utterance and the second spoken utterance is performed before the user has provided the natural language input that identifies the preconfigured multitask command.

7. The method of claim 1,

wherein the natural language input is a single spoken natural language input, and the content of the single spoken natural language input is different from other natural language content verbally provided by the user to the automated assistant during one or more of the previous interactions.

8. The method of claim 7,

wherein invoking the first agent module of the separate agent modules to perform the first task of the multiple tasks includes communicating with a remote server that is separate from a client device that includes the automated assistant interface;

and

wherein invoking the second agent module of the separate agent modules to perform the second task of the multiple tasks includes communicating with a different remote server that is separate from the client device.

9. The method of claim 7, wherein the first agent module and the second agent module are both invoked at least partially in response to the automated assistant interface receiving the single spoken natural language input.

10. The method of claim 1, wherein the preconfigured multitask command is previously stored in response to the automated assistant previously generating the preconfigured multitask command.

11. A system, comprising:

one or more processors; and

memory storing instructions that, when executed by one or more of the processors, cause one or more of the processors to perform operations that include;

determining a correlation between a first spoken utterance and a second spoken utterance during previous interactions between a user and an automated assistant via an automated assistant interface,

wherein determining the correlation is based on a temporal relationship between the first spoken utterance and the second spoken utterance in each of the previous interactions,

wherein the first spoken utterance causes the automated assistant to initialize execution of multiple tasks performed by separate agent modules that include a first agent module and a second agent module, and

wherein the second spoken utterance causes the automated assistant to initialize execution of an additional task performed by an additional agent module;

prompting, based on determining the correlation between the first spoken utterance and the second spoken utterance, the user regarding a preconfigured multitask command,

wherein the preconfigured multitask command corresponds to the multiple tasks and the additional task, and

wherein the prompting is performed without an explicit request, from the user to the automated assistant, to initialize generation of the preconfigured multitask command; and

subsequent to prompting the user regarding the preconfigured multitask command;

determining that content of a natural language input provided to the automated assistant interface identifies the preconfigured multitask command;

invoking the first agent module of the separate agent modules to perform a first task of the multiple tasks and the additional agent module to perform the additional task;

in response to invoking the first agent module and the additional agent module, receiving a first agent output from the first agent module and an additional agent output from the additional agent module;

invoking, with parameters based on the first agent output, the second agent module of the separate agent modules to perform a second task of the multiple tasks, wherein the second task involves processing the parameters,

wherein invoking the second agent module with the parameters based on the first agent output includes incorporating at least some of the first agent output into one or more of the parameters and providing, by the automated assistant, the parameters to the second agent module;

in response to invoking the second agent module, receiving a second agent output from the second agent module; and

providing responsive output to the user, via the automated assistant interface, based on the first agent output, the second agent output, and the additional agent output.

12. The system of claim 11, wherein the first task or the second task is associated with booking a reservation for the user.

13. The system of claim 11, wherein determining the correlation between the first spoken utterance and the second spoken utterance includes:

determining that the temporal relationship between the first spoken utterance and the second spoken utterance indicates that the first spoken utterance and the second spoken utterance were provided within a threshold period of time.

14. The system of claim 11, wherein the operations further include:

providing, via the automated assistant interface, a query to the user regarding the output; and

receiving a subsequent user input from the user, wherein one or more of the parameters provided to the second agent module are based at least in part on the subsequent user input received from the user.

15. The system of claim 11, wherein determining the correlation between the first spoken utterance and the second spoken utterance is performed before the user has provided the natural language input that identifies the preconfigured multitask command.

16. A non-transitory computer readable medium configured to store instructions that, when executed by one or more processors, cause the one or more processors to perform operations that include:

determining a correlation between a first spoken utterance and a second spoken utterance during previous interactions between a user and an automated assistant via an automated assistant interface,

wherein determining the correlation is based on a temporal relationship between the first spoken utterance and the second spoken utterance in each of the previous interactions,

wherein the first spoken utterance causes the automated assistant to initialize execution of multiple tasks performed by separate agent modules that include a first agent module and a second agent module, and

wherein the second spoken utterance causes the automated assistant to initialize execution of an additional task performed by an additional agent module;

prompting, based on determining the correlation between the first spoken utterance and the second spoken utterance, the user regarding a preconfigured multitask command,

wherein the preconfigured multitask command corresponds to the multiple tasks and the additional task, and

wherein the prompting is performed without an explicit request, from the user to the automated assistant, to initialize generation of the preconfigured multitask command; and

subsequent to prompting the user regarding the preconfigured multitask command;

determining that content of a natural language input provided to the automated assistant interface identifies the preconfigured multitask command;

invoking the first agent module of the separate agent modules to perform a first task of the multiple tasks and the additional agent module to perform the additional task;

in response to invoking the first agent module and the additional agent module, receiving a first agent output from the first agent module and an additional agent output from the additional agent module;

invoking, with parameters based on the first agent output, the second agent module of the separate agent modules to perform a second task of the multiple tasks, wherein the second task involves processing the parameters,

wherein invoking the second agent module with the parameters based on the first agent output includes incorporating at least some of the first agent output into one or more of the parameters and providing, by the automated assistant, the parameters to the second agent module;

in response to invoking the second agent module, receiving a second agent output from the second agent module; and

providing responsive output to the user, via the automated assistant interface, based on the first agent output, the second agent output, and the additional agent output.

17. The non-transitory computer readable medium of claim 16, wherein determining that the content of the natural language input includes the preconfigured multitask command comprises:

comparing the content to entries in an index that correlates stored commands to the preconfigured multitask command.

18. The non-transitory computer readable medium of claim 16, wherein each of the first agent module and the second agent module are third-party agent modules controlled by separate third-parties that are unique from a party that controls the automated assistant.

19. The non-transitory computer readable medium of claim 16, wherein the operations further include:

identifying, based on the content of the natural language input, the first agent module and the second agent module via an index that correlates the first agent module and the second agent module with the multiple tasks.

20. The non-transitory computer readable medium of claim 16, wherein the multiple tasks are associated with individual phrases previously provided by the user and stored by the automated assistant.