User-programmable automated assistant

• US 10,431,219 B2
• Filed: 10/03/2017
• Issued: 10/01/2019
• Est. Priority Date: 10/03/2017
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Danila, Mihai, Smither, Albry
• Primary Examiner(s): Chawan, Vijay B
• Application Number:
  US15/724,217
• Publication Number:
  US 20190103101A1
• Time in Patent Office: 728 Days
• Field of Search: 704257, 704246, 704247, 704251, 704252, 704270, 704275, 704 9, 715781, 715784, 715804, 715866, 715973, 717107, 717108, 717109
• CPC Class Codes: G06F 40/279 Recognition of textual entities
• CPC Class Codes: G06F 40/35 Discourse or dialogue representation
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/225 Feedback of the input speech
• CPC Class Codes: G10L 25/51 for comparison or discrimination

Abstract

Techniques described herein relate to allowing users to employ voice-based human-to-computer dialog to program automated assistants with customized routines, or “dialog routines,” that can later be invoked to accomplish task(s). In various implementations, a first free form natural language input—that identifies a command to be mapped to a task and slot(s) required to be filled with values to fulfill the task—may be received from a user. A dialog routine may be stored that includes a mapping between the command and the task, and which accepts, as input, value(s) to fill the slot(s). Subsequent free form natural language input may be received from the user to (i) invoke the dialog routine based on the mapping, and/or (ii) to identify value(s) to fill the slot(s). Data indicative of at least the value(s) may be transmitted to a remote computing device for fulfillment of the task.

Claims

1. A method implemented by an automated assistant executed by one or more processors, comprising:

receiving, from a user at one or more input components of a computing device, a first speech input, wherein the first speech input includes a command to perform a task;

performing semantic processing on first speech recognition output generated from the first speech input;

determining, based on the semantic processing, that the automated assistant is unable to interpret the command;

providing, to the user at one or more output components of the computing device, output that solicits clarification from the user about the command;

receiving, from the same user in response to the output, at one or more of the input components, a second speech input;

identifying, from second speech recognition output generated from the second speech input, one or more slots that are required to be filled with values in order to fulfill the task;

storing a dialog routine that includes a mapping between the command and the task, and which accepts, as input, one or more values to fill the one or more slots;

receiving, at one or more of the input components, a third speech input, wherein the third speech input invokes the dialog routine based on the mapping;

identifying, based on third speech recognition output generated from the third speech input or additional speech input, one or more values to be used to fill the one or more slots that are required to be filled with values in order to fulfill the task; and

transmitting, to a remote computing device, data that is indicative of at least the one or more values to be used to fill the one or more slots, wherein the transmitting causes the remote computing device to fulfill the task.

2. The method of claim 1, further comprising:

comparing the dialog routine to a plurality of candidate tasks that are performable by the automated assistant; and

based on the comparing, selecting the task to which the command is mapped from the plurality of candidate tasks.

3. The method of claim 2, wherein the task to which the command is mapped comprises a third-party agent task, wherein the transmitting causes the remote computing device to perform the third-party agent task using the one or more values to fill the one or more slots.

4. The method of claim 2, wherein the comparing comprises comparing the one or more slots that are required to be filled in order to fulfill the task with one or more slots associated with each of the plurality of candidate tasks.

5. The method of claim 2, further comprising receiving, at one or more of the input components prior to the storing, a fourth speech input from the user, wherein the fourth speech input includes a user-provided enumerated list of possible values to fill one or more of the slots, and wherein the comparing comprises, for each of the plurality of candidate tasks, comparing the user-provided enumerated list of possible values to an enumerated list of possible values for filling one or more slots of the candidate task.

6. The method of claim 1, wherein the data that is indicative of at least the one or more values and that is transmitted to the remote computing device further includes one or both of an indication of the command or the task to which the command is mapped.

7. The method of claim 6, wherein the data that is indicative of at least the one or more values takes the form of natural language output that requests performance of the task based on the one or more values, and the transmitting causes the remote computing device to audibly or visually render, to a second user, the natural language output.

8. A method implemented by an automated assistant executed by one or more processors, comprising:

receiving, at one or more input components of a computing device, a first speech input from a user, wherein the first speech input;

identifying, from first speech recognition output generated from the first speech input, a command that the user intends to be mapped to a task, and one or more slots that are required to be filled with values in order to fulfill the task;

storing a dialog routine that includes a mapping between the command and the task, and which accepts, as input, one or more values to fill the one or more slots;

receiving, at one or more of the input components, a second speech input from the user, wherein the second speech input invokes the dialog routine based on the mapping;

identifying, based on second speech recognition output generated from the second speech input or additional speech input, one or more values to be used to fill the one or more slots that are required to be filled with values in order to fulfill the task; and

transmitting, to a remote computing device, data that is indicative of at least the one or more values to be used to fill the one or more slots, wherein the transmitting causes the remote computing device to fulfill the task.

9. The method of claim 8, further comprising:

comparing the dialog routine to a plurality of candidate tasks that are performable by the automated assistant; and

based on the comparing, selecting the task to which the command is mapped from the plurality of candidate tasks.

10. The method of claim 9, wherein the task to which the command is mapped comprises a third-party agent task, wherein the transmitting causes the remote computing device to perform the third-party agent task using the one or more values to fill the one or more slots.

11. The method of claim 9, wherein the comparing comprises comparing the one or more slots that are required to be filled in order to fulfill the task with one or more slots associated with each of the plurality of candidate tasks.

12. The method of claim 8, wherein the data that is indicative of at least the one or more values and that is transmitted to the remote computing device further includes one or both of an indication of the command or the task to which the command is mapped.

13. The method of claim 12, wherein the data that is indicative of at least the one or more values takes the form of natural language output that requests performance of the task based on the one or more values, and the transmitting causes the remote computing device to audibly or visually render, to a second user, the natural language output.

14. At least one non-transitory computer-readable medium comprising instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to perform the following operations:

receiving, from a user at one or more input components of a computing device, a first speech input, wherein the first speech input includes a command to perform a task;

performing semantic processing on first speech recognition output generated from the first speech input;

determining, based on the semantic processing, that the automated assistant is unable to interpret the command;

providing, at one or more output components of the computing device, output that solicits clarification from the user about the command;

receiving, from the same user at one or more of the input components, a second speech input;

identifying, based on second speech recognition output generated from the second speech input, one or more slots that are required to be filled with values in order to fulfill the task;

storing a dialog routine that includes a mapping between the command and the task, and which accepts, as input, one or more values to fill the one or more slots;

receiving, at one or more of the input components, a third speech input, wherein the third speech input invokes the dialog routine based on the mapping;

identifying, based on third speech recognition output generated from the third speech input or additional speech input, one or more values to be used to fill the one or more slots that are required to be filled with values in order to fulfill the task; and

transmitting, to a remote computing device, data that is indicative of at least the one or more values to be used to fill the one or more slots, wherein the transmitting causes the remote computing device to fulfill the task.

15. The non-transitory computer-readable medium of claim 14, further comprising instructions to perform the following operations:

comparing the dialog routine to a plurality of candidate tasks that are performable by the automated assistant; and

based on the comparing, selecting the task to which the command is mapped from the plurality of candidate tasks.

16. The non-transitory computer-readable medium of claim 15, wherein the task to which the command is mapped comprises a third-party agent task, wherein the transmitting causes the remote computing device to perform the third-party agent task using the one or more values to fill the one or more slots.

17. The non-transitory computer-readable medium of claim 15, wherein the comparing comprises comparing the one or more slots that are required to be filled in order to fulfill the task with one or more slots associated with each of the plurality of candidate tasks.

18. The non-transitory computer-readable medium of claim 15, further comprising instructions for receiving, at one or more of the input components prior to the storing, a fourth speech input from the user, wherein the fourth speech input includes a user-provided enumerated list of possible values to fill one or more of the slots, and wherein the comparing comprises, for each of the plurality of candidate tasks, comparing the user-provided enumerated list of possible values to an enumerated list of possible values for filling one or more slots of the candidate task.

19. The non-transitory computer-readable medium of claim 14, wherein the data that is indicative of at least the one or more values and that is transmitted to the remote computing device further includes one or both of an indication of the command or the task to which the command is mapped.

20. The non-transitory computer-readable medium of claim 19, wherein the data that is indicative of at least the one or more values takes the form of natural language output that requests performance of the task based on the one or more values, and the transmitting causes the remote computing device to audibly or visually render, to a second user, the natural language output.