Task initiation using long-tail voice commands by weighting strength of association of the tasks and their respective commands based on user feedback

• US 10,297,254 B2
• Filed: 10/03/2016
• Issued: 05/21/2019
• Est. Priority Date: 10/03/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Ni, Yuzhao, Wang, Bo, James, Barnaby, Gupta, Pravir, Schairer, David
• Primary Examiner(s): Kazeminezhad, Farzad
• Application Number:
  US15/284,473
• Publication Number:
  US 20180096681A1
• Time in Patent Office: 960 Days
• Field of Search: 704254, 704270, 707723
• CPC Class Codes: G06F 16/243 Natural language query formulation
• CPC Class Codes: G06F 16/3338 Query expansion
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/225 Feedback of the input speech
• CPC Class Codes: G10L 2015/226 using non-speech characteristics

Abstract

In various implementations, upon receiving a given voice command from a user, a voice-based trigger may be selected from a library of voice-based triggers previously used across a population of users. The library may include association(s) between each voice-based trigger and responsive action(s) previously performed in response to the voice-based trigger. The selecting may be based on a measure of similarity between the given voice command and the selected voice-based trigger. One or more responsive actions associated with the selected voice-based trigger in the library may be determined. Based on the one or more responsive actions, current responsive action(s) may be performed by the client device. Feedback associated with performance of the current responsive action(s) may be received from the user and used to alter a strength of an association between the selected voice-based trigger and the one or more responsive actions, wherein the altering includes incrementing or decrementing a count corresponding to the strength of association based on the feedback being positive or negative respectively.

Claims

1. A method comprising:

receiving, at a client device, a given voice command from a user;

selecting, from a library of voice-based triggers previously used across a population of users, a selected voice-based trigger, wherein the library includes one or more associations between each respective voice-based trigger of the library of voice-based triggers and one or more responsive actions previously invoked automatically by one or more other client devices operated by one or more other users of the population of users in response to the respective voice-based trigger, and wherein the selecting is based at least in part on a measure of similarity between the given voice command and the selected voice-based trigger;

determining the one or more responsive actions associated with the selected voice-based trigger in the library;

identifying an application previously executed by one or more of the other client devices to perform the one or more responsive actions associated with the selected voice-based trigger in the library;

analyzing a plurality of distinct applications installed on the client device;

determining, based on the analyzing, that the application previously executed by the one or more of the other client devices to perform the one or more responsive actions is not currently installed on the client device;

identifying, based on the analyzing, an analogous application to the application previously executed by the one or more of the other client devices;

invoking, by the client device, the analogous application to perform the one or more responsive actions associated with the selected voice-based trigger;

receiving, by the client device from the user, feedback associated with the client device invoking the analogous application to perform the one or more responsive actions; and

altering a strength of an association between the selected voice-based trigger and the one or more responsive actions based on the feedback, wherein the altering includes incrementing a count corresponding to the strength of association when the feedback is positive and decrementing the count when the feedback is negative;

wherein the selected voice-based trigger is selected from the library of voice-based triggers further based on a context of the user detected using one or more sensors of the client device.

2. The method of claim 1, wherein determining the one or more responsive actions associated with the selected voice-based trigger in the library comprises selecting at least one of the responsive actions from a plurality of candidate responsive actions associated with the selected voice-based trigger in the library.

3. The method of claim 2, wherein the at least one responsive action is selected based on the context of the user.

4. The method of claim 1, wherein the selected voice-based trigger includes one or more terms or tokens that are analogous to one or more terms or tokens in the given voice command.

5. The method of claim 1, wherein the altering comprises weakening the association between the selected voice-based trigger and the one or more responsive actions based on the feedback.

6. The method of claim 1, wherein the altering comprises strengthening the association between the selected voice-based trigger and the one or more responsive actions based on the feedback.

7. The method of claim 1, wherein the measure of similarity is a measure of syntactic similarity.

8. The method of claim 1, wherein the measure of similarity is a measure of semantic similarity.

9. The method of claim 1, wherein the given voice command fails to specify at least one parameter required by the one or more responsive actions, and the method further includes identifying, based on the context of the user, a value to use as the at least one parameter.

10. The method of claim 1, wherein the given voice command fails to specify at least one parameter required by the one or more responsive actions, and the method further includes identifying, based on a personal library associated with the user, a value to use as the at least one parameter.

11. A system comprising one or more processors and memory operably coupled with the one or more processors, wherein the memory stores instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to operate an interactive assistant module configured to:

receive a given voice command from a user;

select, from a library of voice-based triggers previously used across a population of users, a selected voice-based trigger, wherein the library includes one or more associations between each respective voice-based trigger of the library of voice-based triggers and one or more responsive actions previously invoked automatically by one or more other client devices operated by one or more other users of the population of users in response to the respective voice-based trigger, and wherein the selecting is based at least in part on a measure of similarity between the given voice command and the selected voice-based trigger;

determine the one or more responsive actions associated with the selected voice-based trigger in the library;

identify an application previously executed by one or more of the other client devices to perform the one or more responsive actions associated with the selected voice-based trigger in the library;

analyze a plurality of distinct applications installed on the client device;

determine, based on the analysis, that the application previously executed by the one or more of the other client devices to perform the one or more responsive actions is not currently installed on the client device;

identify, based on the analysis, an analogous application to the application previously executed by the one or more of the other client devices;

invoke the analogous application to perform the one or more responsive actions associated with the selected voice-based trigger;

receive, from the user, feedback associated with the one or more processors invoking analogous application to perform the one or more responsive actions; and

alter a strength of an association between the selected voice-based trigger and the one or more responsive actions based on the feedback, wherein the alteration includes incrementing a count corresponding to the strength of association when the feedback is positive and decrementing the count when the feedback is negative;

wherein the selected voice-based trigger is selected from the library of voice-based triggers further based on a context of the user detected using one or more sensors of the system.

12. The system of claim 11, wherein the interactive assistant module is further configured to weaken the association between the selected voice-based trigger and the one or more responsive actions based on the feedback.

13. The system of claim 11, wherein the given voice command fails to specify at least one parameter required by the one or more responsive actions, and the method further includes identifying, based on the context of the user or based on a personal library associated with the user, a value to use as the at least one parameter.

14. A method comprising:

receiving, from a population of users, a plurality of voice-based triggers and responsive actions invoked automatically at client devices operated by users of the population of users in response to the voice-based triggers;

storing associations between the voice-based triggers and the responsive actions in a library, wherein the associations are endorsed by the population of users;

receiving a new voice command provided by a given user at a given client device;

selecting, from the library of voice-based triggers, a selected voice-based trigger, wherein the selecting is based at least in part on a measure of similarity between the new voice command and the selected voice-based trigger and on a context of the given user detected using one or more sensors of the given client device;

determining one or more responsive actions associated with the selected voice-based trigger in the library;

identifying an application previously executed by one or more of the client devices operated by users of the population of users to perform the one or more responsive actions associated with the selected voice-based trigger in the library;

determining that the application previously executed by the one or more of the client devices operated by users of the population of users to perform the one or more responsive actions is not currently installed on the given client device;

identifying, to the given client device, an analogous application to the application previously executed by the one or more of the other client devices, to trigger the given client device to execute the analogous application to perform the one or more responsive actions;

receiving, by the client device based on input from the user, feedback associated with the given client device invoking the one or more responsive actions; and

altering a strength of an association between the selected voice-based trigger and the one or more responsive actions based on the feedback, wherein the altering includes incrementing a count corresponding to the strength of association when the feedback is positive and decrementing the count when the feedback is negative.