Methods and systems for determining instructions for applications that are recognizable by a voice interface

• US 9,318,128 B1
• Filed: 02/01/2013
• Issued: 04/19/2016
• Est. Priority Date: 01/08/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Mengibar, Pedro J. Moreno, Biadsy, Fadi, Epstein, Mark Edward
• Primary Examiner(s): Pham, Tammy
• Assistant Examiner(s): Yehl, Walter
• Application Number:
  US13/757,257
• Time in Patent Office: 1,173 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G10L 15/19 Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 21/10 Transforming into visible information

Abstract

Methods and systems for facilitating development of voice-enabled applications are provided. The method may comprise receiving, at a computing device, a plurality of actions associated with a given application, parameters associated with each respective action, and example instructions responsive to respective actions. The method may also comprise determining candidate instructions based on the actions, parameters, and example instructions. Each candidate instruction may comprise one or more grammars recognizable by a voice interface for the given application. The method may further comprise the computing device receiving respective acceptance information for each candidate instruction, and comparing at least a portion of the respective acceptance information with a stored acceptance information log comprising predetermined acceptance information so as to determine a correlation. Still further, the method may comprise determining a set of instructions comprising one or more of the candidate instructions based on the determined correlation.

Claims

1. A method comprising:

receiving, at a computing device, information indicating a plurality of actions that a given application is configured to perform, wherein the information indicating each respective action of the plurality of actions includes one or more parameters that are used to enable the given application to perform the respective action;

receiving, at the computing device, for each respective action, one or more example instructions that, when recognized by the given application, causes the given application to perform the respective action, wherein the one or more example instructions comprise natural language;

based on the one or more example instructions for each respective action, the plurality of actions, and the one or more parameters included for each respective action, the computing device determining a plurality of candidate instructions for each respective action that, when recognized by a voice interface of the given application, causes the given application to perform the respective action, wherein each candidate instruction of the plurality of candidate instructions comprises one or more grammars recognizable by the voice interface of the given application;

receiving, at the computing device, a plurality of acceptance information comprising, for each candidate instruction, respective acceptance information indicative of whether the candidate instruction is applicable to the respective action, wherein the plurality of acceptance information indicates, for each respective action of at least one of the plurality of actions, that at least one of the plurality of candidate instructions determined for the respective action is rejected;

based on the plurality of acceptance information, storing, at the computing device, for each candidate instruction, a respective identifier associated with the candidate instruction and indicative of whether the candidate instruction is rejected;

the computing device comparing at least a portion of the respective acceptance information with a stored acceptance information log so as to determine a correlation between the respective acceptance information and the stored acceptance information log, wherein the stored acceptance information log comprises a plurality of predetermined acceptance information associated with a plurality of predetermined example instructions; and

based on the correlation, the computing device referring to the stored identifiers to determine a set of instructions that are recognizable by the voice interface and that, when recognized by the voice interface, causes the given application to perform one or more of the plurality of actions, wherein determining the set of instructions comprises, for each respective action, (i) selecting, based on the stored identifiers, one or more of the plurality of candidate instructions for the respective action to be included in the set of instructions and (ii) selecting, based on the stored identifiers, the at least one rejected candidate instruction for the respective action to not be included in the set of instructions and to not be recognizable by the voice interface to cause the given application to perform the respective action.

2. The method of claim 1, further comprising providing the set of instructions to the voice interface of the given application.

3. The method of claim 1, further comprising based on the correlation, the computing device modifying the stored acceptance information log to include at least a portion of the plurality of acceptance information.

4. The method of claim 1, wherein each respective parameter of the one or more parameters includes at least one argument associated with the respective parameter, wherein the at least one argument includes a text string.

5. The method of claim 1, wherein the respective acceptance information includes one or more of:

an acceptance of the candidate instruction, a rejection of the candidate instruction, a confidence score of the candidate instruction, and a rank of the candidate instruction with respect to at least one other candidate instruction of the plurality of candidate instructions.

6. The method of claim 1, wherein determining the plurality of candidate instructions for each respective action comprises comparing the plurality of actions with stored instruction logs, wherein the stored instruction logs include a plurality of respective stored actions associated with each respective stored instruction of the stored instruction logs.

7. The method of claim 6, wherein the determined plurality of candidate instructions is based at least in part on a match of one or more portions of the plurality of actions to one or more portions of the stored instruction logs.

8. The method of claim 6, wherein determining the plurality of candidate instructions for each respective action further comprises:

matching at least a portion of the respective action to one or more grammars; and

determining a modification to the respective action based at least in part on the matching.

9. The method of claim 1, further comprising:

before determining the plurality of candidate instructions for each respective action, determining, by the computing device, markup language data representative of the given application, wherein the markup language data includes identifiers of the plurality of actions and the one or more parameters; and

the computing device parsing the markup language data to determine, based on the identifiers, the plurality of actions and the one or more parameters, and to further determine information indicating a plurality of input values for the one or more parameters.

10. The method of claim 9, wherein determining the plurality of actions and the information indicating the plurality of input values for the one or more parameters comprises comparing each respective action with stored actions so as to determine a correlation between the respective action and the stored actions, wherein each stored action includes one or more stored parameters associated with the stored action.

11. A non-transitory computer readable storage medium having stored thereon instructions, that when executed by a computing device, cause the computing device to perform functions comprising:

receiving information indicating a plurality of actions that a given application is configured to perform, wherein the information indicating each respective action of the plurality of actions includes one or more parameters that are used to enable the given application to perform the respective action;

receiving, for each respective action, one or more example instructions that, when recognized by the given application, causes the given application to perform the respective action, wherein the one or more example instructions comprise natural language;

based on the one or more example instructions for each respective action, the plurality of actions, and the one or more parameters included for each respective action, determining a plurality of candidate instructions for each respective action that, when recognized by a voice interface of the given application, causes the given application to perform the respective action, wherein each candidate instruction of the plurality of candidate instructions comprises one or more grammars recognizable by the voice interface of the given application;

receiving a plurality of acceptance information comprising, for each candidate instruction, respective acceptance information indicative of whether the candidate instruction is applicable to the respective action, wherein the plurality of acceptance information indicates, for each respective action of at least one of the plurality of actions, that at least one of the plurality of candidate instructions determined for the respective action is rejected;

based on the plurality of acceptance information, storing, at the computing device, for each candidate instruction, a respective identifier associated with the candidate instruction and indicative of whether the candidate instruction is rejected;

comparing at least a portion of the respective acceptance information with a stored acceptance information log so as to determine a correlation between the respective acceptance information and the stored acceptance information log, wherein the stored acceptance information log comprises a plurality of predetermined acceptance information associated with a plurality of predetermined example instructions; and

based on the correlation, referring to the stored identifiers to determine a set of instructions that are recognizable by the voice interface and that, when recognized by the voice interface, causes the given application to perform one or more of the plurality of actions, wherein determining the set of instructions comprises, for each respective action, (i) selecting, based on the stored identifiers, one or more of the plurality of candidate instructions for the respective action to be included in the set of instructions and (ii) selecting, based on the stored identifiers, the at least one rejected candidate instruction for the respective action to not be included in the set of instructions and to not be recognizable by the voice interface to cause the given application to perform the respective action.

12. The non-transitory computer readable storage medium of claim 11, wherein each instruction of the set of instructions includes a text string representative of one or more spoken utterances.

13. The non-transitory computer readable storage medium of claim 11, wherein the given application includes a graphical interface provided by a mobile device.

14. The non-transitory computer readable storage medium of claim 11, wherein the respective acceptance information includes one or more of:

an acceptance of the candidate instruction, a rejection of the candidate instruction, a confidence score of the candidate instruction, and a rank of the candidate instruction with respect to at least one other candidate instruction of the plurality of candidate instructions.

15. The non-transitory computer readable storage medium of claim 14, wherein determining the set of instructions comprises adding the respective candidate instruction to the set of instructions based on the confidence score of the respective candidate instruction being above a threshold.

16. The non-transitory computer readable storage medium of claim 14, wherein determining the set of instructions comprises adding the respective candidate instruction to the set of instructions based on the rank of the candidate instruction is greater than at least one other candidate instruction of the plurality of candidate instructions.

17. A system, comprising:

at least one processor; and

data storage comprising program instructions executable by the at least one processor to perform functions comprising;

receiving information indicating a plurality of actions that a given application is configured to perform, wherein the information indicating each respective action of the plurality of actions includes one or more parameters that are used to enable the given application to perform the respective action;

receiving, for each respective action, one or more example instructions that, when recognized by the given application, causes the given application to perform the respective action, wherein the one or more example instructions comprise natural language;

based on the one or more example instructions for each respective action, the plurality of actions, and the one or more parameters included for each respective action, determining a plurality of candidate instructions for each respective action that, when recognized by a voice interface of the given application, causes the given application to perform the respective action, wherein each candidate instruction of the plurality of candidate instructions comprises one or more grammars recognizable by the voice interface of the given application;

receiving a plurality of acceptance information comprising, for each candidate instruction, respective acceptance information indicative of whether the candidate instruction is applicable to the respective action, wherein the plurality of acceptance information indicates, for each respective action of at least one of the plurality of actions, that at least one of the plurality of candidate instructions determined for the respective action is rejected;

based on the plurality of acceptance information, storing, at the computing device, for each candidate instruction, a respective identifier associated with the candidate instruction and indicative of whether the candidate instruction is rejected;

comparing at least a portion of the respective acceptance information with a stored acceptance information log so as to determine a correlation between the respective acceptance information and the stored acceptance information log, wherein the stored acceptance information log comprises a plurality of predetermined acceptance information associated with a plurality of predetermined example instructions; and

based on the correlation, referring to the stored identifiers to determine a set of instructions that are recognizable by the voice interface and that, when recognized by the voice interface, causes the given application to perform one or more of the plurality of actions, wherein determining the set of instructions comprises, for each respective action, (i) selecting, based on the stored identifiers, one or more of the plurality of candidate instructions for the respective action to be included in the set of instructions and (ii) selecting, based on the stored identifiers, the at least one rejected candidate instruction for the respective action to not be included in the set of instructions and to not be recognizable by the voice interface to cause the given application to perform the respective action.

18. The system of claim 17, further comprising program instructions executable by the at least one processor to perform functions comprising providing the set of instructions to the voice interface of the given application.

19. The system of claim 18, further comprising a software development kit configured to modify the given application so as to include the voice interface.

20. The system of claim 17, wherein the given application includes a graphical interface provided by a mobile device.