Wake word evaluation

• US 9,275,637 B1
• Filed: 11/06/2012
• Issued: 03/01/2016
• Est. Priority Date: 11/06/2012
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Salvador, Stan Weidner, Lilly, Jeffrey Paul, Weber, Frederick V., Adams, Jeffrey Penrod, Thomas, Ryan Paul
• Primary Examiner(s): Pullias, Jesse
• Application Number:
  US13/670,316
• Time in Patent Office: 1,211 Days
• Field of Search: 704231-257, 704/275
• US Class Current: 1/1
• CPC Class Codes: G10L 15/01 Assessment or evaluation of speech recognition systems
• CPC Class Codes: G10L 15/06 Creation of reference templates; Training of speech recognition systems, e.g. adaptation to the characteristics of the speaker's voice G10L15/14 takes precedence
• CPC Class Codes: G10L 15/187 Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams
• CPC Class Codes: G10L 2015/088 Word spotting

Abstract

Natural language controlled devices may be configured to activate command recognition in response to one or more wake words. Techniques are provided to receive a candidate word for evaluation as a wake word that activates a natural language control functionality of a computing device. The candidate word may include one or more words or sounds. Values for multiple wake word metrics are then determined. The candidate word is evaluated based on the various wake word metrics.

Claims

1. A computing system comprising:

one or more processors;

one or more computer-readable storage media storing computer-executable instructions;

a speech recognition module maintained in the one or more computer-readable storage media and executed by the one or more processors to interpret a signal representing the audio input as one or more words; and

a wake word evaluation module maintained in the one or more computer-readable storage media and executed by the one or more processors to;

receive a plurality of candidate wake words for evaluation, wherein the plurality of candidate wake words comprises a first candidate wake word;

obtain a corpus of text for evaluating the plurality of candidate wake words;

obtain a corpus of audio data for evaluating the plurality of candidate wake words;

perform speech recognition on the corpus of audio data using the speech recognition module to obtain speech recognition results;

determine a plurality of values from wake word metrics for the first candidate wake word using at least one of the corpus of text or the speech recognition results, at least one of the plurality of values including a value indicative of an ease of pronouncing the first candidate word;

determine an overall score for the first candidate wake word based at least in part on the plurality of values; and

select a first wake word from the plurality of candidate wake words using the overall score.

2. The system as recited in claim 1, wherein the wake word evaluation module is further executable by the one or more processors to:

select a second wake word from the plurality of wake word candidates using the overall score.

3. The system as recited in claim 1, wherein the plurality of values for the first candidate wake word comprise one or more of:

a value indicating an ease of pronouncing the first candidate wake word;

a value indicating a false alarm count for the first candidate wake word;

a value indicating a lack of ambiguity of a pronunciation of the first candidate wake word;

a value indicating a similarity of the first candidate wake word to one or more other words;

or

a value indicating a similarity of the first candidate wake word to taboo words.

4. The system as recited in claim 1,

wherein the plurality of wake word values for the first candidate wake word include a word frequency value that indicates a frequency of occurrence of the first candidate wake word in the corpus of text.

5. The system as recited in claim 1, wherein the plurality of wake word values for the first candidate wake word include a false alarm value, wherein the false alarm value is computed by comparing a phone sequence of the first candidate wake word to a plurality of phone sequences obtained from the speech recognition results.

6. A method comprising:

under control of one or more computer systems configured with executable instructions,

receiving an audio input including a first candidate word for evaluation as a wake word that activates functionality of a computing device;

determining a plurality of values from wake word metrics for the first candidate word, at least one of the plurality of values including a value indicative of an ease of pronouncing the first candidate word; and

determining an overall score for the first candidate word based at least in part on the plurality of values.

7. The method as recited in claim 6, wherein the plurality of values includes a false alarm score value that is determined for the first candidate word by:

determining a phone sequence of the first candidate word;

comparing the phone sequence of the first candidate word to one or more phone sequences corresponding to speech recognition results; and

determining the false alarm score value based at least in part on the comparison.

8. The method as recited in claim 7, wherein the comparing the phone sequence of the first candidate word to one or more phone sequences corresponding to speech recognition results includes determining an average per-utterance minimum distance match determined by:

for one or more test utterances, determining a plurality of distance matches for each of the test utterances, each distance match corresponding to a distance match between a respective sequence of consecutive phones of a plurality of sequences of phones of the corresponding test utterance to the phone sequence of the first candidate word;

determining a minimum distance match for each of the test utterances; and

determining an average of the minimum distance matches of the test utterances.

9. The method as recited in claim 8, wherein at least two respective consecutive sequences of phones of a test utterance for which a distance match is determined overlap.

10. The method as recited in claim 6, wherein the value indicative of the ease of pronunciation of the first candidate wake word is determined at least in part by determining a number of constraints on syllables in a language that the candidate word violates.

11. The method as recited in claim 10, wherein the value indicative of the ease of pronunciation of the candidate word is further determined at least in part by a weighting of each of the constraints violated by the candidate word.

12. The method as recited in claim 10, wherein the metric indicative of the ease of pronunciation of the candidate word is further determined at least in part by determining a number of constraints on syllables in a second language that the candidate word violates.

13. The method as recited in claim 6, wherein the plurality of values includes at least one of:

a value indicative of a similarity of the first candidate word to a plurality of other words of a language, the similarity to the plurality of other words of the language being determined based on a comparison between a phone sequence of at least one of the other words to a phone sequence of the first candidate word; and

a value indicative of a similarity of the first candidate word to one or more taboo words, the similarity to the one or more taboo words being determined based on a comparison between a sequence of phones of at least one of the taboo words and the phone sequence of the first candidate word.

14. The method as recited in claim 13, wherein a phoneme confusion matrix is used to determine at least one of (1) the comparison between the phone sequence of at least one of the other words to the phone sequence of the first candidate word or (2) the comparison between the phone sequence of at least one of the taboo words to the phone sequence of the first candidate word.

15. A method comprising:

under control of one or more computer systems configured with executable instructions,

receiving an audio input including a first candidate word for evaluation as a wake word that activates functionality of a computing device;

determining a plurality of values from wake word metrics for the first candidate word, wherein the plurality of values includes a false alarm count value that is determined for the first candidate word by;

comparing a phone sequence of the first candidate word to a plurality of phone sequences obtained from speech recognition results;

determining a match for each of the plurality of phone sequences obtained from speech recognition results; and

determining a count of phone sequences obtained from speech recognition results that have a match below a threshold level; and

determining an overall score for the first candidate word based at least in part on the plurality of values.

16. The method as recited in claim 15, wherein the match is a minimum distance match.

17. The method as recited in claim 16, wherein the count is a first count and the threshold level is a first threshold level and wherein the first threshold is indicative of an exact match;

the method further comprising determining a second count of phone sequences that have a minimum distance match below a second threshold level, the second threshold level being indicative of a less than exact match.

18. A method comprising:

under control of one or more computer systems configured with executable instructions,

receiving an audio input including a first candidate word for evaluation as a wake word that activates functionality of a computing device;

determining a plurality of values from wake word metrics for the first candidate word, wherein the plurality of values includes a word frequency value that is determined for the first candidate word by determining a frequency of occurrence of the first candidate word in a test set comprising one or more words; and

determining an overall score for the first candidate word based at least in part on the plurality of values.

19. The method as recited in claim 18, wherein the word frequency value is normalized to value between one and zero.

20. The method as recited in claim 18, wherein the one or more words of the test set is associated with at least one of voicemail transcriptions, online posts, or dictionary entries.

21. A computing system comprising:

one or more processors;

one or more computer-readable storage media storing computer-executable instructions;

a wake word evaluation module maintained in the one or more computer-readable storage media and executed by the one or more processors to;

receive a candidate word for evaluation as a wake word that activates functionality of a computing device, the candidate word received as an audio input;

determine a plurality of values from wake word metrics for the candidate word, at least one of the plurality of values including a word frequency value; and

determine an overall score for the candidate wake word based at least in part on the plurality of values.

22. The system as recited in claim 21, wherein the plurality of values includes a false alarm score value that is determined for the candidate word by:

determining a phone sequence of the candidate word;

comparing the phone sequence of the candidate word to one or more phone sequences corresponding to speech recognition results; and

determining the false alarm score value based at least in part on the comparison.

23. The system as recited in claim 21, wherein the plurality of values includes at least one of:

a value indicative of a similarity of the candidate word to a plurality of other words of a language, the similarity to the plurality of other words of the language being determined based on a comparison between a phone sequence of at least one of the other words to a phone sequence of the candidate word; and

a value indicative of a similarity of the candidate word to one or more taboo words, the similarity to the one or more taboo words being determined based on a comparison between a sequence of phones of at least one of the taboo words and the phone sequence of the candidate word.

24. The system as recited in claim 23, wherein a phoneme confusion matrix is used to determine at least one of (1) the comparison between the phone sequence of at least one of the other words to the phone sequence of the candidate word or (2) the comparison between the phone sequence of at least one of the taboo words to the phone sequence of the candidate word.

25. The system as recited in claim 21, wherein the plurality of values includes a value indicative of an ease of pronouncing the first candidate word.

26. A computing system comprising:

one or more processors;

one or more computer-readable storage media storing computer-executable instructions;

a wake word evaluation module maintained in the one or more computer-readable storage media and executed by the one or more processors to;

receive an audio input including a candidate word for evaluation as a wake word that activates functionality of a computing device;

determine a plurality of values from wake word metrics for the candidate word, wherein the plurality of values includes a word frequency value that is determined for the candidate word by determining a frequency of occurrence of the candidate word in a test set comprising one or more words; and

determine an overall score for the candidate wake word based at least in part on the plurality of values.

27. The computing system as recited in claim 26, wherein the word frequency value is normalized to value between one and zero.

28. The computing system as recited in claim 26, wherein the one or more words of the test set is associated with at least one of voicemail transcriptions, online posts, or dictionary entries.

29. A computing system comprising:

one or more processors;

one or more computer-readable storage media storing computer-executable instructions;

a wake word evaluation module maintained in the one or more computer-readable storage media and executed by the one or more processors to;

receive an audio input including a candidate word for evaluation as a wake word that activates functionality of a computing device;

determine a plurality of values from wake word metrics for the candidate word, wherein the plurality of values includes a value indicative of an ease of pronouncing the candidate word determined at least in part by determining a number of constraints on syllables in a language that the candidate word violates, wherein the determination includes a weighting of each of the constraints violated by the candidate word; and

determine an overall score for the candidate wake word based at least in part on the plurality of values.

30. One or more non-transitory computer-readable media having computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

receiving an audio input including a candidate word for evaluation as a wake word that activates functionality of a computing device, the candidate word comprising a sequence of subword units;

determining a plurality of wake word values for the candidate word, at least one wake word value including a word frequency value; and

determining an overall value for the candidate word based at least in part on the plurality of wake word values.

31. The one or more computer-readable media as recited in claim 30, wherein at least one wake word value determined based, at least in part, on a comparison of the sequence of subword units of the candidate word to a sequence of subword units corresponding to a test data.

32. The one or more computer-readable media as recited in claim 31, wherein the comparison of the subword unit sequence of the candidate word to the one or more subword unit sequences corresponds to at least one of:

the test data comprises using a phoneme confusion matrix;

or

the test data at least one test utterance is a Levenshtein distance.

33. The one or more computer-readable media as recited in claim 31, the operations further comprising:

generating the sequence of subword units corresponding to the test data using a speech recognition system and audio data.

34. The one or more computer-readable media as recited in claim 31, the operations further comprising:

generating the sequence of subword units corresponding to the test data using a pronunciation lexicon and text data.

35. The one or more computer-readable media as recited in claim 30, wherein the plurality of wake word values for the candidate wake word comprise one or more of:

a value indicating an ease of pronouncing the candidate wake word;

a value indicating a false alarm count for the candidate word;

a value indicating a lack of ambiguity of a pronunciation of the candidate wake word;

a value indicating a similarity of the candidate wake word to one or more other words;

a value indicating a similarity of the candidate wake word to taboo words;

or

a value indicating an ease of recognizing the candidate wake word with a speech recognition system.