Determining hotword suitability

• US 10,002,613 B2
• Filed: 01/20/2016
• Issued: 06/19/2018
• Est. Priority Date: 07/03/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Rubin, Andrew E., Schalkwyk, Johan, San Martin, Maria Carolina Parada
• Primary Examiner(s): Ortiz Sanchez, Michael
• Application Number:
  US15/002,044
• Publication Number:
  US 20160133259A1
• Time in Patent Office: 881 Days
• CPC Class Codes: G06F 21/32 using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 21/46 by designing passwords or checking the strength of passwords
• CPC Class Codes: G10L 15/06 Creation of reference templates; Training of speech recognition systems, e.g. adaptation to the characteristics of the speaker's voice G10L15/14 takes precedence
• CPC Class Codes: G10L 15/08 Speech classification or search
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 17/24 the user being prompted to utter a password or a predefined phrase
• CPC Class Codes: G10L 2015/0638 Interactive procedures
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/225 Feedback of the input speech
• CPC Class Codes: G10L 25/51 for comparison or discrimination

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for determining hotword suitability. In one aspect, a method includes receiving speech data that encodes a candidate hotword spoken by a user, evaluating the speech data or a transcription of the candidate hotword, using one or more predetermined criteria, generating a hotword suitability score for the candidate hotword based on evaluating the speech data or a transcription of the candidate hotword, using one or more predetermined criteria, and providing a representation of the hotword suitability score for display to the user.

Claims

1. A computer-implemented method comprising:

providing, for output by a computing device on a display or with a speaker of the computing device, a prompt for a user to speak a candidate hotword that is to be designated a custom hotword, wherein a hotword is a word or phrase that indicates to a voice command processing subsystem on the computing device that one or more terms that are uttered immediately after the hotword are to be processed as voice commands;

receiving, using a microphone of the computing device, one or more utterances of the candidate hotword in response to the prompt;

accepting the candidate hotword as a custom hotword for the computing device based on a determination that the candidate hotword satisfies one or more hotword suitability criteria,

generating, by a user interface generator of the computing device, a user interface having an indication that the candidate hotword has been accepted as a custom hotword;

providing, by the computing device and in response to the one or more utterances of the candidate hotword, the indication, on the display of the computing device or using the speaker of the computing device, that the candidate hotword has been accepted as a custom hotword;

after providing the indication, detecting, using the microphone of the computing device, an utterance of the custom hotword; and

based on detecting the utterance of the custom hotword, causing the computing device to enter a ready state for receiving and processing voice commands uttered after the utterance of the custom hotword.

2. The computer-implemented method of claim 1,

further comprising, after providing the indication that the candidate hotword has been accepted as a custom hotword:

receiving an utterance of (i) the custom hotword, immediately followed by (ii) a particular set of one or more terms; and

determining that the particular set of one or more terms uttered immediately after the custom hotword are to be processed as one or more voice commands.

3. The computer-implemented method of claim 2,

wherein receiving one or more utterances of the candidate hotword in response to the prompt comprises:

while the computing device is in an activated state, receiving one or more utterances of the candidate hotword in response to the prompt, and

wherein receiving the utterance of (i) the custom hotword, immediately followed by (ii) the particular set of one or more terms comprises;

while the computing device is in a dormant or idle state, receiving the utterance of (i) the custom hotword, immediately followed by (ii) the particular set of one or more terms.

4. The computer-implemented method of claim 2,

wherein receiving one or more utterances of the candidate hotword in response to the prompt comprises:

while the computing device is unlocked, receiving one or more utterances of the candidate hotword in response to the prompt, and

wherein receiving the utterance of (i) the custom hotword, immediately followed by (ii) the particular set of one or more terms comprises;

while the computing device is locked, receiving the utterance of (i) the custom hotword, immediately followed by (ii) the particular set of one or more terms.

5. The computer-implemented method of claim 2, further comprising:

after providing the indication that the candidate hotword has been accepted as a hotword,

wherein determining that the particular set of one or more terms uttered immediately after the custom hotword are to be processed as one or more voice commands is performed based at least on receiving data confirming that the candidate hotword is to be designated as a custom hotword.

6. The computer-implemented method of claim 5, wherein receiving data confirming that the candidate hotword is to be designated as a custom hotword comprises receiving data indicating user input provided to the computing device in response to the indication.

7. The computer-implemented method of claim 1, wherein providing the indication that the candidate hotword has been accepted as a hotword comprises providing, in response to the one or more utterances of the candidate hotword, (i) an indication that the candidate hotword has not been accepted as a hotword and (ii) a prompt for the user to speak another, different candidate term that is to be designated a custom hotword.

8. The computer-implemented method of claim 1,

wherein providing, for output, the prompt for the user to speak a candidate term that is to be designated a custom hotword comprises:

providing, for output by the computing device, a prompt for the user to speak a candidate term that is to be designated a custom hotword, and

wherein providing, in response to the one or more utterances of the candidate hotword, the indication that the candidate hotword has been accepted as a hotword comprises;

providing, in response to the one or more utterances of the candidate hotword, the indication for output by the computing device.

9. The computer-implemented method of claim 1, wherein providing, in response to the one or more utterances of the candidate hotword, the indication that the candidate hotword has been accepted as a hotword comprises providing, in response to the one or more utterances of the candidate hotword, an indication that a confidence value that reflects a level of confidence in the candidate hotword representing a suitable hotword has satisfied a predetermined threshold.

10. The computer-implemented method of claim 9, further comprising:

providing, in response to the one or more utterances of the candidate hotword, one or more graphical or textual representations of the confidence value for display on the computing device.

11. The computer-implemented method of claim 1, further comprising:

providing, for output on the display or speaker of the computing device, a second prompt for a user to speak a second candidate hotword that is to be designated a custom hotword;

receiving, using the microphone of the computing device, one or more utterances of the second candidate hotword in response to the prompt;

determining that the second candidate hotword does not satisfy one or more hotword suitability criteria; and

based on determining that the second candidate hotword does not satisfy one or more hotword suitability criteria, providing, on the display of the computing device or using the speaker of the computing device, that the candidate hotword has not been accepted as a hotword.

12. One or more non-transitory computer-readable media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

providing, for output by a computing device on a display or with a speaker of the computing device, a prompt for a user to speak a candidate hotword that is to be designated a custom hotword, wherein a hotword is a term that indicates to a voice command processing subsystem on the computing device that one or more terms that are uttered immediately after the hotword are to be processed as voice commands, wherein the computing device is configured to selectively accept candidate hotwords as hotwords for the computing device;

receiving, using a microphone of the computing device, one or more utterances of the candidate hotword in response to the prompt;

accepting the candidate hotword as a hotword for the computing device based on a determination that the candidate hotword satisfies one or more hotword suitability criteria;

generating, by a user interface generator of the computing device, a user interface having an indication that the candidate hotword has been accepted as a custom hotword;

providing, by the computing device and in response to the one or more utterances of the candidate hotword, the indication, on the display of the computing device or using the speaker of the computing device, that the candidate hotword has been accepted as a custom hotword;

after providing the indication, detecting, using the microphone of the computing device, an utterance of the custom hotword; and

based on detecting the utterance of the custom hotword, causing the computing device to enter a ready state for receiving and processing voice commands uttered after the utterance of the custom hotword.

13. The one or more non-transitory computer-readable media of claim 12,

wherein the operations further comprise, after providing the indication that the candidate hotword has been accepted as a hotword:

receiving an utterance of (i) the custom hotword, immediately followed by (ii) a particular set of one or more terms; and

determining that the particular set of one or more terms uttered immediately after the custom hotword are to be processed as one or more voice commands.

14. The one or more non-transitory computer-readable media of claim 13,

wherein receiving one or more utterances of the candidate hotword in response to the prompt comprises:

while the computing device is in an activated state, receiving one or more utterances of the candidate hotword in response to the prompt, and

wherein receiving the utterance of (i) the custom hotword, immediately followed by (ii) the particular set of one or more terms comprises;

while the computing device is in a dormant or idle state, receiving the utterance of (i) the custom hotword, immediately followed by (ii) the particular set of one or more terms.

15. The one or more non-transitory computer-readable media of claim 13,

wherein receiving one or more utterances of the candidate hotword in response to the prompt comprises:

while the computing device is unlocked, receiving one or more utterances of the candidate hotword in response to the prompt, and

wherein receiving the utterance of (i) the custom hotword, immediately followed by (ii) the particular set of one or more terms comprises;

while the computing device is locked, receiving the utterance of (i) the custom hotword, immediately followed by (ii) the particular set of one or more terms.

16. The one or more non-transitory computer-readable media of claim 13, wherein the operations further comprise:

after providing the indication that the candidate hotword has been accepted as a hotword, receiving data confirming that the candidate hotword is to be designated as a custom hotword;

wherein receiving the utterance of (i) custom a hotword, immediately followed by (ii) the particular set of one or more terms comprises, receiving, after receiving data confirming that the candidate hotword is to be designated as a custom hotword, the utterance of (i) the custom hotword; and

wherein determining that the particular set of one or more terms uttered immediately after the custom hotword are to be processed as one or more voice commands comprises determining, based at least on receiving data confirming that the candidate hotword is to be designated as a custom hotword, that the particular set of one or more terms uttered immediately after the custom hotword are to be processed as one or more voice commands.

17. The one or more non-transitory computer-readable media t of claim 16, wherein receiving data confirming that the candidate hotword is to be designated as a custom hotword comprises receiving data indicating user input provided to the computing device in response to the indication.

18. The one or more non-transitory computer-readable media of claim 12, wherein providing the indication that the candidate hotword has been accepted as a hotword comprises providing, in response to the one or more utterances of the candidate hotword, (i) an indication that the candidate hotword has not been accepted as a hotword and (ii) a prompt for the user to speak another, different candidate term that is to be designated a custom hotword.

19. The one or more non-transitory computer-readable media of claim 12,

wherein providing, for output, the prompt for the user to speak a candidate term that is to be designated a custom hotword comprises:

providing, for output by the computing device, a prompt for the user to speak a candidate term that is to be designated a custom hotword, and

wherein providing, in response to the one or more utterances of the candidate hotword, the indication that the candidate hotword has been accepted as a hotword comprises;

providing, in response to the one or more utterances of the candidate hotword, an indication that the candidate hotword has been accepted as a hotword for output by the computing device.

20. The one or more non-transitory computer-readable media of claim 12, wherein providing, in response to the one or more utterances of the candidate hotword, the indication that the candidate hotword has been accepted as a hotword comprises providing, in response to the one or more utterances of the candidate hotword, an indication that a confidence value that reflects a level of confidence in the candidate hotword representing a suitable hotword has satisfied a predetermined threshold.

21. A computer-implemented method comprising:

receiving, using a microphone of a computing device, a request to designate a particular candidate hotword that is not currently designated as a hotword, as a hotword;

determining that the particular candidate hotword satisfies one or more hotword suitability criteria;

in response to determining that the particular candidate hotword satisfies one or more hotword suitability criteria, designating the particular candidate hotword as a custom hotword;

after designating the particular candidate hotword as a custom hotword, determining that subsequently received audio data received using the microphone of the computing device includes sounds that are characteristic of an utterance of the custom hotword;

in response to determining that the subsequently received audio data includes sounds that are characteristic of an utterance of the custom hotword;

providing, on a display of the computing device or using a speaker of the computing device, an indication that the custom hotword was detected; and

causing the computing device to enter a ready state for receiving and processing voice commands uttered after the utterance of the custom hotword.

22. The computer-implemented method of claim 21, further comprising:

determining that the particular candidate hotword includes a particular quantity of syllables or phones; and

wherein determining that the particular candidate hotword satisfies one or more hotword suitability criteria comprises determining, based at least on the particular quantity of syllables or phones, that the particular candidate hotword satisfies one or more hotword suitability criteria.

23. The computer-implemented method of claim 21, further comprising:

determining that the particular candidate hotword includes one or more particular phones; and

wherein determining that the particular candidate hotword satisfies one or more hotword suitability criteria comprises determining, based at least on the one or more particular phones included in the particular candidate hotword, that the particular candidate hotword satisfies one or more hotword suitability criteria.

24. The computer-implemented method of claim 21, further comprising:

determining that the particular candidate hotword is a particular edit distance from one or more terms; and

wherein determining that the particular candidate hotword satisfies one or more hotword suitability criteria comprises determining, based at least on the particular edit distance between the particular candidate hotword and the one or more terms, that the particular candidate hotword satisfies one or more hotword suitability criteria.

25. The computer-implemented method of claim 21, further comprising:

receiving, using the microphone of the computing device, a request to designate a second candidate hotword that is not currently designated as a hotword, as a hotword;

determining that the second candidate hotword does not satisfy one or more hotword suitability criteria; and

providing, on a display of the computing device or using a speaker of the computing device, an indication that the second hotword candidate is not accepted as a custom hotword.

26. The computer-implemented method of claim 21, further comprising:

before determining that the particular candidate hotword satisfies one or more hotword suitability criteria, receiving audio data that encodes an utterance of one or more words, at least a portion of which represent the particular candidate hotword.

27. The computer-implemented method of claim 26, further comprising:

determining a quantity of frames of the audio data that encode the utterance of one or more words of the particular candidate hotword; and

wherein determining that the particular candidate hotword satisfies one or more hotword suitability criteria comprises determining, based at least on the particular determined quantity of frames, that the particular candidate hotword satisfies one or more hotword suitability criteria.

28. The computer-implemented method of claim 21, comprising:

in response to determining that subsequently received audio data includes sounds that are characteristic of an utterance of the custom hotword, providing one or more commands for waking up the computing device.

29. The computer-implemented method of claim 21, comprising:

in response to determining that subsequently received audio data includes sounds that are characteristic of an utterance of the custom hotword, providing one or more commands for unlocking the computing device.

30. The computer-implemented method of claim 21, comprising:

in response to determining that subsequently received audio data includes sounds that are characteristic of an utterance of the custom hotword, providing one or more commands for applying settings of a user'"'"'s preference.

31. The computer-implemented method of claim 21, further comprising:

in response to determining that the particular candidate hotword satisfies one or more hotword suitability criteria, providing, for output at the computing device, an indication that the particular candidate hotword satisfies one or more hotword suitability criteria.

32. The computer-implemented method of claim 21, further comprising:

obtaining additional audio data encoding one or more utterances of the particular candidate hotword spoken by a user of the computing device; and

determining, based on the received audio data and the additional audio data, that the user'"'"'s pronunciation of the particular candidate hotword varies to a particular extent from utterance to utterance;

wherein determining that the particular candidate hotword satisfies one or more hotword suitability criteria comprises determining, based at least on the particular extent to which the user'"'"'s pronunciation of the particular candidate hotword varies from utterance to utterance, that the particular candidate hotword satisfies one or more hotword suitability criteria.

33. The computer-implemented method of claim 21, further comprising:

determining a confidence value that reflects a level of confidence in the particular candidate hotword representing a suitable hotword; and

wherein determining that the particular candidate hotword satisfies one or more hotword suitability criteria comprises determining, based on the confidence value, that the particular candidate hotword satisfies one or more hotword suitability criteria.

34. The computer-implemented method of claim 33, further comprising:

generating one or more textual or graphical representations of the confidence value that reflects a level of confidence in the particular candidate hotword representing a suitable hotword; and

providing the one or more textual or graphical representations for output at the computing device.

35. The computer-implemented method of claim 34, comprising:

generating a textual or graphical indication that the particular candidate hotword satisfies one or more hotword suitability criteria; and

providing the textual or graphical indication for output at the computing device.

36. The computer-implemented method of claim 21, wherein determining that subsequently received audio data includes sounds that are characteristic of an utterance of the custom hotword comprises:

determining, using a speech recognizer, that subsequently received audio data includes sounds that are characteristic of an utterance of the custom hotword.

37. The computer-implemented method of claim 36, further comprising:

determining that a particular quantity of training examples of the particular candidate hotword are available to the speech recognizer; and

wherein determining that the particular candidate hotword satisfies one or more hotword suitability criteria comprises determining, based at least on the particular quantity of training examples, that the particular candidate hotword satisfies one or more hotword suitability criteria.

38. A system comprising:

one or more computing devices; and

one or more data storage devices storing instructions that, when executed by the one or more computing devices, cause the one or more computing devices to perform operations comprising;

providing, for output by a computing device on a display or with a speaker of the computing device, a prompt for a user to speak a candidate hotword that is to be designated a custom hotword, wherein a hotword is a word or phrase that indicates to a voice command processing subsystem on the computing device that one or more terms that are uttered immediately after the hotword are to be processed as voice commands;

receiving, using a microphone of the computing device, one or more utterances of the candidate hotword in response to the prompt;

accepting the candidate hotword as a custom hotword for the computing device based on a determination that the candidate hotword satisfies one or more hotword suitability criteria;

generating, by a user interface generator of the computing device, a user interface having an indication that the candidate hotword has been accepted as a custom hotword;

providing, by the computing device and in response to the one or more utterances of the candidate hotword, the indication, on the display of the computing device or using the speaker of the computing device, that the candidate hotword has been accepted as a custom hotword;

after providing the indication, detecting, using the microphone of the computing device, an utterance of the custom hotword; and

based on detecting the utterance of the custom hotword, causing the computing device to enter a ready state for receiving and processing voice commands uttered after the utterance of the custom hotword.

39. A system comprising:

one or more computing devices; and

one or more data storage devices storing instructions that, when executed by the one or more computing devices, cause the one or more computing devices to perform operations comprising;

receiving, using a microphone of a computing device, a request to designate a particular candidate hotword that is not currently designated as a hotword, as a hotword;

determining that the particular candidate hotword satisfies one or more hotword suitability criteria;

in response to determining that the particular candidate hotword satisfies one or more hotword suitability criteria, designating the particular candidate hotword as a custom hotword;

after designating the particular candidate hotword as a custom hotword, determining that subsequently received audio data received using the microphone of the computing device includes sounds that are characteristic of an utterance of the custom hotword;

in response to determining that the subsequently received audio data includes sounds that are characteristic of an utterance of the custom hotword;

providing, on a display of the computing device or using a speaker of the computing device, an indication that the custom hotword was detected; and

causing the computing device to enter a ready state for receiving and processing voice commands uttered after the utterance of the custom hotword.

40. One or more non-transitory computer-readable media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

receiving, using a microphone of a computing device, a request to designate a particular candidate hotword that is not currently designated as a hotword, as a hotword;

determining that the particular candidate hotword satisfies one or more hotword suitability criteria;

in response to determining that the particular candidate hotword satisfies one or more hotword suitability criteria, designating the particular candidate hotword as a custom hotword;

after designating the particular candidate hotword as a custom hotword, determining that subsequently received audio data received using the microphone of the computing device includes sounds that are characteristic of an utterance of the custom hotword;

in response to determining that the subsequently received audio data includes sounds that are characteristic of an utterance of the custom hotword;

providing, on a display of the computing device or using a speaker of the computing device, an indication that the custom hotword was detected; and

causing the computing device to enter a ready state for receiving and processing voice commands uttered after the utterance of the custom hotword.