Multiple speech locale-specific hotword classifiers for selection of a speech locale

• US 9,589,564 B2
• Filed: 02/05/2014
• Issued: 03/07/2017
• Est. Priority Date: 02/05/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Sharifi, Matthew
• Primary Examiner(s): Sirjani, Fariba
• Application Number:
  US14/173,264
• Publication Number:
  US 20150221305A1
• Time in Patent Office: 1,126 Days
• Field of Search: 704/235
• US Class Current: 1/1
• CPC Class Codes: G10L 15/005 Language recognition
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/08 Speech classification or search
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 15/32 Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for recognizing speech in an utterance. The methods, systems, and apparatus include actions of receiving an utterance and obtaining acoustic features from the utterance. Further actions include providing the acoustic features from the utterance to multiple speech locale-specific hotword classifiers. Each speech locale-specific hotword classifier (i) may be associated with a respective speech locale, and (ii) may be configured to classify audio features as corresponding to, or as not corresponding to, a respective predefined term. Additional actions may include selecting a speech locale for use in transcribing the utterance based on one or more results from the multiple speech locale-specific hotword classifiers in response to providing the acoustic features from the utterance to the multiple speech locale-specific hotword classifiers. Further actions may include selecting parameters for automated speech recognition based on the selected speech locale.

Claims

1. A computer-implemented method, comprising:

receiving an utterance of a user by a device;

obtaining acoustic features from an initial portion of the utterance, wherein the initial portion of the utterance includes a hotword that is associated with waking the device from a low power state;

providing the acoustic features from the initial portion of the utterance to multiple speech locale-specific hotword classifiers of the device that process the acoustic features from the initial portion of the utterance in parallel, wherein each speech locale-specific hotword classifier is a speech recognition process using an acoustic model of a single language or a single dialect of a language and (i) is associated with a respective speech locale, (ii) is configured to classify acoustic features associated with an initial portion of an utterance as either corresponding to, or as not corresponding to, an utterance of a respective predefined term without one or more of transcribing or semantically interpreting acoustic features of any portion of the utterance, and (iii) is associated with a respective set of automatic speech recognition parameters;

obtaining speech locale-specific hotword confidence scores from the speech locale-specific hotword classifiers of the device, wherein each of the speech locale-specific hotword confidence scores obtained from a corresponding speech locale-specific hotword classifier reflects a likelihood that the initial portion of the utterance includes the respective predefined term corresponding to the speech locale-specific hotword classifier;

selecting a speech locale for use in generating a transcription of a subsequent portion of the utterance based on comparing speech locale-specific hotword confidence scores obtained from the multiple speech locale-specific hotword classifiers in response to the parallel processing of the acoustic features from the initial portion of the utterance by the multiple speech locale-specific hotword classifiers;

selecting the set of automatic speech recognition parameters associated with the selected speech locale; and

generating, by an automated speech recognizer, the transcription of the subsequent portion of the utterance using the selected set of automatic speech recognition parameters associated with the selected speech locale.

2. The method of claim 1, wherein two or more of the speech locale-specific hotword classifiers are associated with different speech locales, and are associated with the same predefined term.

3. The method of claim 1, wherein two or more of the speech locale-specific hotword classifiers are associated with different speech locales, and are associated with different predefined terms.

4. The method of claim 1, wherein providing the acoustic features from the initial portion of the utterance to multiple speech locale-specific hotword classifiers comprises:

providing the acoustic features from the initial portion of the utterance to a particular speech locale-specific hotword classifier that is associated with multiple other speech locale-specific hotword classifiers; and

determining to provide the acoustic features from the initial portion of the utterance to the multiple other speech locale-specific hotword classifiers based on a result from the particular speech locale-specific hotword classifier.

5. The method of claim 4, wherein the particular speech locale-specific hotword classifier comprises a speech locale-specific hotword classifier for a particular language and the multiple other speech locale-specific hotword classifiers associated with the particular speech locale-specific hotword classifier comprise speech locale-specific hotword classifiers for particular dialects for the particular language.

6. The method of claim 1, wherein providing the acoustic features from the initial portion of the utterance to multiple speech locale-specific hotword classifiers comprises selecting a subset of speech locale-specific hotword classifiers to provide the acoustic features from a set of speech locale-specific hotword classifiers based on the previous selections of speech locales used to transcribe previously received utterances.

7. The method of claim 1, wherein selecting a speech locale for use in generating a transcription of a subsequent portion of the utterance based on speech locale-specific hotword confidence scores from the multiple speech locale-specific hotword classifiers in response to providing the acoustic features from the initial portion of the utterance to the multiple speech locale-specific hotword classifiers comprises:

determining a highest speech locale-specific hotword confidence score from the speech locale-specific hotword confidence scores; and

selecting a speech locale associated with the determined highest speech locale-specific hotword confidence score.

8. The method of claim 1, wherein without one or more of transcribing or semantically interpreting the acoustic features comprises without transcribing the acoustic features to one or more of a word or sub-word level.

9. The method of claim 1, wherein receiving an utterance comprises receiving an utterance in a first mode where one or more of transcribing or semantically interpreting acoustic features obtained from utterances is not performed, the method comprising:

after obtaining the speech locale-specific hotword confidence scores from the speech locale-specific hotword classifiers, entering a second mode where one or more of transcribing or semantically interpreting acoustic features obtained from utterances is performed based on one or more results from the multiple speech locale-specific hotword classifiers in response to providing the acoustic features from the initial portion of the utterance to the multiple speech locale-specific hotword classifiers.

10. The method of claim 1, wherein the acoustic features comprise mel-frequency cepstrum coefficients.

11. The method of claim 1, wherein the respective predefined terms of the speech locale-specific hotword classifiers are specific to the respective speech locale.

12. The method of claim 1, wherein the speech locale-specific hotword confidence scores reflect whether the acoustic features from the initial portion of the utterance match acoustic features of the respective predefined term for the respective speech locale spoken in accordance with the respective speech locale.

13. The method of claim 1, comprising:

disabling one or more of the multiple speech-locale specific classifiers from being provided with acoustic features from utterances received in the future based on the one or more multiple speech-local specific classifiers being selected fewer times than another speech-locale specific classifier or having not been selected after a threshold number of speech locales are selected.

14. The method of claim 1, wherein an utterance is an utterance input into the device by the user.

15. The method of claim 1, wherein a low power state is a sleep state or hibernation state.

16. A system comprising:

one or more computers; and

one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

receiving an utterance of a user by a device;

obtaining acoustic features from an initial portion of the utterance, wherein the initial portion of the utterance includes a hotword that is associated with waking the device from a low power state;

providing the acoustic features from the initial portion of the utterance to multiple speech locale-specific hotword classifiers of the device that process the acoustic features from the initial portion of the utterance in parallel, wherein each speech locale-specific hotword classifier is a speech recognition process using an acoustic model of a single language or a single dialect of a language and (i) is associated with a respective speech locale, (ii) is configured to classify acoustic features associated with an initial portion of an utterance as either corresponding to, or as not corresponding to, an utterance of a respective predefined term without one or more of transcribing or semantically interpreting acoustic features of any portion of the utterance, and (iii) is associated with a respective set of automatic speech recognition parameters;

obtaining speech locale-specific hotword confidence scores from the speech locale-specific hotword classifiers of the device, wherein each of the speech locale-specific hotword confidence scores obtained from a corresponding speech locale-specific hotword classifier reflects a likelihood that the initial portion of the utterance includes the respective predefined term corresponding to the speech locale-specific hotword classifier;

selecting a speech locale for use in generating a transcription of a subsequent portion of the utterance based on comparing speech locale-specific hotword confidence scores obtained from the multiple speech locale-specific hotword classifiers in response to the parallel processing of the acoustic features from the initial portion of the utterance by the multiple speech locale-specific hotword classifiers

selecting the set of automatic speech recognition parameters associated with the selected speech locale; and

generating, by an automated speech recognizer, the transcription of the subsequent portion of the utterance using the selected set of automatic speech recognition parameters associated with the selected speech locale.

17. The system of claim 16, wherein two or more of the speech locale-specific hotword classifiers are associated with different speech locales, and are associated with the same predefined term.

18. The system of claim 16, wherein two or more of the speech locale-specific hotword classifiers are associated with different speech locales, and are associated with different predefined terms.

19. The system of claim 16, wherein providing the acoustic features from the initial portion of the utterance to multiple speech locale-specific hotword classifiers comprises:

providing the acoustic features from the initial portion of the utterance to a particular speech locale-specific hotword classifier that is associated with multiple other speech locale-specific hotword classifiers; and

determining to provide the acoustic features from the initial portion of the utterance to the multiple other speech locale-specific hotword classifiers based on a result from the particular speech locale-specific hotword classifier.

20. The system of claim 19, wherein the particular speech locale-specific hotword classifier comprises a speech locale-specific hotword classifier for a particular language and the multiple other speech locale-specific hotword classifiers associated with the particular speech locale-specific hotword classifier comprise speech locale-specific hotword classifiers for particular dialects for the particular language.

21. The system of claim 16, wherein an utterance is an utterance input into the device by the user.

22. The system of claim 16, wherein a low power state is a sleep state or hibernation state.

23. A non-transitory computer-readable medium storing instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

receiving an utterance of a user by a device;

obtaining acoustic features from an initial portion of the utterance, wherein the initial portion of the utterance includes a hotword that is associated with waking the device from a low power state;

providing the acoustic features from the initial portion of the utterance to multiple speech locale-specific hotword classifiers of the device that process the acoustic features from the initial portion of the utterance in parallel, wherein each speech locale-specific hotword classifier is a speech recognition process using an acoustic model of a single language or a single dialect of a language and (i) is associated with a respective speech locale, (ii) is configured to classify acoustic features associated with an initial portion of an utterance as either corresponding to, or as not corresponding to, an utterance of a respective predefined term without one or more of transcribing or semantically interpreting acoustic features of any portion of the utterance, and (iii) is associated with a respective set of automatic speech recognition parameters;

obtaining speech locale-specific hotword confidence scores from the speech locale-specific hotword classifiers of the device, wherein each of the speech locale-specific hotword confidence scores obtained from a corresponding speech locale-specific hotword classifier reflects a likelihood that the initial portion of the utterance includes the respective predefined term corresponding to the speech locale-specific hotword classifier;

selecting a speech locale for use in generating a transcription of a subsequent portion of the utterance based on comparing speech locale-specific hotword confidence scores obtained from the multiple speech locale-specific hotword classifiers in response to the parallel processing of the acoustic features from the initial portion of the utterance by the multiple speech locale-specific hotword classifiers

selecting the set of automatic speech recognition parameters associated with the selected speech locale; and

generating, by an automated speech recognizer, the transcription of the subsequent portion of the utterance using the selected set of automatic speech recognition parameters associated with the selected speech locale.

24. The non-transitory computer readable medium of claim 23, wherein an utterance is an utterance input into the device by the user.

25. The non-transitory computer readable medium of claim 23, wherein a low power state is a sleep state or hibernation state.