Hotword recognition

• US 9,747,926 B2
• Filed: 11/17/2015
• Issued: 08/29/2017
• Est. Priority Date: 10/16/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Sharifi, Matthew, Foerster, Jakob Nicolaus
• Primary Examiner(s): GUERRA-ERAZO, EDGAR X
• Application Number:
  US14/943,287
• Publication Number:
  US 20170110144A1
• Time in Patent Office: 651 Days
• Field of Search: 704231, 704235, 704246, 704270, 7042701, 704271, 704275, 704276, 704277
• CPC Class Codes: G06F 21/31 User authentication
• CPC Class Codes: G06F 21/32 using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/08 Speech classification or search
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 17/08 Use of distortion metrics or a particular distance between probe pattern and reference templates
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 25/51 for comparison or discrimination

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for receiving audio data corresponding to an utterance, determining that the audio data corresponds to a hotword, generating a hotword audio fingerprint of the audio data that is determined to correspond to the hotword, comparing the hotword audio fingerprint to one or more stored audio fingerprints of audio data that was previously determined to correspond to the hotword, detecting whether the hotword audio fingerprint matches a stored audio fingerprint of audio data that was previously determined to correspond to the hotword based on whether the comparison indicates a similarity between the hotword audio fingerprint and one of the one or more stored audio fingerprints that satisfies a predetermined threshold, and in response to detecting that the hotword audio fingerprint matches a stored audio fingerprint, disabling access to a computing device into which the utterance was spoken.

Claims

1. A computer-implemented method comprising:

receiving audio data corresponding to an utterance that is received while a computing device is operating in a lock mode, the computing device being configured to exit the lock mode based on determining that the audio data corresponds to a hotword;

determining that the audio data corresponds to the hotword;

generating a hotword audio fingerprint of the audio data that is determined to correspond to the hotword;

determining a similarity between the hotword audio fingerprint and one or more stored audio fingerprints of audio data that was previously determined to correspond to the hotword;

detecting whether the hotword audio fingerprint matches a stored audio fingerprint of audio data that was previously determined to correspond to the hotword based on whether the similarity between the hotword audio fingerprint and one of the one or more stored audio fingerprints satisfies a predetermined threshold; and

in response to detecting that the hotword audio fingerprint matches a stored audio fingerprint, preventing the computing device into which the utterance was spoken from exiting the lock mode despite determining that the audio data corresponds to the hotword.

2. The computer-implemented method of claim 1, wherein determining that the audio data corresponds to a hotword comprises:

identifying one or more acoustic features of the audio data;

comparing the one or more acoustic features of the audio data to one or more acoustic features associated with one or more hotwords stored in a database; and

determining that the audio data corresponds to one of the one or more hotwords stored in the database based on the comparison of the one or more acoustic features of the audio data to the one or more acoustic features associated with one or more hotwords stored in the database.

3. The computer-implemented method of claim 1, comprising:

receiving additional audio data corresponding to an additional utterance;

identifying speaker-identification d-vectors using the additional audio data;

determining a similarity between the speaker-identification d-vectors from the additional audio data and hotword d-vectors from the audio data corresponding to the utterance;

detecting whether the audio data corresponding to the hotword matches the additional audio data based on whether the similarity between the speaker-identification d-vectors from the additional audio data and the hotword d-vectors from the audio data corresponding to the utterance satisfies a particular threshold; and

in response to detecting that the audio data corresponding to the hotword does not match the additional audio data, disabling access to the computing device.

4. The computer-implemented method of claim 1, wherein the hotword is a particular term that triggers semantic interpretation of an additional term of one or more terms that follow the particular term.

5. The computer-implemented method of claim 1, comprising:

receiving additional audio data corresponding to a voice command or query; and

determining a type of the voice command or query,

wherein the predetermined threshold is adjusted based on the type of the voice command or query.

6. The computer-implemented method of claim 1, wherein determining that the audio data corresponds to a hotword comprises:

determining that an initial portion of the audio data corresponds to an initial portion of the hotword; and

in response to determining that the initial portion of the audio data corresponds to the initial portion of the hotword, causing one of a plurality of unique ultrasonic audio samples to be outputted after the initial portion of the audio data is received.

7. The computer-implemented method of claim 6, comprising:

determining that the received audio data comprises audio data corresponding to one of the plurality of unique ultrasonic audio samples; and

in response to determining that the received audio data comprises audio data corresponding to one of the plurality of unique ultrasonic audio samples, disabling access to the computing device.

8. A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

receiving audio data corresponding to an utterance that is received while a computing device is operating in a lock mode, the computing device being configured to exit the lock mode based on determining that the audio data corresponds to a hotword;

determining that the audio data corresponds to the hotword;

generating a hotword audio fingerprint of the audio data that is determined to correspond to the hotword;

determining a similarity between the hotword audio fingerprint and one or more stored audio fingerprints of audio data that was previously determined to correspond to the hotword;

detecting whether the hotword audio fingerprint matches a stored audio fingerprint of audio data that was previously determined to correspond to the hotword based on whether the similarity between the hotword audio fingerprint and one of the one or more stored audio fingerprints satisfies a predetermined threshold; and

in response to detecting that the hotword audio fingerprint matches a stored audio fingerprint, preventing the computing device into which the utterance was spoken from exiting the lock mode despite determining that the audio data corresponds to the hotword.

9. The system of claim 8, wherein determining that the audio data corresponds to a hotword comprises:

identifying one or more acoustic features of the audio data;

comparing the one or more acoustic features of the audio data to one or more acoustic features associated with one or more hotwords stored in a database; and

determining that the audio data corresponds to one of the one or more hotwords stored in the database based on the comparison of the one or more acoustic features of the audio data to the one or more acoustic features associated with one or more hotwords stored in the database.

10. The system of claim 8, wherein the operations comprise:

receiving additional audio data corresponding to an additional utterance;

identifying speaker-identification d-vectors using the additional audio data;

determining a similarity between the speaker-identification d-vectors from the additional audio data and hotword d-vectors from the audio data corresponding to the utterance;

detecting whether the audio data corresponding to the hotword matches the additional audio data based on whether the similarity between the speaker-identification d-vectors from the additional audio data and the hotword d-vectors from the audio data corresponding to the utterance satisfies a particular threshold; and

in response to detecting that the audio data corresponding to the hotword does not match the additional audio data, disabling access to the computing device.

11. The system of claim 8, wherein the hotword is a particular term that triggers semantic interpretation of an additional term of one or more terms that follow the particular term.

12. The system of claim 8, wherein the operations comprise:

receiving additional audio data corresponding to a voice command or query; and

determining a type of the voice command or query,

wherein the predetermined threshold is weighted based on the type of the voice command or query.

13. The system of claim 8, wherein determining that the audio data corresponds to a hotword comprises:

determining that an initial portion of the audio data corresponds to an initial portion of the hotword; and

in response to determining that the initial portion of the audio data corresponds to the initial portion of the hotword, causing one of a plurality of unique ultrasonic audio samples to be outputted after the initial portion of the audio data is received.

14. The system of claim 13, wherein the operations comprise:

determining that the received audio data comprises audio data corresponding to one of the plurality of unique ultrasonic audio samples; and

in response to determining that the received audio data comprises audio data corresponding to one of the plurality of unique ultrasonic audio samples, disabling access to the computing device.

15. A computer-readable storage device storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

receiving audio data corresponding to an utterance that is received while a computing device is operating in a lock mode, the computing device being configured to exit the lock mode based on determining that the audio data corresponds to a hotword;

determining that the audio data corresponds to the hotword;

generating a hotword audio fingerprint of the audio data that is determined to correspond to the hotword;

determining a similarity between the hotword audio fingerprint and one or more stored audio fingerprints of audio data that was previously determined to correspond to the hotword;

detecting whether the hotword audio fingerprint matches a stored audio fingerprint of audio data that was previously determined to correspond to the hotword based on whether the similarity between the hotword audio fingerprint and one of the one or more stored audio fingerprints satisfies a predetermined threshold; and

in response to detecting that the hotword audio fingerprint matches a stored audio fingerprint, preventing the computing device into which the utterance was spoken from exiting the lock mode despite determining that the audio data corresponds to the hotword.

16. The computer-readable storage device of claim 15, wherein determining that the audio data corresponds to a hotword comprises:

identifying one or more acoustic features of the audio data;

comparing the one or more acoustic features of the audio data to one or more acoustic features associated with one or more hotwords stored in a database; and

determining that the audio data corresponds to one of the one or more hotwords stored in the database based on the comparison of the one or more acoustic features of the audio data to the one or more acoustic features associated with one or more hotwords stored in the database.

17. The computer-readable storage device of claim 15, wherein the operations comprise:

receiving additional audio data corresponding to an additional utterance;

identifying speaker-identification d-vectors using the additional audio data;

determining a similarity between the speaker-identification d-vectors from the additional audio data and hotword d-vectors from the audio data corresponding to the utterance;

detecting whether the audio data corresponding to the hotword matches the additional audio data based on whether the similarity between the speaker-identification d-vectors from the additional audio data and the hotword d-vectors from the audio data corresponding to the utterance satisfies a particular threshold; and

in response to detecting that the audio data corresponding to the hotword does not match the additional audio data, disabling access to the computing device.

18. The computer-readable storage device of claim 15, wherein the hotword is a particular term that triggers semantic interpretation of an additional term of one or more terms that follow the particular term.

19. The computer-readable storage device of claim 15, wherein the operations comprise:

receiving additional audio data corresponding to a voice command or query;

determining a type of the voice command or query,

wherein the predetermined threshold is weighted based on the type of the voice command or query.

20. The computer-readable storage device of claim 15, wherein determining that the audio data corresponds to a hotword comprises:

determining that an initial portion of the audio data corresponds to an initial portion of the hotword;

in response to determining that the initial portion of the audio data corresponds to the initial portion of the hotword, causing one of a plurality of unique ultrasonic audio samples to be outputted after the initial portion of the audio data is received;

determining that the received audio data comprises audio data corresponding to one of the plurality of unique ultrasonic audio samples; and

in response to determining that the received audio data comprises audio data corresponding to one of the plurality of unique ultrasonic audio samples, disabling access to the computing device.