Individualized hotword detection models

• US 10,438,593 B2
• Filed: 07/22/2015
• Issued: 10/08/2019
• Est. Priority Date: 07/22/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Alvarez Guevara, Raziel
• Primary Examiner(s): Colucci, Michael C
• Application Number:
  US14/805,753
• Publication Number:
  US 20170025125A1
• Time in Patent Office: 1,539 Days
• Field of Search: 704239, 704224, 704232, 704233, 704236, 707269, 709206, 709228, 715728
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/07 to the speaker
• CPC Class Codes: G10L 15/075 supervised, i.e. under machine guidance
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 17/04 Training, enrolment or model building
• CPC Class Codes: G10L 17/06 Decision making techniques; Pattern matching strategies
• CPC Class Codes: G10L 17/08 Use of distortion metrics or a particular distance between probe pattern and reference templates
• CPC Class Codes: G10L 17/18 Artificial neural networks; Connectionist approaches
• CPC Class Codes: G10L 17/24 the user being prompted to utter a password or a predefined phrase
• CPC Class Codes: G10L 2015/0638 Interactive procedures
• CPC Class Codes: G10L 2015/088 Word spotting

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for presenting notifications in an enterprise system. In one aspect, a method include actions of obtaining enrollment acoustic data representing an enrollment utterance spoken by a user, obtaining a set of candidate acoustic data representing utterances spoken by other users, determining, for each candidate acoustic data of the set of candidate acoustic data, a similarity score that represents a similarity between the enrollment acoustic data and the candidate acoustic data, selecting a subset of candidate acoustic data from the set of candidate acoustic data based at least on the similarity scores, generating a detection model based on the subset of candidate acoustic data, and providing the detection model for use in detecting an utterance spoken by the user.

Claims

1. A computer-implemented method comprising:

obtaining enrollment acoustic data for a user representing an utterance of a particular, predefined hotword that was spoken by the user during an enrollment process associated with a mobile device;

obtaining a set of candidate acoustic data representing utterances of the same, particular predefined hotword that were previously-spoken by other users;

after receiving the enrollment acoustic data, selecting, from the set of candidate acoustic data, a subset of the candidate acoustic data that is acoustically similar to the enrollment acoustic data;

training a neural network-based, hotword detection model to generate a neural network-based, hotword detection model that is customized for the user, wherein the training uses (1) the enrollment acoustic data for the user and (2) the selected subset of the candidate acoustic data that is acoustically similar to the enrollment acoustic data as examples of acceptable utterances of the particular, predefined hotword for the user; and

providing the neural network-based, hotword detection model that is customized for the user for use in detecting an utterance of the particular, predefined hotword that is subsequently spoken by the user.

2. The method of claim 1, wherein obtaining enrollment acoustic data representing an utterance of a particular, predefined hotword that was spoken by a user during an enrollment process associated with a mobile device comprises:

obtaining enrollment acoustic data for multiple utterances of the particular, predefined hotword spoken by the user.

3. The method of claim 1, wherein obtaining a set of candidate acoustic data representing utterances of the same, particular, predefined hotword that was previously spoken by other users comprises:

determining the utterance is of the particular, predefined hotword; and

identifying candidate acoustic data representing utterances of the particular, predefined hotword spoken by other users.

4. The method of claim 1, wherein selecting, from the set of candidate acoustic data, a subset of the candidate acoustic data that is acoustically similar to the enrollment acoustic data comprises determining, for each candidate acoustic data of the set of candidate acoustic data, that a similarity score that represents a similarity between the enrollment acoustic data and the candidate acoustic data satisfies a similarity threshold, comprising:

determining an acoustic distance between the enrollment acoustic data and the candidate acoustic data; and

determining the similarity score based on the acoustic distance.

5. The method of claim 1, wherein selecting, from the set of candidate acoustic data, a subset of the candidate acoustic data that is acoustically similar to the enrollment acoustic data comprises:

determining similarity scores for the candidate acoustic data based on (i) demographic information of the other user that spoke the utterance represented by the candidate acoustic data, including a gender of the other user and (ii) demographic information of the user that spoke the utterance represented by the enrollment acoustic data, including a gender of the user that spoke the utterance represented by the enrollment acoustic data; and

selecting the subset of the candidate acoustic data based at least in part on the similarity scores.

6. The method of claim 1, wherein selecting a subset of acoustically similar, candidate acoustic data from the set of candidate acoustic data includes selecting a predetermined number of candidate acoustic data.

7. The method of claim 1, further comprising:

detecting an utterance of the particular, predefined hotword using the neural network-based, hotword detection model that is customized for the user.

8. The method of claim 1, wherein providing the neural network-based, hotword detection model that is customized for the user comprises providing the neural network-based, hotword detection model that is customized for the user to a client device associated with the user.

9. The method of claim 8, wherein providing the neural network-based, hotword detection model that is customized for the user to the client device associated with the user comprises providing, over a communication network, the neural network-based, hotword detection model that is customized for the user to a client device that provided the enrollment acoustic data for the user.

10. The method of claim 1, wherein the selecting and training are performed in response to obtaining the enrollment acoustic data for the user, and wherein the selected subset of the candidate acoustic data is a proper subset of the candidate acoustic data;

wherein selecting the subset of the candidate acoustic data comprises selecting each particular candidate acoustic data of the subset based on determining that a similarity score that represents an acoustic similarity between the enrollment acoustic data and the particular candidate acoustic data exceeds a threshold similarity score; and

wherein training the neural network-based, hotword detection model to generate the neural network-based, hotword detection model that is customized for the user comprises;

training the neural network-based, hotword detection model to detect both (1) the enrollment acoustic data for the user and (2) the selected subset of the candidate acoustic data as valid instances of the particular, predefined hotword for the user even though the neural network-based, hotword detection model generated through the training is customized for accepting utterances of the hotword by the user and the selected subset of the candidate acoustic data represents utterances of other users that are different from the user.

11. The method of claim 1, wherein selecting, from the set of candidate acoustic data, a subset of the candidate acoustic data that is acoustically similar to the enrollment acoustic data comprises:

determining similarity scores for the candidate acoustic data based on (i) demographic information of the other user that spoke the utterance represented by the candidate acoustic data, including a location of the other user and (ii) demographic information of the user that spoke the utterance represented by the enrollment acoustic data, including a location of the user that spoke the utterance represented by the enrollment acoustic data; and

selecting the subset of the candidate acoustic data based at least in part on the similarity scores.

12. A system comprising:

one or more computers; and

one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

obtaining enrollment acoustic data for a user representing an utterance of a particular, predefined hotword that was spoken by the user during an enrollment process associated with a mobile device;

obtaining a set of candidate acoustic data representing utterances of the same, particular predefined hotword that were previously-spoken by other users;

after receiving the enrollment acoustic data, selecting, from the set of candidate acoustic data, a subset of the candidate acoustic data that is acoustically similar to the enrollment acoustic data;

training a neural network-based, hotword detection model to generate a neural network-based, hotword detection model that is customized for the user, wherein the training uses (1) the enrollment acoustic data for the user and (2) the selected subset of the candidate acoustic data that is acoustically similar to the enrollment acoustic data as examples of acceptable utterances of the particular, predefined hotword for the user; and

providing the neural network-based, hotword detection model that is customized for the user for use in detecting an utterance of the particular, predefined hotword that is subsequently spoken by the user.

13. The system of claim 12, wherein obtaining enrollment acoustic data representing an utterance of a particular, predefined hotword that was spoken by a user during an enrollment process associated with a mobile device comprises:

obtaining enrollment acoustic data for multiple utterances of the particular, predefined hotword spoken by the user.

14. The system of claim 12, wherein obtaining a set of candidate acoustic data representing utterances of the same, particular, predefined hotword that was previously spoken by other users comprises:

determining the utterance is of the particular, predefined hotword; and

identifying candidate acoustic data representing utterances of the particular, predefined hotword spoken by other users.

15. The system of claim 12, wherein selecting, from the set of candidate acoustic data, a subset of the candidate acoustic data that is acoustically similar to the enrollment acoustic data comprises determining, for each candidate acoustic data of the set of candidate acoustic data, that a similarity score that represents a similarity between the enrollment acoustic data and the candidate acoustic data satisfies a similarity threshold, comprising:

determining an acoustic distance between the enrollment acoustic data and the candidate acoustic data; and

determining the similarity score based on the acoustic distance.

16. The system of claim 12, wherein selecting, from the set of candidate acoustic data, a subset of the candidate acoustic data that is acoustically similar to the enrollment acoustic data comprises:

determining similarity scores for the candidate acoustic data based on (i) demographic information of the other user that spoke the utterance represented by the candidate acoustic data, including an age of the other user and (ii) demographic information of the user that spoke the utterance represented by the enrollment acoustic data, including an age of the user that spoke the utterance represented by the enrollment acoustic data; and

selecting the subset of the candidate acoustic data based at least in part on the similarity scores.

17. The system of claim 12, the operations further comprising:

detecting an utterance of the particular, predefined hotword using the neural network-based, hotword detection model.

18. One or more non-transitory computer-readable media storing instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

obtaining enrollment acoustic data for a user representing an utterance of a particular, predefined hotword that was spoken by the user during an enrollment process associated with a mobile device;

obtaining a set of candidate acoustic data representing utterances of the same, particular predefined hotword that were previously-spoken by other users;

after receiving the enrollment acoustic data, selecting, from the set of candidate acoustic data, a subset of the candidate acoustic data that is acoustically similar to the enrollment acoustic data;

training a neural network-based, hotword detection model to generate a neural network-based, hotword detection model that is customized for the user, wherein the training uses (1) the enrollment acoustic data for the user and (2) the selected subset of the candidate acoustic data that is acoustically similar to the enrollment acoustic data as examples of acceptable utterances of the particular, predefined hotword for the user; and

providing the neural network-based, hotword detection model that is customized for the user for use in detecting an utterance of the particular, predefined hotword that is subsequently spoken by the user.

19. The one or more non-transitory computer-readable media of claim 18, wherein obtaining enrollment acoustic data representing an utterance of a particular, predefined hotword that was spoken by a user during an enrollment process associated with a mobile device comprises:

obtaining enrollment acoustic data for multiple utterances of the particular, predefined hotword spoken by the user.

20. The one or more non-transitory computer-readable media of claim 18, wherein obtaining a set of candidate acoustic data representing utterances of the same, particular, predefined hotword that was previously spoken by other users comprises:

determining the utterance is of the particular, predefined hotword; and

identifying candidate acoustic data representing utterances of the particular, predefined hotword spoken by other users.