Identifying substitute pronunciations

• US 9,747,897 B2
• Filed: 12/17/2013
• Issued: 08/29/2017
• Est. Priority Date: 12/17/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Peng, Fuchun, Beaufays, Francoise, Moreno Mengibar, Pedro J., Strope, Brian Patrick
• Primary Examiner(s): He, Jialong
• Application Number:
  US14/109,316
• Publication Number:
  US 20150170642A1
• Time in Patent Office: 1,351 Days
• CPC Class Codes: G10L 15/005 Language recognition
• CPC Class Codes: G10L 15/187 Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams
• CPC Class Codes: G10L 2015/025 Phonemes, fenemes or fenones being the recognition units
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, including selecting terms; obtaining an expected phonetic transcription of an idealized native speaker of a natural language speaking the terms; receiving audio data corresponding to a particular user speaking the terms in the natural language; obtaining, based on the audio data, an actual phonetic transcription of the particular user speaking the terms in the natural language; aligning the expected phonetic transcription of the idealized native speaker of the natural language with the actual phonetic transcription of the particular user; identifying, based on the aligning, a portion of the expected phonetic transcription that is different than a corresponding portion of the actual phonetic transcription; and based on identifying the portion of the expected phonetic transcription, designating the expected phonetic transcription as a substitute pronunciation for the corresponding portion of the actual phonetic transcription.

Claims

1. A computer-implemented method comprising:

selecting, by a system that includes (i) a confusion matrix manager configured to store, for like-pronunciation groups, phones of expected phonetic transcriptions as substitute pronunciations for corresponding phones of actual phonetic transcriptions in a confusion matrix and (ii) an enhanced speech recognizer configured to obtain, from the confusion matrix manager, using output of an acoustic model associated with the enhanced speech recognizer, the substitute pronunciations before inputting the substitute pronunciations to a language model associated with the enhanced speech recognizer, one or more terms;

obtaining, by the system, an expected phonetic transcription of an idealized native speaker of a natural language speaking the one or more terms;

after obtaining the expected phonetic transcription of the idealized native speaker of the natural language speaking the one or more terms, receiving audio data corresponding to a particular user that is not the idealized native speaker of the natural language speaking the one or more terms in the natural language;

receiving data identifying a like-pronunciation group associated with the particular user;

obtaining, by the system, based on the audio data, an actual phonetic transcription of the particular user that is not the idealized native speaker of the natural language speaking the one or more terms in the natural language;

aligning, by the system, the expected phonetic transcription of the idealized native speaker of the natural language with the actual phonetic transcription of the particular user that is not the idealized native speaker of the natural language;

identifying, by the system, based on aligning the expected phonetic transcription of the idealized native speaker with the actual phonetic transcription of the particular user that is not the idealized native speaker of the natural language, one or more phones of the expected phonetic transcription that is different than one or more corresponding phones of the actual phonetic transcription;

in response to identifying, based on aligning the expected phonetic transcription of the idealized native speaker with the actual phonetic transcription of the particular user that is not the idealized native speaker of the natural language, designating, by the system, the one or more phones of the expected phonetic transcription as a substitute pronunciation for the corresponding phones of the actual phonetic transcription for other terms that (i) are spoken by other users that are also associated with the like-pronunciation group, and (ii) have a respective phonetic transcription that includes the one or more corresponding phones;

obtaining, by the enhanced speech recognizer that is configured to obtain, from the confusion matrix manager, using output of the acoustic model associated with the enhanced speech recognizer, the substitute pronunciations before inputting the substitute pronunciations to the language model associated with the enhanced speech recognizer, a transcription of another term that (i) is spoken by another user that is also associated with the like-pronunciation group, and (ii) has a phonetic transcription that includes the one or more corresponding phones based on the one or more phones of the expected phonetic transcription being designated as a substitute pronunciation for the one or more corresponding phones of the actual phonetic transcription; and

inputting, by the enhanced speech recognizer, using the transcription of another term that (i) is spoken by another user that is also associated with the like-pronunciation group, and (ii) has a phonetic transcription that includes the one or more corresponding phones, the one or more phones of the expected phonetic transcription that is designated as the substitute pronunciation for the one or more corresponding phones of the actual phonetic transcription to the language model associated with the enhanced speech recognizer.

2. The computer-implemented method of claim 1, further comprising storing data of a mapping between the substitute pronunciation and the one or more corresponding phones of the actual phonetic transcription for the term in a database.

3. The computer-implemented method of claim 1, wherein selecting further comprises:

providing, on a user interface, a representation of the one or more terms to the particular user;

receiving a confirmation from the particular user that the representation of the one or more terms correspond to the actual phonetic transcription; and

associating the one or more terms with the actual phonetic transcription.

4. The computer-implemented method of claim 3, wherein associating the one or more terms with the actual phonetic transcription further comprises storing data of a mapping between the one or more terms and the actual phonetic transcription in a database.

5. The computer-implemented method of claim 3, wherein the representation of the one or more terms is provided in response to receiving the audio corresponding to the particular user speaking the one or more terms in the natural language.

6. The computer-implemented method of claim 3, wherein the one or more terms include names of one or more contacts of the particular user.

7. The computer-implemented method of claim 1, further comprising:

generating an identifier based on the like-pronunciation group; and

associating the identifier with the substitute pronunciation.

8. The computer-implemented method of claim 7, further comprising assigning the identifier to the particular user.

9. The computer-implemented method of claim 7, wherein the like-pronunciation group is associated with one or more of a geographic location, a family name, and an origin group.

10. The computer-implemented method of claim 7, further comprising:

identifying a mapping between the substitute pronunciation and an actual pronunciation for the one or more corresponding phones of the actual phonetic transcription;

associating the mapping with the identifier; and

storing the association and the mapping in the confusion matrix.

11. The computer-implemented method of claim 1, wherein obtaining the expected phonetic transcription further comprises:

identifying one or more rules associated with the one or more terms; and

generating the expected phonetic transcription based on the one or more rules.

12. The computer-implemented method of claim 1, wherein obtaining the expected phonetic transcription further comprises:

identifying a phonetic transcription dictionary, the phonetic transcription dictionary including one or more mappings between one or more terms and one or more expected phonetic transcriptions of the one or more terms; and

identifying a particular mapping of the one or more mappings of the phonetic transcription dictionary between the one or more terms and the expected phonetic transcription.

13. The computer-implemented method of claim 1, wherein the one or more phones of the expected phonetic transcription corresponds to a sequence of at least three phonemes.

14. The computer-implemented method of claim 1, further comprising:

receiving additional audio data corresponding to the particular user speaking one or more additional terms in the natural language, the one or more additional terms including the one or more corresponding phones of the actual phonetic transcription;

identifying the substitute pronunciation; and

obtaining, based on the additional audio data and the substitute pronunciation, a text-based transcription of the additional audio data corresponding to the particular user speaking the one or more additional terms in the natural language.

15. The computer-implemented of claim 14, further comprising:

identifying the identifier that is associated with the like-pronunciation group; and

based on identifying the identifier that is associated with the like-pronunciation group, identifying the substitute pronunciation associated with the identifier.

16. The computer-implemented method of claim 15, wherein obtaining the text-based transcription of the additional audio data further comprises:

obtaining, based on the additional audio data, an additional actual phonetic transcription of the particular user speaking the one or more additional terms in the natural language;

identifying an additional one or more phones of the additional actual phonetic transcription that corresponds to the one or more corresponding phones of the actual phonetic transcription;

replacing the additional one or more phones of the additional actual phonetic transcription with the substitute pronunciation; and

based on the replacing, obtaining an updated phonetic transcription of the additional audio data corresponding to the particular user speaking the one or more additional terms in the natural language.

17. The computer-implemented method of claim 16, wherein the text-based transcription is a text-based transcription of the updated phonetic transcription of the additional audio data corresponding to the particular user speaking the one or more terms in the natural language.

18. The computer-implemented method of claim 17, further comprising providing the text-based transcription of the updated phonetic transcription of the additional audio data corresponding to the particular user speaking the one or more terms in the natural language to a user interface manager.

19. The computer-implemented method of claim 17, further comprising providing the text-based transcription of the updated phonetic transcription of the additional audio data corresponding to the particular user speaking the one or more terms in the natural language as a search query to a search engine.

20. The computer-implemented method of claim 1, further comprising designating the expected phonetic transcription as the substitute pronunciation for the one or more phones of the corresponding actual phonetic transcription for a particular group of users, the particular group of users including the particular user.

21. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

selecting, by a system that includes (i) a confusion matrix manager configured to store, for like-pronunciation groups, phones of expected phonetic transcriptions as substitute pronunciations for corresponding phones of actual phonetic transcriptions in a confusion matrix and (ii) an enhanced speech recognizer configured to obtain, from the confusion matrix manager, using output of an acoustic model associated with the enhanced speech recognizer, the substitute pronunciations before inputting the substitute pronunciations to a language model associated with the enhanced speech recognizer, one or more terms;

obtaining, by the system, an expected phonetic transcription of an idealized native speaker of a natural language speaking the one or more terms;

after obtaining the expected phonetic transcription of the idealized native speaker of the natural language speaking the one or more terms, receiving audio data corresponding to a particular user that is not the idealized native speaker of the natural language speaking the one or more terms in the natural language;

receiving data identifying a like-pronunciation group associated with the particular user;

obtaining, by the system, based on the audio data, an actual phonetic transcription of the particular user that is not the idealized native speaker of the natural language speaking the one or more terms in the natural language;

aligning, by the system, the expected phonetic transcription of the idealized native speaker of the natural language with the actual phonetic transcription of the particular user that is not the idealized native speaker of the natural language;

identifying, by the system, based on aligning the expected phonetic transcription of the idealized native speaker with the actual phonetic transcription of the particular user that is not the idealized native speaker of the natural language, one or more phones of the expected phonetic transcription that is different than one or more corresponding phones of the actual phonetic transcription;

in response to identifying, based on aligning the expected phonetic transcription of the idealized native speaker with the actual phonetic transcription of the particular user that is not the idealized native speaker of the natural language, designating, by the system, the one or more phones of the expected phonetic transcription as a substitute pronunciation for the corresponding phones of the actual phonetic transcription for other terms that (i) are spoken by other users that are also associated with the like-pronunciation group, and (ii) have a respective phonetic transcription that includes the one or more corresponding phones;

obtaining, by the enhanced speech recognizer that is configured to obtain, from the confusion matrix manager, using output of the acoustic model associated with the enhanced speech recognizer, the substitute pronunciations before inputting the substitute pronunciations to the language model associated with the enhanced speech recognizer, a transcription of another term that (i) is spoken by another user that is also associated with the like-pronunciation group, and (ii) has a phonetic transcription that includes the one or more corresponding phones based on the one or more phones of the expected phonetic transcription being designated as a substitute pronunciation for the one or more corresponding phones of the actual phonetic transcription; and

inputting, by the enhanced speech recognizer, using the transcription of another term that (i) is spoken by another user that is also associated with the like-pronunciation group, and (ii) has a phonetic transcription that includes the one or more corresponding phones, the one or more phones of the expected phonetic transcription that is designated as the substitute pronunciation for the one or more corresponding phones of the actual phonetic transcription to the language model associated with the enhanced speech recognizer.

22. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

selecting, by a system that includes (i) a confusion matrix manager configured to store, for like-pronunciation groups, phones of expected phonetic transcriptions as substitute pronunciations for corresponding phones of actual phonetic transcriptions in a confusion matrix and (ii) an enhanced speech recognizer configured to obtain, from the confusion matrix manager, using output of an acoustic model associated with the enhanced speech recognizer, the substitute pronunciations before inputting the substitute pronunciations to a language model associated with the enhanced speech recognizer, one or more terms;

obtaining, by the system, an expected phonetic transcription of an idealized native speaker of a natural language speaking the one or more terms;

after obtaining the expected phonetic transcription of the idealized native speaker of the natural language speaking the one or more terms, receiving audio data corresponding to a particular user that is not the idealized native speaker of the natural language speaking the one or more terms in the natural language;

receiving data identifying a like-pronunciation group associated with the particular user;

obtaining, by the system, based on the audio data, an actual phonetic transcription of the particular user that is not the idealized native speaker of the natural language speaking the one or more terms in the natural language;

aligning, by the system, the expected phonetic transcription of the idealized native speaker of the natural language with the actual phonetic transcription of the particular user that is not the idealized native speaker of the natural language;

identifying, by the system, based on aligning the expected phonetic transcription of the idealized native speaker with the actual phonetic transcription of the particular user that is not the idealized native speaker of the natural language, one or more phones of the expected phonetic transcription that is different than one or more corresponding phones of the actual phonetic transcription;

in response to identifying, based on aligning the expected phonetic transcription of the idealized native speaker with the actual phonetic transcription of the particular user that is not the idealized native speaker of the natural language, designating, by the system, the one or more phones of the expected phonetic transcription as a substitute pronunciation for the corresponding phones of the actual phonetic transcription for other terms that (i) are spoken by other users that are also associated with the like-pronunciation group, and (ii) have a respective phonetic transcription that includes the one or more corresponding phones;

obtaining, by the enhanced speech recognizer that is configured to obtain, from the confusion matrix manager, using output of the acoustic model associated with the enhanced speech recognizer, the substitute pronunciations before inputting the substitute pronunciations to the language model associated with the enhanced speech recognizer, a transcription of another term that (i) is spoken by another user that is also associated with the like-pronunciation group, and (ii) has a phonetic transcription that includes the one or more corresponding phones based on the one or more phones of the expected phonetic transcription being designated as a substitute pronunciation for the one or more corresponding phones of the actual phonetic transcription; and

inputting, by the enhanced speech recognizer, using the transcription of another term that (i) is spoken by another user that is also associated with the like-pronunciation group, and (ii) has a phonetic transcription that includes the one or more corresponding phones, the one or more phones of the expected phonetic transcription that is designated as the substitute pronunciation for the one or more corresponding phones of the actual phonetic transcription to the language model associated with the enhanced speech recognizer.