Generating acoustic models of alternative pronunciations for utterances spoken by a language learner in a non-native language

• US 10,068,569 B2
• Filed: 07/01/2013
• Issued: 09/04/2018
• Est. Priority Date: 06/29/2012
• Status: Active Grant

• Current Assignee: Rosetta Stone Limited (Rosetta Stone Incorporated)
• Original Assignee: Rosetta Stone Limited (Rosetta Stone Incorporated)
• Inventors: Stanley, Theban, Hacioglu, Kadri, Siivola, Vesa
• Primary Examiner(s): Lerner, Martin
• Application Number:
  US13/932,506
• Publication Number:
  US 20140006029A1
• Time in Patent Office: 1,891 Days
• Field of Search: 704 2, 704 8, 704 10, 704231, 704270, 704277, 434157, 434185
• CPC Class Codes: G09B 19/06 Foreign languages with audible presentation of material to be studied G09B5/04
• CPC Class Codes: G10L 15/187 Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams
• CPC Class Codes: G10L 15/19 Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules

Abstract

A non-transitory processor-readable medium storing code representing instructions to be executed by a processor includes code to cause the processor to receive acoustic data representing an utterance spoken by a language learner in a non-native language in response to prompting the language learner to recite a word in the non-native language and receive a pronunciation lexicon of the word in the non-native language. The pronunciation lexicon includes at least one alternative pronunciation of the word based on a pronunciation lexicon of a native language of the language learner. The code causes the processor to generate an acoustic model of the at least one alternative pronunciation in the non-native language and identify a mispronunciation of the word in the utterance based on a comparison of the acoustic data with the acoustic model. The code causes the processor to send feedback related to the mispronunciation of the word to the language learner.

Claims

1. A non-transitory processor-readable medium storing code representing instructions to be executed by a processor, the code comprising code to cause the processor to:

receive acoustic data representing an utterance spoken by a language learner in a non-native language in response to prompting the language learner to recite a word in the non-native language;

receive a pronunciation lexicon of the word in the non-native language, the pronunciation lexicon of the word including at least one alternative pronunciation of the word determined based on a pronunciation lexicon of a native language of the language learner, the at least one alternative pronunciation of the word being a phonological error in the non-native language and having a probability greater than a threshold level of being spoken by the language learner when the language learner attempts to recite the word in the non-native language;

generate an acoustic model of the at least one alternative pronunciation of the word from the pronunciation lexicon of the word in the non-native language;

identify a mispronunciation of the word in the utterance based on a comparison of the acoustic data with the at least one alternative pronunciation of the word that is included in the acoustic model; and

send feedback related to the mispronunciation of the word to the language learner, wherein

the acoustic data is first acoustic data, the code to cause the processor to generate the acoustic model includes code to cause the processor to;

generate a maximum likelihood native model based on native data, the native data including a corpus of audio data provided by at least one native speaker,

generate a mix-up native triphone model based on the maximum likelihood native model, and

generate a maximum likelihood non-native model based at least on the mix-up native triphone model and second acoustic data.

2. A non-transitory processor-readable medium storing code representing instructions to be executed by a processor, the code comprising code to cause the processor to:

receive acoustic data representing an utterance spoken by a language learner in a non-native language in response to prompting the language learner to recite a word in the non-native language;

receive a pronunciation lexicon of the word in the non-native language, the pronunciation lexicon of the word including at least one alternative pronunciation of the word determined based on a pronunciation lexicon of a native language of the language learner, the at least one alternative pronunciation of the word being a phonological error in the non-native language and having a probability greater than a threshold level of being spoken by the language learner when the language learner attempts to recite the word in the non-native language;

generate an acoustic model of the at least one alternative pronunciation of the word from the pronunciation lexicon of the word in the non-native language;

identify a mispronunciation of the word in the utterance based on a comparison of the acoustic data with the at least one alternative pronunciation of the word that is included in the acoustic model; and

send feedback related to the mispronunciation of the word to the language learner, wherein

the acoustic data is first acoustic data, the code to cause the processor to generate the acoustic model includes code to cause the processor to;

generate a maximum likelihood non-native model based at least in part on second acoustic data and a mix-up native triphone model generated by a maximum likelihood native model training process, and

generate the acoustic model as a minimum phone error trained acoustic model based on the maximum likelihood non-native model.