Pronunciation learning through correction logs

• US 9,589,562 B2
• Filed: 02/21/2014
• Issued: 03/07/2017
• Est. Priority Date: 02/21/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Kibre, Nicholas, Ozertem, Umut, Parthasarathy, Sarangarajan, Al Bawab, Ziad
• Primary Examiner(s): Yang, Qian
• Application Number:
  US14/186,476
• Publication Number:
  US 20150243278A1
• Time in Patent Office: 1,110 Days
• US Class Current: 1/1
• CPC Class Codes: G10L 15/01 Assessment or evaluation of speech recognition systems
• CPC Class Codes: G10L 15/075 supervised, i.e. under machine guidance
• CPC Class Codes: G10L 15/187 Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams
• CPC Class Codes: G10L 2015/025 Phonemes, fenemes or fenones being the recognition units
• CPC Class Codes: G10L 2015/225 Feedback of the input speech

Abstract

A new pronunciation learning system for dynamically learning new pronunciations assisted by user correction logs. The user correction logs provide a record of speech recognition events and subsequent user behavior that implicitly confirms or rejects the recognition result and/or shows the user'"'"'s intended words by via subsequent input. The system analyzes the correction logs and distills them down to a set of words which lack acceptable pronunciations. Hypothetical pronunciations, constrained by spelling and other linguistic knowledge, are generated for each of the words. Offline recognition determines the hypothetical pronunciations with a good acoustical match to the audio data likely to contain the words. The matching pronunciations are aggregated and adjudicated to select new pronunciations for the words to improve general or personalized recognition models.

Claims

1. A method for dynamically learning new pronunciations for speech recognition assisted by subsequent user inputs, the method comprising:

determining that a task initiated by a spoken utterance was not completed successfully;

determining that a recognition result for the spoken utterance includes a misrecognized word based on the determination that the task initiated by the recognition result was not completed successfully;

determining that a subsequent task initiated by subsequent user inputs was completed successfully;

associating the spoken utterance with the subsequent user inputs based on similarity between the spoken utterance and the subsequent user inputs;

after associating the spoken utterance with the subsequent user inputs based on similarity between the spoken utterance and the subsequent user inputs, generating hypothetical pronunciations for the misrecognized word in the spoken utterance based on a predicted intended word derived from the associated subsequent user inputs;

recognizing the spoken utterance using a language model containing the hypothetical pronunciations to find matching hypothetical pronunciations; and

accepting a new pronunciation for the predicted intended word from the matching hypothetical pronunciations.

2. The method of claim 1 further comprising the act of determining that a spoken utterance was misrecognized based on a user confirmation.

3. The method of claim 1 further comprising the act of determining that subsequent user inputs correspond to spoken utterances.

4. The method of claim 3 wherein the act of determining that subsequent user inputs correspond to spoken utterances containing the misrecognized word further comprises the acts of:

determining that tasks initiated by spoken utterances were not completed successfully; and

determining that tasks initiated by subsequent user inputs were completed successfully; and

pairing tasks initiated by spoken utterances that were not completed successfully with tasks initiated by subsequent user inputs that were completed successfully into successive input pairs.

5. The method of claim 4 wherein the act of determining that subsequent user inputs correspond to spoken utterances containing the misrecognized word further comprises the acts of determining that successive input pairs correspond to spoken utterances containing the misrecognized word and subsequent user inputs correcting the misrecognized word using at least one of session features, lexical features, and phonetic features.

6. The method of claim 4 wherein the act of determining that subsequent user inputs correspond to spoken utterances containing the misrecognized word further comprises the acts of determining that subsequent user inputs are closely related in time to spoken utterances containing the misrecognized word.

7. The method of claim 4 wherein the act of determining that subsequent user inputs correspond to spoken utterances containing the misrecognized word further comprises the acts of determining that subsequent user inputs are lexically similar to spoken utterances containing the misrecognized word.

8. The method of claim 4 wherein the act of determining that subsequent user inputs correspond to spoken utterances containing the misrecognized word further comprises the acts of determining that subsequent user inputs are phonetically similar to spoken utterances containing the misrecognized word.

9. The method of claim 3 wherein the act of accepting a new pronunciation for the predicted intended word from the matching hypothetical pronunciations further comprises the act of accepting the most frequent matching hypothetical pronunciation as the new pronunciation for the predicted intended word.

10. The method of claim 9 wherein the act of selecting a new pronunciation for the predicted intended word from the matching hypothetical pronunciations further comprises the acts of:

determining how often phonemes occur in the matching hypothetical pronunciations for each grapheme; and

accepting the most frequent matching hypothetical pronunciation as the new pronunciation if the most frequent phonemes for each common phone occur in the most frequent matching hypothetical pronunciation.

11. The method of claim 1 wherein the act of generating hypothetical pronunciations for misrecognized word in spoken utterances based on a predicted intended word from corresponding subsequent user inputs further comprises the acts of selecting possible phonemes corresponding to graphemes in the predicted intended word according to linguistic knowledge.

12. The method of claim 1 further comprising the act of collecting audio of recognized spoken utterances, subsequent user inputs following spoken utterances, and information indicating whether tasks initiated by the spoken utterances and subsequent user inputs were successful.

13. A speech recognition system for assisted dynamic learning of new pronunciations for user input, the speech recognition system comprising:

at least one processor;

a memory connected to the at least one processor; and

a recognition event store encoded on the memory storing recognition event data for tasks initiated by spoken utterances, the recognition event data including audio data of the spoken utterances, recognition results obtained by decoding the spoken utterances, subsequent user inputs, and indicators of whether outcomes of tasks initiated based on the recognition results and the subsequent user inputs were accepted or rejected by users;

wherein the memory contains computer executable instructions for;

an event classifier configured to classify recognition results as misrecognized spoken utterances based on determining that tasks initiated by the recognition result were not completed successfully based on an indication that outcomes of the tasks were not accepted, determining that subsequent tasks initiated based on subsequent user inputs were completed successfully based on an indication that outcomes of the subsequent tasks being accepted, and determining that a subsequent user input and recognition result pair from a single source have significant similarity, and configured to identify misrecognized portions of the recognition results based on the subsequent user inputs;

a pronunciation generator configured to generate hypothetical pronunciations for the identified misrecognized portions using the corresponding portions of the subsequent user inputs after the event classifier has classified recognition results as misrecognized spoken utterances based on determining that tasks initiated by the recognition result were not completed successfully based on an indication that outcomes of the tasks were not accepted, determining that subsequent tasks initiate based on subsequent user inputs were completed successfully based on an indication that outcomes of the subsequent tasks being accepted, and determining that a subsequent user input and recognition result pair from a single source have significant similarity;

a speech recognizer configured to match hypothetical pronunciations with the audio data of spoken utterances that produced recognition results classified as misrecognized spoken utterances; and

an aggregation adjudicator configured to select new pronunciations for the misrecognized words from the matching pronunciations.

14. The speech recognition system of claim 13 wherein the aggregation adjudicator is further configured to aggregate the matching pronunciations and select a new pronunciation based on the occurrence frequency of the matching pronunciations.

15. The speech recognition system of claim 14 wherein the aggregation adjudicator is configured to confirm a selection of the new pronunciation by determining that the new pronunciation is constructed from the most popular phonetic units for each common phone in the aggregated matching pronunciations.

16. The speech recognition system of claim 13 wherein the event classifier uses temporal proximity of the subsequent user input and recognition result pair and at least one lexical or phonetic similarity measures to determine that the subsequent user input is a corrective input for a misrecognized recognition result.

17. The speech recognition system of claim 13 wherein the pronunciation generator is configured to identify possible graphic and phonetic combinations for the misrecognized portions using the corresponding portions of the subsequent user inputs, create a model to score the possible graphic and phonetic combinations using linguistic knowledge, and build a custom grammar from a set of the possible graphic and phonetic combinations selected based on scores obtained using the model.

18. A system for learning new pronunciations, the system comprising:

at least one processor; and

a memory connected to the at least one processor and containing computer executable instructions which, when executed by the at least one processor, cause the at least one processor to perform a method for assisted dynamic learning of new pronunciations for user input, the method comprising;

determining that a spoken utterance has been misrecognized based on implicit indications that a task initiated based on the spoken utterance was unsuccessful and a task initiated based on a subsequent user input was successful and that the subsequent user input is similar to the spoken utterance;

after determining that a spoken utterance has been misrecognized based on implicit indications that a task initiated based on the spoken utterance was unsuccessful and a task initiated based on a subsequent user input was successful and that the subsequent user input is similar to the spoken utterance, identifying a misrecognized word based on a difference in linguistic unit between the spoken utterance and the subsequent user input;

after identifying a misrecognized word based on a difference in linguistic unit between the spoken utterance and the subsequent user input, generating hypothetical pronunciations for the identified misrecognized word in spoken utterances based on a similar word from corresponding subsequent user inputs;

recognizing misrecognized spoken utterances using a language model containing the hypothetical pronunciations to find matching hypothetical pronunciations; and

accepting a new pronunciation for the similar word from the matching hypothetical pronunciations.

19. The system of claim 18 wherein the method further comprises the act of adding the accepted pronunciation to a general language model used to provide real-time speech recognition to multiple users.

20. The system of claim 18 wherein the method further comprises the act of adding the accepted pronunciation to a personal language model used to provide real-time speech recognition to a single user.