System and method for adapting automatic speech recognition pronunciation by acoustic model restructuring

• US 8,812,315 B2
• Filed: 10/01/2013
• Issued: 08/19/2014
• Est. Priority Date: 06/09/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Ljolje, Andrej, Conkie, Alistair D., Syrdal, Ann K.
• Primary Examiner(s): COLUCCI, MICHAEL C
• Application Number:
  US14/043,175
• Publication Number:
  US 20140032214A1
• Time in Patent Office: 322 Days
• Field of Search: 704/254, 704/245, 704/9, 704/257, 704/256, 704/255, 704/251, 704/249, 704/246, 704/243, 704/242, 704/236, 704/235, 704/10, 345/473
• US Class Current: 704/235
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/07 to the speaker
• CPC Class Codes: G10L 15/14 using statistical models, e.g. Hidden Markov Models [HMMs] G10L15/18 takes precedence
• CPC Class Codes: G10L 15/187 Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 17/14 Use of phonemic categorisation or speech recognition prior to speaker recognition or verification
• CPC Class Codes: G10L 2015/025 Phonemes, fenemes or fenones being the recognition units

Abstract

Disclosed herein are systems, computer-implemented methods, and computer-readable storage media for recognizing speech by adapting automatic speech recognition pronunciation by acoustic model restructuring. The method identifies an acoustic model and a matching pronouncing dictionary trained on typical native speech in a target dialect. The method collects speech from a new speaker resulting in collected speech and transcribes the collected speech to generate a lattice of plausible phonemes. Then the method creates a custom speech model for representing each phoneme used in the pronouncing dictionary by a weighted sum of acoustic models for all the plausible phonemes, wherein the pronouncing dictionary does not change, but the model of the acoustic space for each phoneme in the dictionary becomes a weighted sum of the acoustic models of phonemes of the typical native speech. Finally the method includes recognizing via a processor additional speech from the target speaker using the custom speech model.

Claims

1. A method comprising:

identifying an acoustic model, wherein the acoustic model is trained on native speech in a target dialect;

transcribing collected speech from a speaker, to yield a lattice of plausible phonemes which depend on a property of the target dialect; and

replacing a phoneme in the acoustic model with a modified phoneme, wherein the modified phoneme is a weighted sum of plausible phonemes in the lattice of plausible phonemes.

2. The method of claim 1, wherein upon replacing the each phoneme in the acoustic model, the acoustic model is a Gaussian mixture model.

3. The method of claim 1, wherein the native speech represents a class of speakers.

4. The method of claim 1, wherein the target dialect comprises one of a regional dialect and a foreign accent.

5. The method of claim 1, wherein the replacing of each phoneme in the acoustic model is based on evaluating an objective function.

6. The method of claim 1, wherein the transcribing of the collected speech is based on one of reference transcriptions and recognition output.

7. The method of claim 1, wherein the replacing of each phoneme in the acoustic model is performed iteratively.

8. The method of claim 1, wherein a weighted average of the plausible phonemes is used instead of the weighted sum of the plausible phonemes.

9. A system comprising:

a processor; and

a computer-readable storage medium having instructions stored which, when executed by the processor, cause the processor to perform operations comprising;

identifying an acoustic model, wherein the acoustic model is trained on native speech in a target dialect;

transcribing collected speech from a speaker, to yield a lattice of plausible phonemes which depend on a property of the target dialect; and

replacing each phoneme in the acoustic model with a modified phoneme, wherein the modified phoneme is a weighted sum of plausible phonemes in the lattice of plausible phonemes.

10. The system of claim 9, wherein upon replacing the each phoneme in the acoustic model, the acoustic model is a Gaussian mixture model.

11. The system of claim 9, wherein the native speech represents a class of speakers.

12. The system of claim 9, wherein the target dialect comprises one of a regional dialect and a foreign accent.

13. The system of claim 9, wherein the replacing of each phoneme phoneme in the acoustic model is based on evaluating an objective function.

14. The system of claim 9, wherein the transcribing of the collected speech is based on one of reference transcriptions and recognition output.

15. The system of claim 9, wherein the replacing of each phoneme in the acoustic model is performed iteratively.

16. The system of claim 9, wherein a weighted average of the plausible phonemes is used instead of the weighted sum of the plausible phonemes.

17. A computer-readable storage device having instructions stored which, when executed by a computing device, cause the computing device to perform operations comprising:

identifying an acoustic model, wherein the acoustic model is trained on native speech in a target dialect;

transcribing collected speech from a speaker, to yield a lattice of plausible phonemes which depend on a property of the target dialect; and

replacing each phoneme in the acoustic model with a modified phoneme, wherein the modified phoneme is a weighted sum of plausible phonemes in the lattice of plausible phonemes.

18. The computer-readable storage device of claim 17, wherein upon replacing the each phoneme in the acoustic model, the acoustic model is a Gaussian mixture model.

19. The computer-readable storage device of claim 17, wherein the native speech represents a class of speakers.