Multiple pass automatic speech recognition methods and apparatus

• US 9,940,927 B2
• Filed: 08/23/2013
• Issued: 04/10/2018
• Est. Priority Date: 08/23/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Georges, Munir Nikolai Alexander, Kanthak, Stephan
• Primary Examiner(s): COLUCCI, MICHAEL C
• Application Number:
  US14/364,156
• Publication Number:
  US 20150058018A1
• Time in Patent Office: 1,691 Days
• Field of Search: 704257, 704231, 704235, 704200, 704243, 704246
• CPC Class Codes: G10L 15/08 Speech classification or search
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 15/32 Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems

Abstract

In some aspects, a method of recognizing speech that comprises natural language and at least one word specified in at least one domain-specific vocabulary is provided. The method comprises performing a first speech processing pass comprising identifying, in the speech, a first portion including the natural language and a second portion including the at least one word specified in the at least one domain-specific vocabulary, and recognizing the first portion including the natural language. The method further comprises performing a second speech processing pass comprising recognizing the second portion including the at least one word specified in the at least one domain-specific vocabulary.

Claims

1. At least one non-transitory computer readable medium storing instructions that, when executed by at least one processor, cause at least one computer to perform a method of recognizing speech that comprises natural language and at least one word specified in at least one domain-specific vocabulary including a first domain-specific vocabulary, the method comprising:

performing a first speech processing pass comprising;

identifying, in the speech, a first portion including the natural language and a second portion including the at least one word specified in the at least one domain-specific vocabulary, the identifying comprising using at least one filler model including a first filler model to assist in identifying the second portion,

wherein the first filler model is trained, at least in part, on data in the first domain-specific vocabulary,

wherein the at least one filler model comprises at least a first acoustic model, at least one morpheme model, at least one pronunciation model and/or at least one phoneme model,

wherein identifying the second portion does not include attempting to recognize words in the second portion using a language model comprising language statistics for words in the at least one domain-specific vocabulary; and

recognizing the first portion including the natural language using at least a second acoustic model and at least one language model; and

performing a second speech processing pass comprising;

recognizing the second portion including the at least one word specified in the at least one domain-specific vocabulary.

2. The at least one non-transitory computer readable medium of claim 1, wherein performing the first speech processing pass comprises:

using at least one language model representing probabilities of transitioning from one or more words to one or more classes corresponding to the at least one domain-specific vocabulary and/or representing probabilities of transitioning from the one or more classes to the one or more words to recognize the first portion including the natural language.

3. The at least one non-transitory computer readable medium of claim 2, wherein performing the first speech processing pass results in at least one recognized word from the first portion including the natural language and at least one tag associated with the at least one domain-specific vocabulary to identify the second portion, and wherein performing the second speech processing pass comprises recognizing the second portion identified by the at least one tag.

4. The at least one non-transitory computer readable medium of claim 2, wherein performing the second speech processing pass comprises recognizing a portion having words specified in the first domain-specific vocabulary using at least one grammar associated with the first domain-specific vocabulary.

5. The at least one non-transitory computer readable medium of claim 2, wherein the at least one filler model includes at least one phoneme model to determine phoneme likelihoods in the second portion, the at least one phoneme model including a phoneme loop model and/or a phoneme N-gram model.

6. The at least one non-transitory computer readable medium of claim 1, wherein the first speech processing pass is performed at a first location on a network and the second speech processing pass is performed at a second location on the network, wherein the first location is different from the second location.

7. The at least one non-transitory computer readable medium of claim 2, wherein the second portion including the at least one word specified in the at least one domain-specific vocabulary includes a third portion including at least one word specified in a first domain-specific vocabulary and a fourth portion including at least one word specified in a second domain-specific vocabulary, wherein the at least one filler model further includes a second filler model trained at least in part on data in the second domain-specific vocabulary, and wherein performing the second speech processing pass comprises:

using a first domain-specific speech recognizer adapted to recognize words in the first domain-specific vocabulary to recognize the third portion identified at least in part using the first filler model; and

using a second domain-specific recognizer adapted to recognize words in the second domain-specific vocabulary to recognize the fourth portion identified at least in part using the second filler model.

8. A system comprising at least one computer for recognizing speech that comprises natural language and at least one word specified in at least one domain-specific vocabulary including a first domain-specific vocabulary, the system comprising:

at least one computer readable medium storing instructions; and

at least one processor programmed by executing the instructions to perform;

a first speech processing pass comprising;

identifying, in the speech, a first portion including the natural language and a second portion including the at least one word specified in the at least one domain-specific vocabulary, the identifying comprising using at least one filler model including a first filler model to assist in identifying the second portion,

wherein the first filler model is trained, at least in part, on data in the first domain-specific vocabulary,

wherein the at least one filler model comprises at least a first acoustic model, at least one morpheme model, at least one pronunciation model and/or at least one phoneme model,

wherein identifying the second portion does not include attempting to recognize words in the second portion using a language model comprising language statistics for words in the at least one domain-specific vocabulary; and

recognizing the first portion including the natural language using at least a second acoustic model and at least one language model; and

a second speech processing pass comprising;

recognizing the second portion including the at least one word specified in the at least one domain-specific vocabulary.

9. The system of claim 8, wherein the at least one processor is programmed to perform the first speech processing pass further comprising:

using at least one language model representing probabilities of transitioning from one or more words to one or more classes corresponding to the at least one domain-specific vocabulary and/or representing probabilities of transitioning from the one or more classes to the one or more words to recognize the first portion including the natural language.

10. The system of claim 9, wherein performing the first speech processing pass results in at least one recognized word from the first portion including the natural language and at least one tag associated with the at least one domain-specific vocabulary to identify the second portion, and wherein performing the second speech processing pass comprises recognizing the second portion identified by the at least one tag.

11. The system of claim 9, wherein the at least one processor is programmed to perform the second speech processing pass comprising recognizing a portion having words specified in the first domain-specific vocabulary using at least one grammar associated with the first domain-specific vocabulary.

12. The system of claim 9, wherein the at least one filler model includes at least one phoneme model to determine phoneme likelihoods in the second portion, the at least one phoneme model including a phoneme loop model and/or a phoneme N-gram model.

13. The system of claim 8, wherein the first speech processing pass is performed at a first location on a network and the second speech processing pass is performed at a second location on the network, wherein the first location is different from the second location.

14. A method for recognizing speech that comprises natural language and at least one word specified in at least one domain-specific vocabulary including a first domain-specific vocabulary, the method comprising:

performing a first speech processing pass comprising;

identifying, in the speech, a first portion including the natural language and a second portion including the at least one word specified in the at least one domain-specific vocabulary, the identifying comprising using at least one filler model including a first filler model to assist in identifying the second portion,

wherein the first filler model is trained, at least in part, on data in the first domain-specific vocabulary,

wherein the at least one filler model comprises at least a first acoustic model, at least one morpheme model, at least one pronunciation model and/or at least one phoneme model,

wherein identifying the second portion does not include attempting to recognize words in the second portion using a language model comprising language statistics for words in the at least one domain-specific vocabulary; and

recognizing the first portion including the natural language using at least a second acoustic model and at least one language model; and

performing a second speech processing pass comprising;

recognizing the second portion including the at least one word specified in the at least one domain-specific vocabulary.

15. The method of claim 14, wherein performing the first speech processing pass comprises:

using at least one language model representing probabilities of transitioning from one or more words to one or more classes corresponding to the at least one domain-specific vocabulary and/or representing probabilities of transitioning from the one or more classes to the one or more words to recognize the first portion including the natural language.

16. The method of claim 15, wherein performing the first speech processing pass results in at least one recognized word from the first portion including the natural language and at least one tag associated with the at least one domain-specific vocabulary to identify the second portion, and wherein performing the second speech processing pass comprises recognizing the second portion identified by the at least one tag.

17. The method of claim 15, wherein performing the second speech processing pass comprises recognizing a portion having words specified in the first domain-specific vocabulary using at least one grammar associated with the first domain-specific vocabulary.

18. The method of claim 15, wherein the at least one filler model includes at least one phoneme model to determine phoneme likelihoods in the second portion, the at least one phoneme model including a phoneme loop model and/or a phoneme N-gram model.

19. The method of claim 14, wherein the first speech processing pass is performed at a first location on a network and the second speech processing pass is performed at a second location on the network, wherein the first location is different from the second location.

20. The method of claim 15, wherein the second portion including the at least one word specified in the at least one domain-specific vocabulary includes a third portion including at least one word specified in a first domain-specific vocabulary and a fourth portion including at least one word specified in a second domain-specific vocabulary, wherein the at least one filler model further includes a second filler model trained at least in part on data in the second domain-specific vocabulary, and wherein performing the second speech processing pass comprises:

using a first domain-specific speech recognizer adapted to recognize words in the first domain-specific vocabulary to recognize the third portion identified at least in part using the first filler model; and

using a second domain-specific recognizer adapted to recognize words in the second domain-specific vocabulary to recognize the fourth portion identified at least in part using the second filler model.