Speech Recognition

• US 20100057462A1
• Filed: 09/02/2009
• Published: 03/04/2010
• Est. Priority Date: 09/03/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Raab, Martin, Brueckner, Raymond, Gruhn, Rainer, Herbig, Tobias
• Granted Patent: US 8,275,619 B2
• Time in Patent Office: Days
• US Class Current: 704/243
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/06 Creation of reference templates; Training of speech recognition systems, e.g. adaptation to the characteristics of the speaker's voice G10L15/14 takes precedence
• CPC Class Codes: G10L 15/142 Hidden Markov Models [HMMs]
• CPC Class Codes: G10L 2015/025 Phonemes, fenemes or fenones being the recognition units

Abstract

The present invention relates to a method for speech recognition of a speech signal comprising the steps of providing at least one codebook comprising codebook entries, in particular, multivariate Gaussians of feature vectors, that are frequency weighted such that higher weights are assigned to entries corresponding to frequencies below a predetermined level than to entries corresponding to frequencies above the predetermined level and processing the speech signal for speech recognition comprising extracting at least one feature vector from the speech signal and matching the feature vector with the entries of the codebook.

Claims

1. A processor implemented method for speech recognition of a speech signal comprising:

within a processor;

providing at least one codebook comprising codebook entries, in particular, multivariate Gaussians of feature vectors, that are frequency weighted; and

processing the speech signal for speech recognition comprising;

extracting at least one feature vector from the speech signal and matching the feature vector with the entries of the codebook.

2. The method according to claim 1, wherein the entries of the at least one code-book are obtained by

detecting training speech signals corresponding to utterances of one or more native speakers;

transforming the training speech signals into the frequency domain, in particular, considering only the envelope of the training speech signals;

applying a MEL filterbank to the transformed training speech signals and saving the log value of each bank to derive first representations of the training speech signals;

applying a Discrete Cosine Transform to the first representations of the training speech signals to obtain second transformed training speech signals;

applying a Linear Discriminant Analysis to the second transformed training speech signals to obtain third transformed training speech signals;

extracting feature vectors from the third transformed training speech signals;

determining covariance matrices for the extracted feature vectors in the Linear Discriminant Analysis domain;

transforming the covariance matrices into the log-MEL domain;

applying weights to the covariances of the covariance matrices in the log-MEL domain to obtain modified covariance matrices in the log-MEL domain; and

transforming the modified covariance matrices from the log-MEL domain into the Linear Discriminant Analysis domain;

determining Gaussians from the modified covariance matrices to obtain the entries of the at least one codebook.

3. The method according to claim 1, wherein the entries of the at least one code-book are obtained by:

detecting training speech signals corresponding to utterances of one or more native speakers;

transforming the training speech signals into the frequency domain considering only the envelope of the training speech signals;

applying a MEL filterbank to the transformed training speech signals and saving the log value of each bank to derive first representations of the training speech signals;

applying a Discrete Cosine Transform to the first representations of the training speech signals to obtain second transformed training speech signals;

applying a Linear Discriminant Analysis to the second transformed training speech signals to obtain third transformed training speech signals;

extracting feature vectors from the third transformed training speech signals;

determining covariance matrices for the extracted feature vectors in the Discrete Cosine Transform domain;

transforming the covariance matrices into the log-MEL domain;

applying weights to the covariances of the covariance matrices in the log-MEL domain to obtain modified covariance matrices in the log-MEL domain; and

transforming the modified covariance matrices from the log-MEL domain into the Linear Discriminant Analysis domain;

determining Gaussians from the modified covariance matrices to obtain the entries of the at least one codebook.

4. The method according to claim 1, further comprising:

transforming the speech signal into the frequency domain, in particular, considering only the envelope of the speech signal;

applying a MEL filterbank to the transformed speech signal and save the log value of each bank to derive a first representation of the speech signal;

applying a Discrete Cosine Transform to the first representation of the speech signal to obtain a second transformed speech signal;

applying a Linear Discriminant Analysis to the second transformed speech signal to obtain a third transformed speech signal;

extracting at least one feature vector from the third transformed speech signal; and

recognizing the speech signal by determining a best matching Gaussian of the codebook for the extracted at least one feature vector.

5. The method according to claim 1, further comprising:

generating likelihoods of sequences of speech segments by a Hidden Markov Model representing phonemes of speech; and

recognizing at least one spoken word by means of the at least one codebook and the generated likelihoods.

6. The method according to claim 1, further comprising:

providing at least one additional codebook comprising codebook entries, in particular, multivariate Gaussians of feature vectors, without frequency weights;

determining whether the speech signal corresponds to the utterance of a native speaker or to the utterance of a non-native speaker; and

if it is determined that the speech signal corresponds to the utterance of a native speaker, using the at least one additional codebook comprising codebook entries without frequency weights for the speech recognition;

or

if it is determined that the speech signal corresponds to the utterance of a non-native speaker, using the at least one codebook comprising codebook entries that are frequency weighted for the speech recognition.

7. The method according to claim 6, wherein determining whether the speech signal corresponds to the utterance of a native speaker or to the utterance of a non-native speaker comprises:

either automatically classifying the speech signal as a signal corresponding to an utterance by a native or a non-native speaker or detecting an input by a user indicating that the speech signal corresponds to an utterance by a native or a non-native speaker.

8. A processor implemented method for retrofitting a speech recognition system that comprises at least one first code-book having codebook entries, in particular, multivariate Gaussians, the method comprising:

in a processor;

weighting the entries of the at least one first codebook such that higher weights are assigned to entries corresponding to frequencies below a predetermined level than to entries corresponding to frequencies above the predetermined level; and

generating at least one second codebook to be used for speech recognition by the speech recognition means by storing the weighted entries of the at least one first codebook.

9. The method according to claim 8, wherein weighting the entries of the at least one first codebook comprises

transforming covariance matrices stored in the at least one first codebook into the log-MEL domain;

applying weights to the covariances of the covariance matrices in the log-MEL domain to obtain modified covariance matrices in the log-MEL domain; and

transforming the modified covariance matrices from the log-MEL domain into the Linear Discriminant Analysis domain;

determining Gaussians from the modified covariance matrices to obtain the entries of the at least one second codebook.

10. A computer program product comprising a computer readable medium having computer-executable instructions thereon for speech recognition of a speech signal comprising, the computer-executable instructions comprising:

computer code for providing at least one codebook comprising codebook entries, in particular, multivariate Gaussians of feature vectors, that are frequency weighted; and

computer code for processing the speech signal for speech recognition comprising;

computer code for extracting at least one feature vector from the speech signal and computer code for matching the feature vector with the entries of the codebook.

11. The computer program product according to claim 10, wherein the entries of the at least one code-book are obtained by:

detecting training speech signals corresponding to utterances of one or more native speakers;

transforming the training speech signals into the frequency domain, in particular, considering only the envelope of the training speech signals;

applying a MEL filterbank to the transformed training speech signals and saving the log value of each bank to derive first representations of the training speech signals;

applying a Discrete Cosine Transform to the first representations of the training speech signals to obtain second transformed training speech signals;

applying a Linear Discriminant Analysis to the second transformed training speech signals to obtain third transformed training speech signals;

extracting feature vectors from the third transformed training speech signals;

determining covariance matrices for the extracted feature vectors in the Linear Discriminant Analysis domain;

transforming the covariance matrices into the log-MEL domain;

applying weights to the covariances of the covariance matrices in the log-MEL domain to obtain modified covariance matrices in the log-MEL domain;

transforming the modified covariance matrices from the log-MEL domain into the Linear Discriminant Analysis domain; and

determining Gaussians from the modified covariance matrices to obtain the entries of the at least one codebook.

12. The computer program product according to claim 10, wherein the entries of the at least one code-book are obtained by:

detecting training speech signals corresponding to utterances of one or more native speakers;

transforming the training speech signals into the frequency domain considering only the envelope of the training speech signals;

applying a MEL filterbank to the transformed training speech signals and saving the log value of each bank to derive first representations of the training speech signals;

applying a Discrete Cosine Transform to the first representations of the training speech signals to obtain second transformed training speech signals;

applying a Linear Discriminant Analysis to the second transformed training speech signals to obtain third transformed training speech signals;

extracting feature vectors from the third transformed training speech signals;

determining covariance matrices for the extracted feature vectors in the Discrete Cosine Transform domain;

transforming the covariance matrices into the log-MEL domain;

applying weights to the covariances of the covariance matrices in the log-MEL domain to obtain modified covariance matrices in the log-MEL domain;

transforming the modified covariance matrices from the log-MEL domain into the Linear Discriminant Analysis domain; and

determining Gaussians from the modified covariance matrices to obtain the entries of the at least one codebook.

13. The computer program product according to claim 10, further comprising:

computer code for transforming the speech signal into the frequency domain, in particular, considering only the envelope of the speech signal;

computer code for applying a MEL filterbank to the transformed speech signal and save the log value of each bank to derive a first representation of the speech signal;

computer code for applying a Discrete Cosine Transform to the first representation of the speech signal to obtain a second transformed speech signal;

computer code for applying a Linear Discriminant Analysis to the second transformed speech signal to obtain a third transformed speech signal;

computer code for extracting at least one feature vector from the third transformed speech signal; and

computer code for recognizing the speech signal by determining a best matching Gaussian of the codebook for the extracted at least one feature vector.

14. The computer program product according to claim 10, further comprising:

computer code for generating likelihoods of sequences of speech segments by a Hidden Markov Model representing phonemes of speech; and

computer code for recognizing at least one spoken word by means of the at least one codebook and the generated likelihoods.

15. The computer program product according to claim 10, further comprising:

computer code for providing at least one additional codebook comprising codebook entries, in particular, multivariate Gaussians of feature vectors, without frequency weights;

computer code for determining whether the speech signal corresponds to the utterance of a native speaker or to the utterance of a non-native speaker; and

computer code for using the at least one additional codebook comprising codebook entries without frequency weights for the speech recognition if it is determined that the speech signal corresponds to the utterance of a native speaker;

computer code for using the at least one codebook comprising codebook entries that are frequency weighted for the speech recognition if it is determined that the speech signal corresponds to the utterance of a non-native speaker.

16. The computer program product according to claim 15, wherein the computer code for determining whether the speech signal corresponds to the utterance of a native speaker or to the utterance of a non-native speaker comprises:

computer code for automatically classifying the speech signal as a signal corresponding to an utterance by a native or a non-native speaker.

17. The computer program product according to claim 15 wherein the computer code for determining whether the speech signal corresponds to the utterance of a native speaker or to the utterance of a non-native speaker comprises:

computer code for detecting an input by a user indicating that the speech signal corresponds to an utterance by a native or a non-native speaker.

23. The speech recognition means according to claim 13, further comprising a selection means configured to allow a user of the speech recognition means to cause the control means to either control speech recognition of the speech signal input into the speech recognition means based on the at least one code book comprising the weighted multivariate Gaussians or based on the at least one additional code book comprising the multivariate Gaussians that are not frequency weighted.

18. A computer program product comprising a computer readable medium having computer executable code thereon for retrofitting a speech recognition system that comprises at least one first code-book having codebook entries that include multivariate Gaussians, the computer executable code comprising:

computer code for weighting the entries of the at least one first codebook such that higher weights are assigned to entries corresponding to frequencies below a predetermined level than to entries corresponding to frequencies above the predetermined level; and

computer code for generating at least one second codebook to be used for speech recognition by the speech recognition means by storing the weighted entries of the at least one first codebook.

19. The computer program product according to claim 18, wherein weighting the entries of the at least one first codebook comprises:

computer code for transforming covariance matrices stored in the at least one first codebook into the log-MEL domain;

computer code for applying weights to the covariances of the covariance matrices in the log-MEL domain to obtain modified covariance matrices in the log-MEL domain;

computer code for transforming the modified covariance matrices from the log-MEL domain into the Linear Discriminant Analysis domain; and

computer code for determining Gaussians from the modified covariance matrices to obtain the entries of the at least one second codebook.

20. A Speech recognition system, comprising:

a speech processing engine for receiving a speech signal;

memory containing;

at least one codebook comprising multivariate Gaussians;

wherein the multivariate Gaussians are weighted such that higher weights are assigned to entries corresponding to frequencies below a predetermined level than to entries corresponding to frequencies above the predetermined level and wherein the codebook is used by the speech processing engine to determine words within the speech signal.

21. The speech recognition system according to claim 20, wherein the memory further contains Hidden Markov Models and some language model and the speech processing engine is configured for speech recognition of the speech signal based on the at least one code book, the Hidden Markov Models and the language model.

22. The speech recognition means according to claim 20, wherein the memory further comprises at least one additional codebook comprising multivariate Gaussians that are not frequency weighted and wherein the speech recognition system further comprises a controller configured to control speech recognition of a speech signal input into the speech recognition system either based on the at least one code book comprising the weighted multivariate Gaussians or based on the at least one additional codebook comprising the multivariate Gaussians that are not frequency weighted.

24. The speech recognition system according to claim 22, further comprising

an analysis module configured to determine whether the speech signal corresponds to an utterance of a native speaker or an utterance of a non-native speaker;

and to

transmit a first signal to the controller, if it is determined that the speech signal corresponds to an utterance of a native speaker and to transmit a second signal to the controller, if it is determined that the speech signal corresponds to an utterance of a non-native speaker; and

wherein

the controller is configured to control speech recognition of the speech signal input into the speech processing means based on the at least one additional code book comprising the multivariate Gaussians that are not frequency weighted, if it receives the first signal from the analysis module, and to control speech recognition of the speech signal input into the speech processing engine based on the at least one code book comprising the weighted multivariate Gaussians, if it receives the second signal from the analysis module.