Method and apparatus for speech recognition using neural networks with speaker adaptation

• US 9,721,561 B2
• Filed: 12/05/2013
• Issued: 08/01/2017
• Est. Priority Date: 12/05/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Tang, Yun, Nagesha, Venkatesh, Fan, Xing
• Primary Examiner(s): COLUCCI, MICHAEL C
• Application Number:
  US14/098,259
• Publication Number:
  US 20150161994A1
• Time in Patent Office: 1,335 Days
• Field of Search: 704232, 704259, 704255, 704250, 704246, 704244, 704241, 704236, 704235, 704234, 84615, 714 33, 706 16
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/07 to the speaker
• CPC Class Codes: G10L 15/16 using artificial neural networks
• CPC Class Codes: G10L 2015/025 Phonemes, fenemes or fenones being the recognition units

Abstract

In a speech recognition system, deep neural networks (DNNs) are employed in phoneme recognition. While DNNs typically provide better phoneme recognition performance than other techniques, such as Gaussian mixture models (GMM), adapting a DNN to a particular speaker is a real challenge. According to at least one example embodiment, speech data and corresponding speaker data are both applied as input to a DNN. In response, the DNN generates a prediction of a phoneme based on the input speech data and the corresponding speaker data. The speaker data may be generated from the corresponding speech data.

Claims

1. A computer implemented method for speech recognition, the method implemented by one or more processors and comprising:

receiving, by a deep neural network at a hidden layer, input speech data at a first set of nodes of the hidden layer of the deep neural network and corresponding speaker data at a second set of nodes of the hidden layer of the deep neural network, the second set of nodes serving as an extra input to the deep neural network; and

generating, by the deep neural network, a prediction of a phoneme corresponding to the input speech data based on the corresponding speaker data, wherein the generating comprises multiplying the input speech data received at the first set of nodes with a first matrix of weighting coefficients and multiplying the speaker data received at the second set of nodes with a second matrix of weighting coefficients, multiplying the speaker data with the second matrix of weighting coefficients removing speaker variability from the input speech data.

2. The method as recited in claim 1, wherein the input speech data is part of training data and said receiving and generating are repeated for input speech data and corresponding speaker data associated with multiple speakers, the method further comprising:

iteratively updating weighting coefficients of the deep neural network based on the prediction of the phoneme generated and information in the training data.

3. The method as recited in claim 1, wherein the input speech data is deployment speech data collected by a speech recognition system.

4. The method as recited in claim 1, wherein the speaker data is generated from the corresponding input speech data using maximum likelihood linear regression (MLLR).

5. The method as recited in claim 1, wherein the speaker data is generated from the corresponding input speech data using constrained maximum likelihood linear regression (CMLLR).

6. The method as recited in claim 1, wherein the prediction of the phoneme generated includes a probability score.

7. The method as recited in claim 1, wherein the prediction of the phoneme generated includes an indication of a phoneme.

8. The method as recited in claim 1 further comprising training a Gaussian mixture models'"'"' module using the prediction of the phoneme generated by the deep neural network.

9. The method as recited in claim 8, wherein the Gaussian mixture models'"'"' module is adapted using the speaker data.

10. The method as recited in claim 1 further comprising reducing dimensionality of the speaker data using principal component analysis (PCA) prior to reception by the deep neural network.

11. An apparatus for speech recognition comprising:

at least one processor; and

at least one memory with computer code instructions stored thereon,

the at least one processor and the at least one memory with computer code instructions being configured to cause the apparatus to;

receive, by a deep neural network at a hidden layer, input speech data at a first set of nodes of the hidden layer of the deep neural network and corresponding speaker data at a second set of nodes of the hidden layer of the deep neural network, the second set of nodes serving as an extra input to the deep neural network; and

generate, at an output layer of the deep neural network, a prediction of a phoneme corresponding to the input speech data based on the corresponding speaker data, wherein the generating comprises multiplying the input speech data received at the first set of nodes with a first matrix of weighting coefficients and multiplying the speaker data received at the second set of nodes with a second matrix of weighting coefficients, multiplying the speaker data with the second matrix of weighting coefficients removing speaker variability from the input speech data.

12. The apparatus as recited in claim 11, wherein the input speech data is part of training data and wherein the at least one processor and the at least one memory with computer code instructions are further configured to cause the apparatus to:

repeat receiving input speech data and corresponding speaker data and generating the prediction of the phoneme for multiple speakers; and

iteratively update weighting coefficients of the deep neural network based on the prediction of the phoneme generated and information in the training data.

13. The apparatus as recited in claim 11, wherein the input speech data is deployment speech data.

14. The apparatus as recited in claim 11, wherein the at least one processor and the at least one memory with computer code instructions are further configured to cause the apparatus to generate the speaker data from the corresponding input speech data using maximum likelihood linear regression (MLLR).

15. The apparatus as recited in claim 11, wherein the at least one processor and the at least one memory with computer code instructions are further configured to cause the apparatus to generate the speaker data from the corresponding input speech data using constrained maximum likelihood linear regression (CMLLR).

16. The apparatus as recited in claim 11, wherein the prediction of the phoneme generated includes a probability score or an indication of a phoneme.

17. The apparatus as recited in claim 11, wherein the at least one processor and the at least one memory with computer code instructions are further configured to cause the apparatus to train a Gaussian mixture models'"'"' module using the prediction of the phoneme generated.

18. The apparatus as recited in claim 17, wherein the Gaussian mixture models'"'"' module is adapted using the speaker data.

19. The apparatus as recited in claim 17, wherein the at least one processor and the at least one memory with computer code instructions are further configured to cause the apparatus to reduce the dimensionality of the speaker data using principal component analysis (PCA) prior to reception by the deep neural network.

20. A non-transitory computer-readable medium with computer code instructions stored thereon, the computer code instructions being configured, when executed by a processor, to cause an apparatus to:

receive, by a deep neural network at a hidden layer, input speech data at a first set of nodes of the hidden layer of the deep neural network and corresponding speaker data at a second set of nodes of the hidden layer of the deep neural network, the second set of nodes serving as an extra input to the deep neural network; and

generate, by the deep neural network, a prediction of a phoneme corresponding to the input speech data based on the corresponding speaker data, wherein the generating comprises multiplying the input speech data received at the first set of nodes with a first matrix of weighting coefficients and multiplying the speaker data received at the second set of nodes with a second matrix of weighting coefficients, multiplying the speaker data with the second matrix of weighting coefficients removing speaker variability from the input speech data.