Adversarial Teacher-Student Learning for Unsupervised Domain Adaptation

• US 20190287515A1
• Filed: 03/16/2018
• Published: 09/19/2019
• Est. Priority Date: 03/16/2018
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Li, Jinyu, Meng, Zhong, Gong, Yifan
• Granted Patent: US 10,643,602 B2
• Time in Patent Office: Days
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/16 using artificial neural networks

Abstract

Methods, systems, and computer programs are presented for training, with adversarial constraints, a student model for speech recognition based on a teacher model. One method includes operations for training a teacher model based on teacher speech data, initializing a student model with parameters obtained from the teacher model, and training the student model with adversarial teacher-student learning based on the teacher speech data and student speech data. Training the student model with adversarial teacher-student learning further includes minimizing a teacher-student loss that measures a divergence of outputs between the teacher model and the student model; minimizing a classifier condition loss with respect to parameters of a condition classifier; and maximizing the classifier condition loss with respect to parameters of a feature extractor. The classifier condition loss measures errors caused by acoustic condition classification. Further, speech is recognized with the trained student model.

Claims

1. A method comprising:

training, by one or more processors, a teacher model based on teacher speech data;

initializing, by the one or more processors, a student model with parameters obtained from the trained teacher model;

training, by the one or more processors, the student model with adversarial teacher-student learning based on the teacher speech data and student speech data, training the student model with adversarial teacher-student learning further comprising;

minimizing a teacher-student loss that measures a divergence of outputs between the teacher model and the student model;

minimizing a classifier condition loss with respect to parameters of a condition classifier, the classifier condition loss measuring errors caused by acoustic condition classification; and

maximizing the classifier condition loss with respect to parameters of a feature extractor; and

recognizing speech with the trained student model.

2. The method as recited in claim 1, wherein the condition classifier is a neural network for mapping each deep feature to an acoustic condition.

3. The method as recited in claim 1, wherein the divergence is a Kullback-Leibler divergence that measures how an output distribution of the teacher model diverges from an output distribution of the student model.

4. The method as recited in claim 1, wherein the student model further comprises:

a classifier to classify units of speech, the units of speech being one of a senone, a phoneme, a tri-phone, a syllable, a character, a part of a word, or a word; and

a feature extractor for extracting deep features from the student speech data.

5. The method as recited in claim 1, wherein the teacher speech data comprises a plurality of utterances in a teacher domain, wherein the student speech data comprises the plurality of utterances in a student domain, wherein training the student model further comprises:

providing the plurality of utterances from the teacher speech data in parallel to the plurality of utterances in the student speech data.

6. The method as recited in claim 1, wherein the teacher-student loss is calculated by:

calculating a teacher senone posterior;

calculating a student senone posterior for a deep feature; and

calculating the teacher-student loss as a difference between the teacher senone posterior and the student senone posterior.

7. The method as recited in claim 1, wherein a condition defines characteristics of a speaker and an environment where speech is captured.

8. The method as recited in claim 1, wherein training the student model with adversarial teacher-student learning causes the student model to recognize senones similarly to how the teacher model recognizes senones in a condition-robust fashion.

9. The method as recited in claim 1, wherein training the student model with adversarial teacher-student learning causes the student model to lack differentiation among different conditions.

10. The method as recited in claim 1, wherein training the student model is performed iteratively by analyzing the teacher speech data and the student speech data.

11. A system comprising:

a memory comprising instructions; and

one or more computer processors, wherein the instructions, when executed by the one or more computer processors, cause the one or more computer processors to perform operations comprising;

training a teacher model based on teacher speech data;

initializing a student model with parameters obtained from the trained teacher model;

training the student model with adversarial teacher-student learning based on the teacher speech data and student speech data, training the student model with adversarial teacher-student learning further comprising;

minimizing a teacher-student loss that measures a divergence of outputs between the teacher model and the student model;

minimizing a classifier condition loss with respect to parameters of a condition classifier, the classifier condition loss measuring errors caused by acoustic condition classification; and

maximizing the classifier condition loss with respect to parameters of a feature extractor; and

recognizing speech with the trained student model.

12. The system as recited in claim 11, wherein the condition classifier is a neural network for mapping each deep feature to an acoustic condition.

13. The system as recited in claim 11, wherein the student model further comprises:

a classifier to classify units of speech, the units of speech being one of a senone, a phoneme, a tri-phone, a syllable, a character, a part of a word, or a word; and

a feature extractor for extracting deep features from the student speech data.

14. The system as recited in claim 11, wherein the teacher speech data comprises a plurality of utterances in a teacher domain, wherein the student speech data comprises the plurality of utterances in a student domain, wherein training the student model further comprises:

providing the plurality of utterances from the teacher speech data in parallel to the plurality of utterances in the student speech data.

15. The system as recited in claim 11, wherein training the student model with adversarial teacher-student learning that causes the student model to recognize senones similarly to how the teacher model recognizes senones in a condition-robust fashion, wherein training the student model with adversarial teacher-student learning that cause the student model to lack differentiation among different conditions.

16. A machine-readable storage medium including instructions that, when executed by a machine, cause the machine to perform operations comprising:

training a teacher model based on teacher speech data;

initializing a student model with parameters obtained from the trained teacher model;

training the student model with adversarial teacher-student learning based on the teacher speech data and student speech data, training the student model with adversarial teacher-student learning further comprising;

minimizing a teacher-student loss that measures a divergence of outputs between the teacher model and the student model;

minimizing a classifier condition loss with respect to parameters of a condition classifier, the classifier condition loss measuring errors caused by acoustic condition classification; and

maximizing the classifier condition loss with respect to parameters of a feature extractor; and

recognizing speech with the trained student model.

17. The machine-readable storage medium as recited in claim 16, wherein the condition classifier is a neural network mapping each deep feature to an acoustic condition.

18. The machine-readable storage medium as recited in claim 16, wherein the student model further comprises:

a classifier to classify units of speech, the units of speech being one of a senone, a phoneme, a tri-phone, a syllable, a character, a part of a word, or a word; and

a feature extractor for extracting deep features from the student speech data.

19. The machine-readable storage medium as recited in claim 16, wherein the teacher speech data comprises a plurality of utterances in a teacher domain, wherein the student speech data comprises the plurality of utterances in a student domain, wherein training the student model further comprises:

providing the plurality of utterances from the teacher speech data in parallel to the plurality of utterances in the student speech data.

20. The machine-readable storage medium as recited in claim 16, wherein training the student model with adversarial teacher-student learning causes the student model to recognize senones similarly to how the teacher model recognizes senones in a condition-robust fashion, wherein training the student model with adversarial teacher-student learning cause the student model to lack differentiation among different conditions.