Integrated speech recognition and semantic classification

• US 20080177547A1
• Filed: 01/19/2007
• Published: 07/24/2008
• Est. Priority Date: 01/19/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Deng, Li, Yu, Dong, Acero, Alejandro, Wang, Ye-Yi, Yaman, Sibel
• Granted Patent: US 7,856,351 B2
• Time in Patent Office: Days
• US Class Current: 704/257
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning

Abstract

A novel system integrates speech recognition and semantic classification, so that acoustic scores in a speech recognizer that accepts spoken utterances may be taken into account when training both language models and semantic classification models. For example, a joint association score may be defined that is indicative of a correspondence of a semantic class and a word sequence for an acoustic signal. The joint association score may incorporate parameters such as weighting parameters for signal-to-class modeling of the acoustic signal, language model parameters and scores, and acoustic model parameters and scores. The parameters may be revised to raise the joint association score of a target word sequence with a target semantic class relative to the joint association score of a competitor word sequence with the target semantic class. The parameters may be designed so that the semantic classification errors in the training data are minimized.

Claims

1. A method comprising:

defining a joint association score indicative of a correspondence of a semantic class and a word sequence for an acoustic signal, wherein the joint association score incorporates one or more parameters that are applied to one or more features of the word sequence for signal-to-class modeling of the acoustic signal;

identifying a target word sequence from among a set of word sequences, such that the target word sequence has a highest joint association score with a target semantic class;

identifying a competitor word sequence from among the set of word sequences other than the target word sequence, such that the competitor word sequence has a highest remaining joint association score with any available semantic class other than the target semantic class; and

revising one or more of the parameters to raise the joint association score of the target word sequence with the target semantic class relative to the joint association score of the competitor word sequence with the target semantic class.

2. The method of claim 1, wherein revising the parameters comprises increasing one or more of the parameters that apply to signal-to-class modeling features of the target word sequence and not to signal-to-class modeling features of the competitor word sequence.

3. The method of claim 1, wherein revising the parameters comprises decreasing one or more of the parameters that apply to signal-to-class modeling features of the competitor word sequence and not to signal-to-class modeling features of the target word sequence.

4. The method of claim 1, wherein the signal-to-class modeling comprises an automatic speech recognition language model configured for matching an acoustic signal with a word sequence, and the parameters are applied to lexical features of the word sequence.

5. The method of claim 4, wherein the lexical features comprise n-grams in the word sequence.

6. The method of claim 4, wherein the automatic speech recognition language model is configured for matching an acoustic signal with a word sequence using speech lattices.

7. The method of claim 6, wherein the automatic speech recognition language model is further configured for matching an acoustic signal with a word sequence by summing up acoustic scores of all paths of the speech lattices that correspond to possible word sequences for the acoustic signal.

8. The method of claim 1, wherein the signal-to-class modeling comprises a semantic classification model configured for matching a word sequence with a semantic class, and the parameters are applied to semantic features of a semantic class.

9. The method of claim 8, wherein the semantic features correspond to lexical features of the word sequence.

10. The method of claim 1, further comprising:

iteratively identifying additional competitor word sequences from among the remaining word sequences other than the target word sequence and previously identified competitor word sequences, such that each of the competitor word sequences has a highest remaining joint association score with any remaining semantic class that has not previously been used for a highest joint association score; and

increasing one or more of the parameters that apply to signal-to-class modeling features of the target word sequence and not to signal-to-class modeling features of any of the competitor word sequences.

11. The method of claim 1, further comprising:

identifying one or more additional target word sequences from among the set of word sequences, the additional target word sequences having highest joint association scores with one or more additional target semantic classes, for one or more additional acoustic signals; and

for each of the target semantic classes, revising weighting factors to increase the joint association scores of the target word sequences with the target semantic classes.

12. The method of claim 11, further comprising:

assigning a total loss function that parameterizes the joint association scores of the target word sequences and the competitor word sequences with the target semantic classes;

using the total loss function as a weighting parameter applied to a component of the joint classification score that indicates a probability of a word sequence corresponding to an acoustic signal; and

iteratively adjusting the weighting parameters applied to one or more features of the word sequence for signal-to-class modeling of the acoustic signal to reduce the total loss function.

13. The method of claim 1, further comprising providing a system configured for automatic semantic classification of spoken language using an automatic speech recognition language model that incorporates one or more of the revised factors.

14. The method of claim 1, further comprising providing a system configured for automatic semantic classification of spoken language using a semantic classification model that incorporates one or more of the revised factors.

15. The method of claim 1, wherein the joint association scores of the word sequences with the target semantic class are modeled using a maximum entropy model.

16. The method of claim 1, wherein updating one of the parameter values comprises using an update equation that is formed by using minimum classification error training.

17. The method of claim 1, wherein updating one of the parameter values comprises using an update equation that is formed by using a steepest descent optimization technique.

18. The method of claim 1, wherein the acoustic signal comprises a spoken utterance.

19. A medium comprising instructions that are readable and executable by a computing system, wherein the instructions configure the computing system to train an integrated automatic speech recognition and semantic classification system configured for recognizing and semantically classifying acoustic signals, comprising configuring the computing system to:

use a speech lattice to determine language model parameters for matching a set of acoustic signals to a set of word sequences;

use a maximum entropy model to determine semantic classification model parameters for matching the set of word sequences to a set of semantic classes;

evaluate a classification decision rule configured to match acoustic signals to corresponding semantic classes, incorporating the language model parameters and the semantic classification model parameters;

determine a total classification loss function indicative of a rate of error in matching the acoustic signals to word sequences to semantic classes;

weight the language model parameters in the classification decision rule to account for the total classification loss function;

iteratively update at least one of the language model parameters and the semantic classification model parameters to reduce the total classification loss function that incorporates errors in semantically classifying acoustic signals due to the language model and the semantic classification model; and

provide the classification decision rule incorporating the at least one of the iteratively updated language model parameters and the semantic classification model parameters, for the integrated automatic speech recognition and semantic classification system.

20. A medium comprising instructions that are readable and executable by a computing system, wherein the instructions configure the computing system to perform an integrated process of automatic speech recognition and automatic speech semantic classification on speech inputs, comprising configuring the computing system to:

receive a speech input;

match the speech input to a semantic class and to a word sequence corresponding to the semantic class by applying a semantic classification tool to the speech input, wherein the semantic classification rule selects for the highest joint association score between a semantic class and a word sequence for the speech input; and

provide the semantic classes matched to the speech input to an application configured to provide user output that is dependent on the semantic classes matched to the speech input;

wherein the joint association score comprises parameters of a language model and parameters of a semantic classification model that have been iteratively trained to reduce a total loss function indicative of a rate of error in semantically classifying speech inputs due to errors in either the language model or the semantic classification model.