ACTIVE LABELING FOR SPOKEN LANGUAGE UNDERSTANDING

• US 20090254344A1
• Filed: 06/16/2009
• Published: 10/08/2009
• Est. Priority Date: 05/29/2003
• Status: Active Grant

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: AT&T Corporation (AT&T, Inc.)
• Inventors: Hakkani-Tur, Dilek Z., Rahim, Mazin G., Tur, Gokhan
• Granted Patent: US 7,949,525 B2
• Time in Patent Office: Days
• US Class Current: 704/251
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding

Abstract

A spoken language understanding method and system are provided. The method includes classifying a set of labeled candidate utterances based on a previously trained classifier, generating classification types for each candidate utterance, receiving confidence scores for the classification types from the trained classifier, sorting the classified utterances based on an analysis of the confidence score of each candidate utterance compared to a respective label of the candidate utterance, and rechecking candidate utterances according to the analysis. The system includes modules configured to control a processor in the system to perform the steps of the method.

Claims

1. A spoken language understanding system comprising:

a processor;

a module configured to control the processor to classify a set of labeled candidate utterances based on a previously trained classifier and generate classification types for each candidate utterance, each classification type having a respective confidence score;

a module configured to control the processor to sort the candidate utterances based on an analysis of the confidence score of each candidate utterance compared to a respective label of the candidate utterance; and

a module configured to control the processor to recheck candidate utterances according to the analysis.

2. The system of claim 1, wherein the confidence scores are generated for a plurality of labels associated with the classification types.

3. The system of claim 1, wherein the confidence scores are received from the previously trained classifier.

4. The system of claim 1, wherein the analysis is based on a comparison of the generated classification type of the candidate utterance and the label of the candidate utterance, and the module rechecks the candidate utterances when the generated classification type does not match the label of the candidate utterance.

5. The system of claim 1, wherein the analysis is based on a Kullback-Leibler divergence, and the module rechecks the candidate utterances when the Kullback-Leibler divergence is greater than a threshold

6. The system of claim 5, wherein the Kullback-Leibler divergence is based on the computation:

$\mathrm{KL}(P<br><br><br><br>Q)=\underset{i\in <br><br>L}{\sum <br><br>}<br><br>p_i\times <br><br>\log <br><br>\left(\frac{p_i}{q_i}\right)+\left(1-p_i\right)\times <br><br>\log <br><br>\left(\frac{1-p_i}{1-q_i}\right)$

where L is the set of all classification types, q_i is the probability of the i^th classification type obtained from the trained classifier, and p=1 if that classification type is previously labeled and p=0 if otherwise.

7. A tangible computer-readable medium storing instructions for controlling a computer device, as part of a spoken language understanding system, to perform the steps of:

classifying a set of labeled candidate utterances based on a previously trained classifier;

generating classification types for each candidate utterance, each classification type having a respective confidence score;

sorting the candidate utterances based on an analysis of the confidence score of each candidate utterance compared to a respective label of the candidate utterance; and

rechecking the candidate utterances that do not match the label of the respective candidate utterance.

8. The tangible computer-readable medium of claim 7, further comprising generating confidence scores for a plurality of labels associated with the classification types.

9. The tangible computer-readable medium of claim 7, further comprising receiving confidence scores from the previously trained classifier.

10. The tangible computer-readable medium of claim 7, further comprising identifying candidate utterances having a top choice of classification types that does not match the label of the respective candidate utterances.

11. A computer-implemented method for automatic speech recognition that extracts words from user speech, the system comprising:

classifying, via a processor, a set of labeled candidate utterances based on a previously trained classifier;

generating, via a processor, classification types for each candidate utterance;

receiving confidence scores for the classification types from the trained classifier;

sorting the classified utterances based on an analysis of the confidence score of each candidate utterance compared to a respective label of the candidate utterance; and

rechecking candidate utterances according to the analysis.

12. The computer-implemented method of claim 11, wherein the analysis is based on a comparison of the generated classification type of the candidate utterance and the label of the candidate utterance, and the module rechecks the candidate utterances when the generated classification type does not match the label of the candidate utterance.

13. The computer-implemented method of claim 11, wherein the analysis is based on a Kullback-Leibler divergence, and the module rechecks the candidate utterances when the Kullback-Leibler divergence is greater than a threshold

14. The computer-implemented method of claim 13, wherein the Kullback-Leibler divergence is based on the computation:

$\mathrm{KL}(P<br><br><br><br>Q)=\underset{i\in <br><br>L}{\sum <br><br>}<br><br>p_i\times <br><br>\log <br><br>\left(\frac{p_i}{q_i}\right)+\left(1-p_i\right)\times <br><br>\log <br><br>\left(\frac{1-p_i}{1-q_i}\right)$

where L is the set of all classification types, q_i is the probability of the i^th classification type obtained from the trained classifier, and p=1 if that classification type is previously labeled and p=0 if otherwise.

15. The computer-implemented method of claim 11, wherein the classification types include call classification types.

16. The computer-implemented method of claim 11, wherein the trained classifier comprises an iterative classifier.