Active learning for spoken language understanding

• US 7,263,486 B1
• Filed: 04/01/2003
• Issued: 08/28/2007
• Est. Priority Date: 10/25/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: AT&T Corporation (AT&T, Inc.)
• Inventors: Hakkani-Tur, Dilek Z., Schapire, Robert Elias, Tur, Gokhan
• Primary Examiner(s): Hudspeth, David
• Assistant Examiner(s): Han, Qi
• Application Number:
  US10/404,699
• Time in Patent Office: 1,610 Days
• Field of Search: 704/243, 704/245, 704/241, 704/231
• US Class Current: 704/243
• CPC Class Codes: G10L 15/063 Training

Abstract

Disclosed is a system and method of training a spoken language understanding module. Such a module may be utilized in a spoken dialog system. The method of training a spoken language understanding module comprises training acoustic and language models using a small set of transcribed data S_T, recognizing utterances in a set S_u that are candidates for transcription using the acoustic and language models, computing confidence scores of the utterances, selecting k utterances that have the smallest confidence scores from S_u and transcribing them into a new set S_i, redefining S_t as the union of S_t and S_i, redefining S_u as S_u minus S_i, and returning to the step of training acoustic and language models if word accuracy has not converged.

Claims

1. A method generating a classifier from training data S_t and a larger amount of unlabeled data in a pool S_u, the method comprising:

(1) training a classifier using current training data S_t, the training data S_t generated by sampling a plurality of utterances;

(2) classifying utterances in a pool S_u using the trained classifier;

(3) computing a call type confidence score for each utterance;

(4) sorting candidate utterances with respect to the confidence score of the maximum scoring call type;

(5) selecting the lowest scored k utterances from S_u using the confidence scores and labeling them to define a labeled set S_i;

(6) redefining S_t=S_t∪

S_i; and

(7) redefining S_u=S_u−

S_i.

2. The method of claim 1, wherein steps 1 through 7 are practiced until labelers and utterances are no longer available.

3. The method of claim 1, wherein k is more than one.

4. The method of claim 1, wherein selecting k utterances from S_u further comprises leaving out utterances with confidence scores indicating that the utterances were correctly recognized.

5. The method of claim 1, wherein selecting k utterances from S_u further comprises selecting the lowest scoring k utterances from S_u.

6. The method of claim 1, wherein selecting k utterances from S_u further comprises selecting utterances according to a confidence score distribution that is closest to a prior distribution.

7. A system having spoken language understanding module generated according to a method comprising:

(1) training a plurality of classifiers independently using a training data set S_t, the training data S_t generated by sampling a plurality of utterances;

(2) classifying utterances in a set S_u using the plurality of classifiers and computing a call type confidence score for all utterances;

(3) sorting candidate utterances with respect to a score of the maximum scoring call type according to one of the classifiers if the classifiers disagree;

(4) selecting and labeling the lowest scored k utterances from S_u to define a labeled set S_i and redefining S_t and S_u as follows;

(5) S_t=S_t∪

S_i; and

(6) S_u=S_u−

S_t, wherein the labeled utterances are used to generate the spoken language understanding module.

8. The system of claim 7, wherein selecting k utterances from S_u further comprises selecting the lowest scoring k utterances from S_u.

9. The system of claim 7, wherein the method further comprises performing steps 1-6 while labelers and utterances are available.

10. A system having spoken language understanding module trained using a method comprising:

(1) training acoustic and language models using a small set of transcribed data S_t, the training data S_t generated by sampling a plurality of utterances;

(2) recognizing utterances in a set S_u that are candidates for transcription using the acoustic and language models;

(3) computing confidence scores of the utterances;

(4) selecting the lowest scored k utterances from S_u using the confidence scores and transcribing them into a new set S_i;

(5) redefining S_t as the union of S_t and S_i;

(6) redefining S_u as S_u minus S_i; and

(7) returning to step (1) of word accuracy has not converged.

11. The system of claim 10, wherein selecting k utterances from S_u further comprises selecting utterances according to a confidence score distribution that is closest to a prior distribution.

12. The system of claim 10, wherein selecting k utterances from S_u further comprises selecting the lowest scoring k utterances from S_u.

13. A method of generating a spoken language understanding module, the method comprising, from small amount of training data S_t and a larger amount of unlabeled data S_u:

(1) training a plurality a classifiers independently using a training data set S_i, the training data S_t generated by sampling a plurality of utterances;

(2) classifying utterances in a set S_u using the plurality of classifiers and computing a call type confidence score for all utterances;

(3) sorting candidate utterances with respect to a score of the maximum scoring call type according to one of the classifiers if the classifiers disagree;

(4) selecting and labeling the lowest scored k utterances from S_u to define a labeled set S_i and redefining S_t and S_u as follows;

(5) S_t=S_t∪

S_i; and

(6) S_u=S_u−

S_i, wherein the labeled utterances are used to generate the spoken language understanding module.

14. The method of claim 13, wherein the steps occur only while labelers and utterances are available.

15. The method of claim 13, wherein selecting k utterances from S_u further comprises selecting utterances according to a confidence score distribution that is closet to a prior distribution.

16. The method of claim 13, wherein selecting k utterances from S_u further comprises selecting the lowest scoring k utterances from S_u.