SYSTEMS AND METHODS FOR REDUCING ANNOTATION TIME

• US 20080270130A1
• Filed: 07/01/2008
• Published: 10/30/2008
• Est. Priority Date: 04/04/2003
• Status: Active Grant

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: AT&T Corporation (AT&T, Inc.)
• Inventors: Riccardi, Giuseppe, Hakkani-Tur, Dilek Z., Hollister, Barbara B., Rahim, Mazin G., Tur, Gokhan, Wilson, James M., Rose, Lawrence Lyon, Bromberg, Ilana, Alonso, Tirso M., Stern, Daniel Leon
• Granted Patent: US 7,860,713 B2
• Time in Patent Office: Days
• US Class Current: 704/236
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 15/19 Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules
• CPC Class Codes: H04M 3/4936 Speech interaction details speech recognition per se G10L15/00

Abstract

Systems and methods for annotating speech data. The present invention reduces the time required to annotate speech data by selecting utterances for annotation that will be of greatest benefit. A selection module uses speech models, including speech recognition models and spoken language understanding models, to identify utterances that should be annotated based on criteria such as confidence scores generated by the models. These utterances are placed in an annotation list along with a type of annotation to be performed for the utterances and an order in which the annotation should proceed. The utterances in the annotation list can be annotated for speech recognition purposes, spoken language understanding purposes, labeling purposes, etc. The selection module can also select utterances for annotation based on previously annotated speech data and deficiencies in the various models.

Claims

1. In a system that uses annotated speech data, a method for annotating speech data by processing a portion of unannotated speech data with one or more models, the processing comprising:

generating a label for a particular utterance; and

including the particular utterance in an annotation list if the label does not match an existing label of the particular utterance.

2. The method of claim 1, further comprising:

evaluating a performance of the one or more models with respect to each utterance in the portion of unannotated speech data using a criterion; and

creating an annotation list that includes utterances that do not satisfy the criterion.

3. The method of claim 2, further comprising:

identifying an order in which the utterances on the annotation list are to be annotated.

4. The method of claim 3, wherein creating an annotation list that includes utterances that do not satisfy the criterion further comprises:

using system deficiencies in combination with the criterion to identify utterances to be included in the annotation list; and

using previously annotated speech data in combination with the criterion or the system deficiencies to identify utterances to be included in the annotation list.

5. The method of claim 3, further comprising:

searching the speech data for additional utterances that are similar to the utterances that do not satisfy the criterion; and

including the additional utterances in the annotation list.

6. The method of claim 1, wherein processing at least a portion of the speech data with one or more models, wherein each utterance in the portion of the speech data is evaluated using a criterion further comprises at least one of:

assigning each utterance in the portion of the speech data a confidence score, wherein utterances having a confidence score below a threshold confidence score are included in the annotation list;

evaluating a dialog context of each utterance in the portion of the speech data; and

evaluating a feature of each utterance in the portion of the speech data.

7. A system that collects speech data for use in developing a dialog application, the system for annotating the speech data for the dialog application, the system comprising:

a module configured to analyze unannotated speech data with one or more speech recognition models, wherein each utterance in the speech data receives a recognition confidence score;

a module configured to analyze the speech data that is not annotated with one or more spoken language understanding models, wherein each utterance in the speech data receives an understanding confidence score; and

a module configured to create an annotation list that includes at least a portion of the utterances having a recognition confidence score below a confidence threshold score and that includes at least a portion of the utterances having an understanding confidence score below an understanding threshold score.

8. The system of claim 7, wherein the module configured to analyze speech data that is not annotated with one or more speech recognition models further processes the speech data using one or more language models and one or more acoustic models.

9. The system of claim 7, wherein the module configured to analyze the speech data that is not annotated with one or more spoken language understanding models further processes speech data that has been recognized by the one or more speech recognition models.

10. The system of claim 7, further comprising:

a module configured to generate a call type for a particular utterance, wherein the call type is included in an annotation guide and wherein the particular utterance has an existing call type; and

a module configured to include the particular utterance in the annotation list if the call type of the particular utterance does not match the existing call type of the particular utterance.

11. The system of claim 7, wherein the module configured to create an annotation list further identifies a type of annotation to be performed for each utterance included in the annotation list.

12. The system of claim 11, wherein the module configured to create an annotation list further selects utterances to be included in the annotation list based on previously annotated speech data.

13. The system of claim 11, wherein the module configured to create an annotation list further selects utterances to be included in the annotation list based on deficiencies of the one or more speech understanding models or on deficiencies of the one or more spoken language understanding models.

14. The system of claim 11, wherein the module configured to create an annotation list further establishes an order in which the utterances included in the annotation list are to be annotated.

15. The system of claim 11, further comprising:

a module configured to search the speech data for additional utterances that are similar to the utterances having a recognition confidence score below a confidence threshold score and that are similar to utterances having an understanding confidence score that is lower than an understanding threshold score; and

a module configured to include the additional utterances in the annotation list.

16. A system that collects speech data for developing a dialog application, wherein the dialog application includes speech recognition models, spoken language understanding models, and labeling models, the system reducing the time required to annotate the speech data, the system comprising:

a module configured to select one or more utterances from speech data for annotation based on confidence scores of the one or more utterances, wherein the confidence scores are generated by at least one of;

speech recognition models, spoken language understanding models, and labeling models;

a module configured to select one or more utterances from the speech data for annotation based on deficiencies of a dialog application; and

a module configured to create an annotation list that includes the selected one or more utterances, wherein the annotation list identifies a type of annotation to be performed for each of the one or more utterances in the annotation list.