System and method for improving performance of semantic classifiers in spoken dialog systems

• US 8,543,401 B2
• Filed: 04/17/2009
• Issued: 09/24/2013
• Est. Priority Date: 04/17/2009
• Status: Active Grant

• Current Assignee: Verint Americas Incorporated (Verint Systems Incorporated)
• Original Assignee: Synchronoss Technologies Incorporated
• Inventors: Suendermann, David, Evanini, Keelan, Liscombe, Jackson, Dayanidhi, Krishna, Pieraccini, Roberto
• Primary Examiner(s): Guerra-Erazo, Edgar
• Assistant Examiner(s): Sharma, Neeraj
• Application Number:
  US12/425,892
• Publication Number:
  US 20100268536A1
• Time in Patent Office: 1,621 Days
• Field of Search: 704/257, 704/5, 704/9, 704/10, 704/243, 704/275, 706/54, 382/224, 707/104.1, 707/5, 707/100
• US Class Current: 704/243
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/0635 updating or merging of old and new templates; Mean values; Weighting

Abstract

A method and apparatus for continuously improving the performance of semantic classifiers in the scope of spoken dialog systems are disclosed. Rule-based or statistical classifiers are replaced with better performing rule-based or statistical classifiers and/or certain parameters of existing classifiers are modified. The replacement classifiers or new parameters are trained and tested on a collection of transcriptions and annotations of utterances which are generated manually or in a partially automated fashion. Automated quality assurance leads to more accurate training and testing data, higher classification performance, and feedback into the design of the spoken dialog system by suggesting changes to improve system behavior.

Claims

1. A method for optimizing speech processing in a spoken dialog system, comprising a computer including a processor, and memory, wherein the computer is configured to execute at least one of the steps of the method, the method comprising:

providing the spoken dialog system with an initial set of semantic classifiers;

collecting and processing a plurality of utterances using the semantic classifiers with the spoken dialog system;

transcribing the collected set of utterances;

annotating the collected utterances with a semantic category;

applying a quality assurance criterion to the annotated utterances;

if the quality assurance criterion is met, training a classifier update candidate for the initial set of semantic classifiers using data from the annotated utterances;

comparing a performance of the update candidate against the initial set of semantic classifiers by testing the update candidate and the initial set of semantic classifiers against a baseline criterion; and

upgrading the initial set of semantic classifiers of the spoken dialog system with the update candidate if the update candidate outperforms the initial set of classifiers;

wherein the quality assurance criteria are selected from at least two or more of the group consisting of;

completeness of the annotated utterances;

consistency of the annotated utterances;

congruence of the annotated utterances;

correlation of the annotated utterances;

confusion of the annotated utterances;

coverage of the annotated utterances; and

corpus size of the annotated utterances.

2. The method of claim 1 wherein the utterance collection is selected from the group consisting of:

all contexts of the spoken dialog system;

a subset of contexts of the spoken dialog system;

a fixed number of utterances per context of the spoken dialog system; and

all utterances processed by the spoken dialog system in a certain time frame.

3. The method of claim 1 wherein the method further includes:

deriving testing data and training data from the data from the annotated utterances.

4. The method of claim 3 wherein the method further includes:

further deriving development data and training data from the training data.

5. The method of claim 4 wherein the development data is used to optimize the parameters used to generate the classifier update candidate.

6. The method of claim 1 wherein the method includes:

logging data associated with the collected set of utterances.

7. The method of claim 6 wherein the logging data includes:

storing instances of speech recognition events identified by a speech recognition device at the semantic classifiers together with logging data for the semantic classifiers.

8. The method of claim 7 wherein the logging data is selected from the group consisting of:

a name and version of an active classifier;

a classifier name and version resulting in a highest classification score of the utterance;

date and time in which the utterance was recognized;

data about a context in which the utterance was recognized;

the speech recognizer'"'"'s hypothesis of the utterance;

an acoustic confidence score associated with the speech recognizer'"'"'s hypothesis of the utterance;

the speech recognizer'"'"'s n best hypotheses of the utterance;

an acoustic confidence score associated with the n best hypotheses of the utterance;

a semantic classification, result for the speech recognizer'"'"'s hypothesis;

a semantic confidence score associated with the semantic classification result;

m best semantic classification results selected from a plurality of the semantic classification results for the speech recognizers hypothesis;

a semantic confidence score associated with the in best semantic classification results;

m·

n best semantic classification results for the n best speech recognition hypotheses;

a semantic confidence score associated with the m·

n best semantic classification results;

speech data;

data about a spoken dialog system;

data about a caller; and

data about the utterance.

9. The method of claim 1 wherein the trained, update candidate is used to replace classifier parameters with new parameters while keeping the initial classifier in place.

10. The method of claim 1 wherein the trained update candidate is used to replace the initial classifier with a new classifier.

11. The method of claim 10 wherein the new classifier is selected from the group consisting of a rule-based classifier, a statistical classifier, or a classifier that is part rule-based and part statistical.

12. The method of claim 1 wherein the method further includes:

using the data from the annotated utterances to automate either or both of the transcription and annotation of utterances.

13. The method of claim 12 wherein the training further includes

training a speech recognizer using the data from the transcribed utterances, the speech recognizer being configured to respond to specific data processed by a specific spoken dialog system.

14. The method of claim 13 wherein the training is selected from the group consisting of:

training of language models;

training of acoustic models;

training a global speech recognizer; and

training a context-dependent speech recognizer.

15. The method of claim 14 wherein the method further includes:

automating the transcription by transcribing the utterances using the trained speech recognizer.

16. The method of claim 15 wherein the method further includes:

receiving a confidence score for the utterance from the speech recognizer;

establishing a threshold corresponding to the confidence score;

testing the confidence score against the threshold; and

accepting the automated transcription if the confidence score meets the threshold.

17. The method of claim 16 wherein establishing a threshold corresponding to the confidence score includes:

establishing a maximum error threshold;

applying the speech recognizer to a set of test utterances distinct from the utterances the speech recognizer was trained on producing speech recognition hypotheses and confidence scores for each of the processed utterances; and

determining the minimum confidence score for which those utterances whose confidence score exceeds this minimum score produce an error lower than the maximum error threshold.

18. The method in claim 17 wherein the error threshold is selected from the group of word error rate and sentence error rate.

19. The method of claim 12 wherein the annotation includes partially automating the annotation.

20. The method of claim 12 wherein the method includes:

performing automated annotation of utterances based on models trained on the data from the annotated utterances.

21. The method of claim 12 wherein an utterance that is similar or identical to a previously annotated utterance is automatedly assigned the same semantic class as the previously annotated utterance.

22. The method of claim 12 wherein the automated annotation includes applying an initial classifier to the utterances to be annotated.

23. The method of claim 22 wherein the initial classifier is selected from the group consisting of a statistical classifier, a rule-based classifier, and a classifier that is partially rule-based, partially statistical.

24. The method of claim 1 wherein the comparing includes:

measuring baseline performance by applying a baseline classifier to a speech recognition hypotheses generated by the speech recognizer on the stored data about utterances.

25. The method of claim 1 wherein the comparing includes:

measuring the baseline performance by comparing output classes produced by the set of classifiers used by the system applied to a speech recognition hypotheses generated by the speech recognizer for given utterances with the annotations of the same utterances.

26. The method of claim 1 wherein the upgrading is selected from the group consisting of:

manually upgrading the initial semantic classifier with the update candidate; and

performing an automated upgrade of the initial semantic classifier with the update candidate.

27. The method of claim 1 wherein the method further includes:

repeating the optimization process in an iterative cycle.

28. The method of claim 27 wherein the method further comprises, when the update candidate does not outperform the initial semantic classifier;

maintaining the initial semantic classifier;

continuing the collection of utterances; and

repeating the optimization process.

29. The method of claim 1 wherein the method further comprises, when the initial set of semantic classifier are upgraded with the update candidate,

repeating the optimization process using the updated classifiers as the initial set of classifiers in a next iteration.

30. The method of claim 1 wherein the method further comprises;

providing a remote hosting service for at least one of the steps of the optimization process.

31. The method of claim 1 wherein the method further comprises:

incorporating a module into a spoken dialog system for at least one of the steps of the optimization process.

32. A spoken dialog system, comprising:

a computer including a processor, and memory, including;

a signal input for receiving an audio input;

a speech recognition engine;

a data store comprising an initial set of semantic classifiers;

a data store for a plurality of utterances received via the audio input;

a data store for storing transcribed utterances;

a data store for storing annotated utterances;

a semantic classifier component including a semantic classifier program for, when executed by the processor, mapping the utterances to a set of semantic classes;

a quality assurance component including a program for, when executed by the processor, applying quality assurance criteria to the annotated utterances;

a classifier update generation component including a program for, when executed by the processor, generating an update candidate for the initial set of semantic classifiers using data from the annotated utterances if a quality assurance criterion is met;

and a comparison component including a program for, when executed by the processor, comparing a performance of the update candidate against the initial set of semantic classifiers by testing the update candidate and the initial set of semantic classifiers against a baseline criterion, whereby the initial set of semantic classifiers is updated with the update candidate if the update candidate outperforms the initial set of classifiers,

wherein the quality assurance criteria are selected from at least two or more of the group consisting of;

completeness of the annotated utterances;

consistency of the annotated utterances;

congruence of the annotated utterances;

correlation of the annotated utterances;

confusion of the annotated utterances;

coverage of the annotated utterances; and

corpus size of the annotated utterances.

33. The system of claim 32 wherein the system further includes:

a data derivation component including a program for deriving testing data and training data from the annotated utterances.

34. The system of claim 33 wherein the system further includes:

an update generation speech recognition engine included in the update generation component and

the deriving component further includes program instructions for further separating the training data into training data and development data.

35. The system of claim 34 wherein the system further includes:

program instructions for, when executed by the processor, optimizing the parameters used to generate the classifier update candidate using the development data.

36. The system of claim 32 wherein the system includes:

a logging component including program for, when executed by a processor, logging and storing data associated with the collected set of utterances.

37. The system of claim 36 wherein the logging data includes:

a data store for storing instances of speech recognition events identified by the speech recognition engine at the semantic classifiers together with logging data for the semantic classifiers.

38. The system of claim 32 wherein the generated update candidate is used to replace classifier parameters with new parameters while keeping the initial classifier in place.

39. The system of claim 32 wherein the generated update candidate is used to replace the initial classifier with a new classifier.

40. The system of claim 32 wherein the system further includes:

a transcription component including a program that, when executed by a processor automates the transcription of the utterances.

41. The system of claim 32 wherein the system further includes:

an annotation component including a program that, when executed by a processor, automates the annotation of the utterances.

42. The system of claim 32 wherein the system further includes:

a program for, when executed by a processor, training a speech recognition engine using the data from the annotated utterances, the speech recognition engine being configured to respond to specific data processed by a specific spoken dialog system.

43. The system of claim 42 wherein the training program includes program instruction for, when executed by a processor:

training of language models;

training of acoustic models;

training a global speech recognizer; and

training a context-dependent speech recognizer.

44. The system of claim 42 wherein the system further includes:

a transcription component including a program that, when executed by a processor, automates the transcription of the utterances;

wherein said transcription component includes the trained speech recognition engine.

45. The system of claim 44 wherein the transcription component further includes program instructions for, when executed by the processor:

receiving a confidence score for the utterance from the speech recognition engine;

establishing a threshold corresponding to the confidence score;

testing the confidence score against the threshold; and

accepting the automated transcription if the confidence score meets the threshold.

46. The system of claim 45 wherein the program instructions further include:

establishing a maximum error threshold;

applying the speech recognition engine to a set of test utterances distinct from the utterances the speech recognition engine was trained on producing speech recognition hypotheses and confidence scores for each of the processed utterances; and

determining the minimum confidence score for which those utterances whose confidence score exceeds this minimum score produce an error lower than the maximum error threshold thereby establishing a threshold corresponding to the confidence score.

47. The system of claim 46 wherein the error threshold is selected from the group of word error rate and sentence error rate.

48. The system of claim 32 wherein the system further comprises;

a comparison speech recognition engine included, in the comparison component;

and the comparison program including program instructions for, when executed by the processor, measuring baseline performance by applying a baseline classifier to the speech recognition hypotheses generated by the comparison speech recognition engine on the stored data about utterances.

49. The system of claim 32 wherein the comparison program includes program instructions for:

measuring the baseline performance by comparing the output classes produced by the set of classifiers used by the system applied to the speech recognition engine'"'"'s hypotheses for given utterances with the annotations of the same utterances.

50. The system of claim 32 wherein the system includes a upgrading program that, when executed by the processor, comprises:

program instructions for performing an automated upgrade of the initial semantic classifier with the update candidate.

51. The system program of claim 32 wherein the upgrading program further includes program instructions for:

repeating the optimization process in an iterative cycle.

52. The system of claim 32 wherein the system further comprises:

a remote hosting service for at least one of the components of the system.

53. The system of claim 32 wherein the at least one of the components of the system includes a separate module that can be incorporated into a spoken dialog system.