System and method for improving performance of semantic classifiers in spoken dialog systems
First Claim
1. A method for optimizing speech processing in a spoken dialog system, comprising a computer including a processor, and memory, wherein the computer is configured to execute at least one of the steps of the method, the method comprising:
- providing the spoken dialog system with an initial set of semantic classifiers;
collecting and processing a plurality of utterances using the semantic classifiers with the spoken dialog system;
transcribing the collected set of utterances;
annotating the collected utterances with a semantic category;
applying a quality assurance criterion to the annotated utterances;
if the quality assurance criterion is met, training a classifier update candidate for the initial set of semantic classifiers using data from the annotated utterances;
comparing a performance of the update candidate against the initial set of semantic classifiers by testing the update candidate and the initial set of semantic classifiers against a baseline criterion; and
upgrading the initial set of semantic classifiers of the spoken dialog system with the update candidate if the update candidate outperforms the initial set of classifiers;
wherein the quality assurance criteria are selected from at least two or more of the group consisting of;
completeness of the annotated utterances;
consistency of the annotated utterances;
congruence of the annotated utterances;
correlation of the annotated utterances;
confusion of the annotated utterances;
coverage of the annotated utterances; and
corpus size of the annotated utterances.
6 Assignments
0 Petitions
Accused Products
Abstract
A method and apparatus for continuously improving the performance of semantic classifiers in the scope of spoken dialog systems are disclosed. Rule-based or statistical classifiers are replaced with better performing rule-based or statistical classifiers and/or certain parameters of existing classifiers are modified. The replacement classifiers or new parameters are trained and tested on a collection of transcriptions and annotations of utterances which are generated manually or in a partially automated fashion. Automated quality assurance leads to more accurate training and testing data, higher classification performance, and feedback into the design of the spoken dialog system by suggesting changes to improve system behavior.
-
Citations
53 Claims
-
1. A method for optimizing speech processing in a spoken dialog system, comprising a computer including a processor, and memory, wherein the computer is configured to execute at least one of the steps of the method, the method comprising:
-
providing the spoken dialog system with an initial set of semantic classifiers; collecting and processing a plurality of utterances using the semantic classifiers with the spoken dialog system; transcribing the collected set of utterances; annotating the collected utterances with a semantic category; applying a quality assurance criterion to the annotated utterances; if the quality assurance criterion is met, training a classifier update candidate for the initial set of semantic classifiers using data from the annotated utterances; comparing a performance of the update candidate against the initial set of semantic classifiers by testing the update candidate and the initial set of semantic classifiers against a baseline criterion; and upgrading the initial set of semantic classifiers of the spoken dialog system with the update candidate if the update candidate outperforms the initial set of classifiers; wherein the quality assurance criteria are selected from at least two or more of the group consisting of; completeness of the annotated utterances; consistency of the annotated utterances; congruence of the annotated utterances; correlation of the annotated utterances; confusion of the annotated utterances; coverage of the annotated utterances; and corpus size of the annotated utterances. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31)
-
-
32. A spoken dialog system, comprising:
-
a computer including a processor, and memory, including; a signal input for receiving an audio input; a speech recognition engine; a data store comprising an initial set of semantic classifiers; a data store for a plurality of utterances received via the audio input; a data store for storing transcribed utterances; a data store for storing annotated utterances; a semantic classifier component including a semantic classifier program for, when executed by the processor, mapping the utterances to a set of semantic classes; a quality assurance component including a program for, when executed by the processor, applying quality assurance criteria to the annotated utterances; a classifier update generation component including a program for, when executed by the processor, generating an update candidate for the initial set of semantic classifiers using data from the annotated utterances if a quality assurance criterion is met; and a comparison component including a program for, when executed by the processor, comparing a performance of the update candidate against the initial set of semantic classifiers by testing the update candidate and the initial set of semantic classifiers against a baseline criterion, whereby the initial set of semantic classifiers is updated with the update candidate if the update candidate outperforms the initial set of classifiers, wherein the quality assurance criteria are selected from at least two or more of the group consisting of; completeness of the annotated utterances; consistency of the annotated utterances; congruence of the annotated utterances; correlation of the annotated utterances; confusion of the annotated utterances; coverage of the annotated utterances; and corpus size of the annotated utterances. - View Dependent Claims (33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53)
-
Specification