SYSTEM AND METHOD FOR IMPROVING PERFORMANCE OF SEMANTIC CLASSIFIERS IN SPOKEN DIALOG SYSTEMS
First Claim
1. A method for optimizing speech processing in a spoken dialog system, comprising:
- providing an initial set of semantic classifiers;
collecting and processing a plurality of utterances using the semantic classifiers;
transcribing the collected set of utterances;
annotating the collected utterances with a semantic category;
applying a quality assurance criterion to the annotated utterances;
training a classifier update candidate for the initial set of semantic classifiers using data from the annotated utterances;
comparing a performance of the update candidate against the initial set of semantic classifiers by testing the update candidate and the initial set of semantic classifiers against a baseline criterion; and
upgrading the initial set of semantic classifiers with the update candidate if the update candidate outperforms the initial set of classifiers.
6 Assignments
0 Petitions
Accused Products
Abstract
A method and apparatus for continuously improving the performance of semantic classifiers in the scope of spoken dialog systems are disclosed. Rule-based or statistical classifiers are replaced with better performing rule-based or statistical classifiers and/or certain parameters of existing classifiers are modified. The replacement classifiers or new parameters are trained and tested on a collection of transcriptions and annotations of utterances which are generated manually or in a partially automated fashion. Automated quality assurance leads to more accurate training and testing data, higher classification performance, and feedback into the design of the spoken dialog system by suggesting changes to improve system behavior.
69 Citations
55 Claims
-
1. A method for optimizing speech processing in a spoken dialog system, comprising:
-
providing an initial set of semantic classifiers; collecting and processing a plurality of utterances using the semantic classifiers; transcribing the collected set of utterances; annotating the collected utterances with a semantic category; applying a quality assurance criterion to the annotated utterances; training a classifier update candidate for the initial set of semantic classifiers using data from the annotated utterances; comparing a performance of the update candidate against the initial set of semantic classifiers by testing the update candidate and the initial set of semantic classifiers against a baseline criterion; and upgrading the initial set of semantic classifiers with the update candidate if the update candidate outperforms the initial set of classifiers. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 43)
-
-
33. A spoken dialog system, comprising:
a computer including a processor, and memory, including; a signal input for receiving an audio input; a speech recognition engine; a data store comprising an initial set of semantic classifiers; a data store for a plurality of utterances received via the audio input; a data store for storing transcribed utterances; a data store for storing annotated utterances; a semantic classifier component including a semantic classifier program for, when executed by the processor, mapping the utterances to a set of semantic classes; a quality assurance component including a program for, when executed by the processor, applying quality assurance criteria to the annotated utterances; a classifier update generation component including a program for, when executed by the processor, generating an update candidate for the initial set of semantic classifiers using data from the annotated utterances; and a comparison component including a program for, when executed by the processor, comparing a performance of the update candidate against the initial set of semantic classifiers by testing the update candidate and the initial set of semantic classifiers against a baseline criterion, whereby the initial set of semantic classifiers is updated with the update candidate if the update candidate outperforms the initial set of classifiers. - View Dependent Claims (34, 35, 36, 37, 38, 39, 40, 41, 42, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55)
Specification