Method and system of optimal selection strategy for statistical classifications in dialog systems

• US 8,050,929 B2
• Filed: 08/24/2007
• Issued: 11/01/2011
• Est. Priority Date: 08/24/2007
• Status: Active Grant

• Current Assignee: Robert Bosch GmbH
• Original Assignee: Robert Bosch GmbH
• Inventors: Weng, Fuliang, Hu, Junling, Liu, Xue, Morbini, Fabrizio
• Primary Examiner(s): CHAWAN, VIJAY B
• Application Number:
  US11/845,013
• Publication Number:
  US 20090055164A1
• Time in Patent Office: 1,530 Days
• Field of Search: 704/275, 704/270, 704/270.1, 704/240, 704/238, 704/236, 704/9, 715/764, 715/804, 715/858, 715/256, 715/205, 715/854, 715/209
• US Class Current: 704/270.1
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding

Abstract

An optimal selection or decision strategy is described through an example that includes use in dialog systems. The selection strategy or method includes receiving multiple predictions and multiple probabilities. The received predictions predict the content of a received input and each of the probabilities corresponds to one of the predictions. In an example dialog system, the received input includes an utterance. The selection method includes dynamically selecting a set of predictions from the received predictions by generating ranked predictions. The ranked predictions are generated by ordering the plurality of predictions according to descending probability.

Claims

1. A processor-based dialog system comprising:

a central processing unit;

a classifier circuit coupled to the central processing unit and configured to generate a plurality of predictions on a dialog act corresponding to an utterance received at an input; and

a decision circuit coupled to the central processing unit and configured to generate a plurality of probabilities each of which corresponds to a prediction, dynamically select a set of predictions from the plurality of predictions, and classify the utterance according to an accumulated probability of the set of predictions, wherein the accumulated probability is an accumulated probability of variable n-best outputs, wherein the value n is a non-fixed integer value that is determined dynamically based on a classifier distribution of the utterance.

2. The system of claim 1, wherein the decision circuit is configured to classify the utterance as understood or misunderstood when the accumulated probability of the set of predictions is at or above a threshold, and wherein the decision circuit is further configured to classify the utterance as not-understood when the accumulated probability of the set of predictions is below the threshold.

3. The system of claim 2, further comprising a performance measurement circuit coupled to the decision circuit and configured to derive a Receiver Operating Curve (ROC) that graphs a false positive rate and a true positive rate, by varying the threshold, and wherein an area defined by the ROC represents a quantification of performance of the decision circuit in classifying the utterance.

4. The system of claim 3, wherein the decision circuit is configured to dynamically select the set of predictions including generating ranked predictions by ordering the plurality of predictions according to descending probability.

5. The system of claim 4, wherein the decision circuit is configured to dynamically select the set of predictions by determining a sequential pair of predictions in the ranked predictions between which the difference in corresponding probabilities is greatest relative to any other sequential pair of predictions, wherein the sequential pair of predictions include a first prediction and a second prediction, the first prediction having a higher probability than the second prediction.

6. The system of claim 5, wherein the decision circuit is configured to dynamically select the set of predictions by identifying the set of predictions to include those predictions in a group bounded by a highest ranked prediction and the first prediction.

7. The system of claim 3, comprising a database maintained in a memory coupled to the decision circuit, wherein the decision circuit is configured to transfer utterances classified as not-understood to the database and the database is configured to store the utterances classified as not-understood.

8. The system of claim 7, comprising a re-training module coupled to the classifier circuit and the memory, wherein the re-training module is configured to dynamically re-train the classifier using a not-understood utterance prior to another plurality of predictions.

9. The system of claim 7, comprising a graphical user interface (GUI) controlling a display coupled to the memory, wherein the GUI is configured to generate a labeled utterance by applying a label to a not-understood utterance.

10. A computer-implemented method of classifying an utterance comprising:

receiving, in a classifier circuit of a processor-based dialog system, the utterance;

generating, in a decision circuit of the processor-based dialog system, a plurality of predictions on a dialog act corresponding to the utterance;

generating a plurality of probabilities each of which corresponds to a prediction;

dynamically selecting a set of predictions from the plurality of predictions; and

classifying, in a classifier circuit of the processor-based dialog system, the utterance according to an accumulated probability of the set of predictions, wherein the accumulated probability is an accumulated probability of variable n-best outputs, wherein the value n is a non-fixed integer value that is determined dynamically based on a classifier distribution of the utterance.

11. The method of claim 10, wherein the classifying includes classifying the utterance as understood or misunderstood when the accumulated probability of the set of predictions is at or above a threshold, and further wherein the classifying includes classifying the utterance as not-understood when the accumulated probability of the set of predictions is below a threshold.

12. The method of claim 11 further comprising quantifying, in a performance measurement circuit of the processor-based dialog system, a performance of the decision circuit by deriving a Receiver Operating Curve (ROC) that graphs a false positive rate and a true positive rate, by varying the threshold, and calculating an area defined by the ROC.

13. The method of claim 11, wherein a number of predictions in the set of predictions corresponds to the utterance.

14. The method of claim 11, wherein dynamically selecting the set of predictions comprises generating ranked predictions by ordering the plurality of predictions according to descending probability.

15. The method of claim 14, wherein dynamically selecting the set of predictions comprises determining a sequential pair of predictions in the ranked predictions between which the difference in corresponding probabilities is greatest relative to any other sequential pair of predictions, wherein the sequential pair of predictions include a first prediction and a second prediction, the first prediction having a higher probability than the second prediction.

16. The method of claim 15, wherein dynamically selecting the set of predictions comprises identifying the set of predictions to include those predictions in a group bounded by a highest ranked prediction and the first prediction.

17. The method of claim 10, comprising generating a second plurality of predictions on a second dialog act corresponding to a second utterance, the generating of the second plurality of predictions dynamically using data of at least one utterance classified as not-understood.

18. The method of claim 10, comprising:

storing any utterance classified as not-understood; and

generate labeled utterances by labeling a stored utterance.

19. The method of claim 18, comprising:

training a classifier using training data, wherein the classifier generates the plurality of predictions; and

dynamically re-training the classifier using the labeled utterances, the re-training occurring prior to generating a second plurality of predictions.

20. A physical, non-transitory computer readable media including executable instructions which, when executed in a processor-based dialog system, classify an utterance by:

receiving, in a classifier circuit of the processor-based dialog system, the utterance;

generating, in a decision circuit of the processor-based dialog system, a plurality of predictions on a dialog act corresponding to the utterance;

generating a plurality of probabilities each of which corresponds to a prediction;

dynamically selecting a set of predictions from the plurality of predictions; and

classifying, in a classifier circuit of the processor-based dialog system, the utterance according to an accumulated probability of the set of predictions, wherein the accumulated probability is an accumulated probability of variable n-best outputs, wherein the value n is a non-fixed integer value that is determined dynamically based on a classifier distribution of the utterance; and

quantifying, in a performance measurement circuit coupled to the decision circuit, the performance of the decision circuit in classifying the utterance by deriving a Receiver Operating Curve (ROC) that graphs a false positive rate and a true positive rate, by varying the threshold, and calculating an area defined by the ROC.

21. The media of claim 20, wherein execution of the instructions classifies the utterance as understood or misunderstood when the accumulated probability of the set of predictions is at or above a threshold, wherein execution of the instructions classifies the utterance as not-understood when the accumulated probability of the set of predictions is below a threshold.

22. The media of claim 20, wherein execution of the instructions dynamically selects the set of predictions including generating ranked predictions by ordering the plurality of predictions according to descending probability.

23. The media of claim 22, wherein execution of the instructions dynamically selects the set of predictions by determining a sequential pair of predictions in the ranked predictions between which the difference in corresponding probabilities is greatest relative to any other sequential pair of predictions, wherein the sequential pair of predictions include a first prediction and a second prediction, the first prediction having a higher probability than the second prediction.