Confidence threshold tuning

• US 8,396,715 B2
• Filed: 06/28/2005
• Issued: 03/12/2013
• Est. Priority Date: 06/28/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Odell, Julian J., Jiang, Li, Zhang, Wei, He, Xiaodong
• Primary Examiner(s): Dorvil, Richemond
• Assistant Examiner(s): ADESANYA, OLUJIMI A
• Application Number:
  US11/168,278
• Publication Number:
  US 20060293886A1
• Time in Patent Office: 2,814 Days
• Field of Search: 704/231, 704/251, 704/270, 704/275
• US Class Current: 704/270
• CPC Class Codes: G10L 15/08 Speech classification or search

Abstract

An expected dialog-turn (ED) value is estimated for evaluating a speech application. Parameters such as a confidence threshold setting can be adjusted based on the expected dialog-turn value. In a particular example, recognition results and corresponding confidence scores are used to estimate the expected dialog-turn value. The recognition results can be associated with a possible outcome for the speech application and a cost for the possible outcome can be used to estimate the expected dialog-turn value.

Claims

1. A computer-implemented method comprising:

identifying possible outcomes for at least one step in a speech application having a plurality of user configurable speech application parameters and assigning a cost for each possible outcome in terms of expected dialog turns, wherein identifying possible outcomes for the at least one step in the speech application comprises developing a cost matrix for the at least one step, wherein the cost matrix includes a probability and associated cost for each of the possible outcomes for the at least one step, the associated cost comprising a floating cost, the floating cost comprising a situation when a first dialog-turn is wasted requiring a user to initiate another dialog-turn, the floating cost comprising a large number when the user is rejected a first time, starts over, and there is a high possibility that the user will be rejected a next time, the floating cost comprising a small number when the user is rejected the first time, starts over, and there is a high possibility that the user will not be rejected the next time;

accessing recognition results and corresponding confidence scores for the at least one step in the speech application;

associating each recognition result and corresponding confidence score with one of the possible outcomes based on at least one application parameter from the plurality of user configurable speech application parameters configured by the user;

calculating a cost for the at least one step based on the association of each recognition result and corresponding confidence score with one of the possible outcomes and the cost for each possible outcome; and

adjusting, using a processor of the computer, the at least one parameter from the plurality of user configurable speech application parameters configured by the user to reduce the cost for the at least one step.

2. The computer implemented method of claim 1 wherein the at least one parameter includes a confidence threshold.

3. The computer-implemented method of claim 1 wherein the at least one parameter includes:

a high confidence threshold wherein a recognition result having a corresponding confidence score above the high confidence threshold is accepted; and

a low confidence threshold wherein a recognition result having a corresponding confidence score below the low confidence threshold is rejected and wherein a recognition result with a corresponding confidence score between the high confidence threshold and the low confidence threshold initiates a confirmation process with a user.

4. The method of claim 3 and further comprising:

determining a number of recognition results having a corresponding confidence score above the high confidence threshold, a number of recognition results having a corresponding confidence score below the low confidence threshold and a number of recognition results having a corresponding confidence score between the high confidence threshold and the low confidence threshold; and

calculating the cost for the at least one step based on each number.

5. The method of claim 1 and further comprising:

assigning a false acceptance penalty to the speech application; and

estimating the cost for the at least one step based on the false acceptance penalty.

6. The method of claim 5 wherein the false acceptance penalty is assigned based on a desired acceptance rate for the speech application.

7. The method of claim 1 and further comprising determining a number of recognition results associated with each possible outcome and calculating the cost for the at least one step based on the number.

20. The method of claim 1, wherein the associated cost for each of the possible outcomes comprises an indication of an expected number of dialog-turns to complete the at least one step in the speech application.

8. A computer-implemented method, comprising:

identifying possible outcomes for at least one step in a speech application having a plurality of user configurable speech application parameters and assigning a cost to each possible outcome in terms of expected dialog-turns, wherein identifying possible outcomes for the at least one step in the speech application comprises developing a cost matrix for the at least one step, wherein the cost matrix includes a probability and associated cost for each of the possible outcomes for the at least one step, the associated cost comprising a floating cost, the floating cost comprising a situation when a first dialog-turn is wasted requiring a user to initiate another dialog-turn, the floating cost comprising a large number when the user is rejected a first time, starts over, and there is a high possibility that the user will be rejected a next time, the floating cost comprising a small number when the user is rejected the first time, starts over, and there is a high possibility that the user will not be rejected the next time;

accessing recognition results and corresponding confidence scores for the at least one step in the speech application;

estimating a first expected dialog-turn value of the speech application for a first set of confidence thresholds based on the recognition results and corresponding confidence scores, wherein the first set of confidence thresholds comprises at least a pair of threshold values;

estimating a second expected dialog-turn value of the speech application for a second set of confidence thresholds based on the recognition results and corresponding confidence scores, wherein the second set of confidence thresholds comprises at least a pair of threshold values;

comparing, using a processor of the computer, the first expected dialog turn value and the second expected dialog turn value;

selecting one of the first set and second set of confidence thresholds for the speech application, wherein the first set of confidence thresholds is selected if the first expected dialog-turn value is less than the second expected dialog-turn value and the second set of confidence thresholds is selected for the speech application if the second expected dialogturn value is less than the first expected dialog-turn value; and

adjusting, using a processor of the computer, at least one parameter from the plurality of user configurable speech application parameters to reduce the cost for the at least one step.

9. The method of claim 8 and further comprising:

identifying possible outcomes for the speech application and assigning a cost to each outcome.

10. The method of claim 9 and further comprising:

associating each recognition result and corresponding confidence score with one of the possible outcomes based on the first set of confidence thresholds and estimating the first expected dialog-turn value based on the cost for said one of the possible outcomes; and

associating each recognition result and corresponding confidence score with one of the possible outcomes based on the second set of confidence thresholds and estimating the second expected dialog-turn value based on the cost for said one of the possible outcomes.

11. The method of claim 8 and further comprising:

assigning a false acceptance penalty to the speech application.

12. The method of claim 11 and further comprising:

estimating the first expected dialog-turn value and the second expected dialog-turn value based on the false acceptance penalty.

13. A computer-implemented method comprising:

identifying possible outcomes for at least one step in a speech application having a plurality of user configurable speech application parameters and assigning a cost to each possible outcome in terms of expected dialog-turns, wherein identifying possible outcomes for the at least one step in the speech application comprises developing a cost matrix for the at least one step, wherein the cost matrix includes a probability and associated cast for each of the possible outcomes for the at least one step, the associated cost comprising a floating cost, the floating cost comprising a situation when a first dialog-turn is wasted requiring a user to initiate another dialog-turn, the floating cost comprising a large number when the user is rejected a first time, starts over, and there is a high possibility that the user will be rejected a next time, the floating cost comprising a small number when the user is rejected the first time, starts over, and there is a high possibility that the user will not be rejected the next time;

accessing recognition results and corresponding confidence scores for the at least one step in the speech application;

identifying a plurality of confidence thresholds, wherein the plurality of confidence thresholds comprise pairs of confidence thresholds, each pair of confidence thresholds comprising a low confidence threshold and a high confidence threshold, wherein, for each pair, recognition results having a corresponding confidence score below the low confidence threshold are rejected, recognition results having a corresponding confidence score above the high confidence threshold are accepted and recognition results having a corresponding confidence score between the low confidence threshold and the high confidence threshold initiate a confirmation process;

for each pair of confidence thresholds,

associating each recognition result and corresponding confidence score with one of the possible outcomes using the pair of confidence thresholds; and

estimating, using a processor of the computer, an expected dialog-turn value for the pair of confidence thresholds based on how many recognition results are associated with each possible outcome and the cost of each possible outcome;

selecting one of the pairs of confidence thresholds for the at least one step in the speech application based on the corresponding expected dialog-turn values for the confidence thresholds; and

adjusting, using the processor of the computer, at least one parameter from the plurality of user configurable speech application parameters to reduce the cost for the at least one step.

14. The method of claim 13 and further comprising:

assigning a false acceptance penalty to the speech application; and

estimating each expected dialog-turn value based on the false acceptance penalty.

15. The method of claim 13 and further comprising:

obtaining additional recognition results and corresponding confidence scores for the speech application; and

estimating a second expected dialog-turn value for each of the plurality of confidence thresholds based on how many recognition results and additional recognition results are associated with each possible outcome and the cost of each possible outcome.

16. The method of claim 13 and further comprising:

calculating a probability for each possible outcome by determining a number of recognition results associated with each possible outcome.

17. The method of claim 13, wherein selecting one of the pairs of confidence thresholds comprises:

selecting the threshold pair that has a minimum expected dialog-turn value.

18. The method of claim 13, wherein the probability for each of the possible outcomes is computed as the number of instances for the possible outcome divided by the total number of instances for all of the possible outcomes.

19. The method of claim 13, wherein the associated cost for each of the possible outcomes comprises an indication of an expected number of dialog-turns to complete the at least one step in the speech application.