Confidence threshold tuning

• US 20060293886A1
• Filed: 06/28/2005
• Published: 12/28/2006
• Est. Priority Date: 06/28/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Wei, Jiang, Li, Odell, Julian J., He, Xiaodong
• Granted Patent: US 8,396,715 B2
• Time in Patent Office: Days
• US Class Current: 704/231
• CPC Class Codes: G10L 15/08 Speech classification or search

Abstract

An expected dialog-turn (ED) value is estimated for evaluating a speech application. Parameters such as a confidence threshold setting can be adjusted based on the expected dialog-turn value. In a particular example, recognition results and corresponding confidence scores are used to estimate the expected dialog-turn value. The recognition results can be associated with a possible outcome for the speech application and a cost for the possible outcome can be used to estimate the expected dialog-turn value.

Claims

1. A computer-implemented method comprising:

identifying possible outcomes for at least one step in a speech application and assigning a cost for each possible outcome in terms of expected dialog turns;

accessing recognition results and corresponding confidence scores for the at least one step in the speech application;

associating each recognition result and corresponding confidence score with one of the possible outcomes based on at least one application parameter;

calculating a cost for the at least one step based on the association of each recognition result and corresponding confidence score with one of the possible outcomes and the cost for each possible outcome; and

adjusting the at least one parameter of the speech application to reduce the cost for the at least one step.

2. The computer implemented method of claim 1 wherein the at least one parameter includes a confidence threshold.

3. The computer-implemented method of claim 1 wherein the at least one parameter includes:

a high confidence threshold wherein a recognition result having a corresponding confidence score above the high confidence threshold is accepted and;

a low confidence threshold wherein a recognition result having a corresponding confidence score below the low confidence threshold is rejected and wherein a recognition result with a corresponding confidence score between the high confidence threshold and the low confidence threshold initiates a confirmation process with a user.

4. The method of claim 3 and further comprising:

determining a number of recognition results having a corresponding confidence score above the high confidence threshold, a number of recognition results having a corresponding confidence score below the low confidence threshold and a number of recognition results having a corresponding confidence score between the high confidence threshold and the low confidence threshold; and

calculating the cost for the at least one step based on each number.

5. The method of claim 1 and further comprising:

assigning a false acceptance penalty to the speech application; and

estimating the cost for the at least one step based on the false acceptance penalty.

6. The method of claim 1 wherein the false acceptance penalty is assigned based on a desired acceptance rate for the speech application.

7. The method of claim 1 and further comprising determining a number of recognition results associated with each possible outcome and calculating the cost for the at least one step based on the number.

8. A computer-implemented method, comprising:

accessing recognition results and corresponding confidence scores for a speech application;

estimating a first expected dialog-turn value of the speech application for a first confidence threshold based on the recognition results and corresponding confidence scores;

estimating a second expected dialog-turn value of the speech application for a second confidence threshold based on the recognition results and corresponding confidence scores; and

comparing the first expected dialog turn value and the second expected dialog turn value.

9. The method of claim 8 and further comprising:

identifying possible outcomes for the speech application and assigning a cost to each outcome.

10. The method of claim 9 and further comprising:

associating each recognition result and corresponding confidence score with one of the possible outcomes based on the first confidence threshold and estimating the first expected dialog-turn value based on the cost for said one of the possible outcomes; and

associating each recognition result and corresponding confidence score with one of the possible outcomes based on the second confidence threshold and estimating the second expected dialog-turn value based on the cost for said one of the possible outcomes.

11. The method of claim 8 and further comprising:

assigning a false acceptance penalty to the speech application.

12. The method of claim 11 and further comprising:

estimating the first expected dialog-turn value and the second expected dialog-turn value based on the false acceptance penalty.

13. The method of claim 8 and further comprising:

selecting the first confidence threshold for the speech application if the first expected dialog-turn value is less than the second expected dialog-turn value and selecting the second confidence threshold for the speech application if the second expected dialog-turn value is less than the first expected dialog-turn value.

14. A computer-implemented method comprising:

identifying possible outcomes for a speech application and assigning a cost to each possible outcome;

accessing recognition results and corresponding confidence scores for the speech application;

associating each recognition result and corresponding confidence score with one of the possible outcomes for a plurality of confidence thresholds; and

estimating an expected dialog-turn value for each of the plurality of confidence thresholds based on how many recognition results are associated with each possible outcome and the cost of each possible outcome.

15. The method of claim 14 and further comprising:

selecting one threshold of the plurality of confidence thresholds based on the corresponding expected dialog turn value for the one threshold.

16. The method of claim 14 wherein the plurality of confidence thresholds are part of a pair including a low confidence threshold and a high confidence threshold.

17. The method of claim 16 wherein, for each pair, recognition results having a corresponding confidence score below the low confidence threshold are rejected, recognition results having a corresponding confidence score above the high confidence threshold are accepted and recognition results having a corresponding confidence score between the low confidence threshold and the high confidence threshold initiate a confirmation process.

18. The method of claim 14 and further comprising:

assigning a false acceptance penalty to the speech application; and

estimating each expected dialog-turn value based on the false acceptance penalty.

19. The method of claim 14 and further comprising:

obtaining additional recognition results and corresponding confidence scores for the speech application; and

estimating a second expected dialog-turn value for each of the plurality of confidence thresholds based on how many recognition results and additional recognition results are associated with each possible outcome and the cost of each possible outcome.

20. The method of claim 14 and further comprising:

calculating a probability for each possible outcome by determining a number of recognition results associated with each possible outcome.