Confidence threshold tuning
First Claim
Patent Images
1. A computer-implemented method comprising:
- identifying possible outcomes for at least one step in a speech application having a plurality of user configurable speech application parameters and assigning a cost for each possible outcome in terms of expected dialog turns, wherein identifying possible outcomes for the at least one step in the speech application comprises developing a cost matrix for the at least one step, wherein the cost matrix includes a probability and associated cost for each of the possible outcomes for the at least one step, the associated cost comprising a floating cost, the floating cost comprising a situation when a first dialog-turn is wasted requiring a user to initiate another dialog-turn, the floating cost comprising a large number when the user is rejected a first time, starts over, and there is a high possibility that the user will be rejected a next time, the floating cost comprising a small number when the user is rejected the first time, starts over, and there is a high possibility that the user will not be rejected the next time;
accessing recognition results and corresponding confidence scores for the at least one step in the speech application;
associating each recognition result and corresponding confidence score with one of the possible outcomes based on at least one application parameter from the plurality of user configurable speech application parameters configured by the user;
calculating a cost for the at least one step based on the association of each recognition result and corresponding confidence score with one of the possible outcomes and the cost for each possible outcome; and
adjusting, using a processor of the computer, the at least one parameter from the plurality of user configurable speech application parameters configured by the user to reduce the cost for the at least one step.
3 Assignments
0 Petitions
Accused Products
Abstract
An expected dialog-turn (ED) value is estimated for evaluating a speech application. Parameters such as a confidence threshold setting can be adjusted based on the expected dialog-turn value. In a particular example, recognition results and corresponding confidence scores are used to estimate the expected dialog-turn value. The recognition results can be associated with a possible outcome for the speech application and a cost for the possible outcome can be used to estimate the expected dialog-turn value.
-
Citations
20 Claims
-
1. A computer-implemented method comprising:
-
identifying possible outcomes for at least one step in a speech application having a plurality of user configurable speech application parameters and assigning a cost for each possible outcome in terms of expected dialog turns, wherein identifying possible outcomes for the at least one step in the speech application comprises developing a cost matrix for the at least one step, wherein the cost matrix includes a probability and associated cost for each of the possible outcomes for the at least one step, the associated cost comprising a floating cost, the floating cost comprising a situation when a first dialog-turn is wasted requiring a user to initiate another dialog-turn, the floating cost comprising a large number when the user is rejected a first time, starts over, and there is a high possibility that the user will be rejected a next time, the floating cost comprising a small number when the user is rejected the first time, starts over, and there is a high possibility that the user will not be rejected the next time; accessing recognition results and corresponding confidence scores for the at least one step in the speech application; associating each recognition result and corresponding confidence score with one of the possible outcomes based on at least one application parameter from the plurality of user configurable speech application parameters configured by the user; calculating a cost for the at least one step based on the association of each recognition result and corresponding confidence score with one of the possible outcomes and the cost for each possible outcome; and adjusting, using a processor of the computer, the at least one parameter from the plurality of user configurable speech application parameters configured by the user to reduce the cost for the at least one step. - View Dependent Claims (2, 3, 4, 5, 6, 7, 20)
-
-
8. A computer-implemented method, comprising:
-
identifying possible outcomes for at least one step in a speech application having a plurality of user configurable speech application parameters and assigning a cost to each possible outcome in terms of expected dialog-turns, wherein identifying possible outcomes for the at least one step in the speech application comprises developing a cost matrix for the at least one step, wherein the cost matrix includes a probability and associated cost for each of the possible outcomes for the at least one step, the associated cost comprising a floating cost, the floating cost comprising a situation when a first dialog-turn is wasted requiring a user to initiate another dialog-turn, the floating cost comprising a large number when the user is rejected a first time, starts over, and there is a high possibility that the user will be rejected a next time, the floating cost comprising a small number when the user is rejected the first time, starts over, and there is a high possibility that the user will not be rejected the next time; accessing recognition results and corresponding confidence scores for the at least one step in the speech application; estimating a first expected dialog-turn value of the speech application for a first set of confidence thresholds based on the recognition results and corresponding confidence scores, wherein the first set of confidence thresholds comprises at least a pair of threshold values; estimating a second expected dialog-turn value of the speech application for a second set of confidence thresholds based on the recognition results and corresponding confidence scores, wherein the second set of confidence thresholds comprises at least a pair of threshold values; comparing, using a processor of the computer, the first expected dialog turn value and the second expected dialog turn value; selecting one of the first set and second set of confidence thresholds for the speech application, wherein the first set of confidence thresholds is selected if the first expected dialog-turn value is less than the second expected dialog-turn value and the second set of confidence thresholds is selected for the speech application if the second expected dialogturn value is less than the first expected dialog-turn value; and adjusting, using a processor of the computer, at least one parameter from the plurality of user configurable speech application parameters to reduce the cost for the at least one step. - View Dependent Claims (9, 10, 11, 12)
-
-
13. A computer-implemented method comprising:
-
identifying possible outcomes for at least one step in a speech application having a plurality of user configurable speech application parameters and assigning a cost to each possible outcome in terms of expected dialog-turns, wherein identifying possible outcomes for the at least one step in the speech application comprises developing a cost matrix for the at least one step, wherein the cost matrix includes a probability and associated cast for each of the possible outcomes for the at least one step, the associated cost comprising a floating cost, the floating cost comprising a situation when a first dialog-turn is wasted requiring a user to initiate another dialog-turn, the floating cost comprising a large number when the user is rejected a first time, starts over, and there is a high possibility that the user will be rejected a next time, the floating cost comprising a small number when the user is rejected the first time, starts over, and there is a high possibility that the user will not be rejected the next time; accessing recognition results and corresponding confidence scores for the at least one step in the speech application; identifying a plurality of confidence thresholds, wherein the plurality of confidence thresholds comprise pairs of confidence thresholds, each pair of confidence thresholds comprising a low confidence threshold and a high confidence threshold, wherein, for each pair, recognition results having a corresponding confidence score below the low confidence threshold are rejected, recognition results having a corresponding confidence score above the high confidence threshold are accepted and recognition results having a corresponding confidence score between the low confidence threshold and the high confidence threshold initiate a confirmation process; for each pair of confidence thresholds, associating each recognition result and corresponding confidence score with one of the possible outcomes using the pair of confidence thresholds; and estimating, using a processor of the computer, an expected dialog-turn value for the pair of confidence thresholds based on how many recognition results are associated with each possible outcome and the cost of each possible outcome; selecting one of the pairs of confidence thresholds for the at least one step in the speech application based on the corresponding expected dialog-turn values for the confidence thresholds; and adjusting, using the processor of the computer, at least one parameter from the plurality of user configurable speech application parameters to reduce the cost for the at least one step. - View Dependent Claims (14, 15, 16, 17, 18, 19)
-
Specification