Method and system having hypothesis type variable thresholds

• US 8,725,512 B2
• Filed: 03/13/2007
• Issued: 05/13/2014
• Est. Priority Date: 03/13/2007
• Status: Expired due to Fees

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Claiborn, David, Goel, Vaibhava, Gopinath, Ramesh, Pethachi, Pichappan
• Primary Examiner(s): He, Jialong
• Application Number:
  US11/685,328
• Publication Number:
  US 20080228486A1
• Time in Patent Office: 2,618 Days
• Field of Search: 704/270, 704/270.1, 704/275
• US Class Current: 704/270
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

A method (and system) for spoken dialog confirmation classifies a plurality of spoken dialog hypotheses, and assigns a threshold to each class of spoken dialog hypotheses.

Claims

1. A method of configuring a system to perform spoken dialog confirmation, comprising:

operating at least one programmed processor to perform;

classifying a plurality of spoken dialog hypotheses into a plurality of classes of spoken dialog hypotheses, each of the plurality of spoken dialog hypotheses being a potential semantic interpretation that a semantic interpreter is able to output as a result of a semantic interpretation process performed on an input, the plurality of classes comprising a first class of spoken dialog hypotheses comprising at least one first spoken dialog hypothesis and a second class of spoken dialog hypotheses comprising at least one second spoken dialog hypothesis, the at least one first spoken dialog hypothesis being different from the at least one second spoken dialog hypothesis; and

determining, for each of said plurality of classes of spoken dialog hypotheses, a threshold for use in evaluating a confidence level of the semantic interpreter in an output of the semantic interpreter that comprises one of the one or more hypotheses in the class, wherein the determining comprises determining a first threshold for the first class and a second threshold for the second class, the first threshold having a different value than the second threshold, wherein the determining comprises adjusting the first threshold based at least in part on a number of times the at least one first spoken dialog hypothesis was previously output by the semantic interpreter.

2. The method of claim 1, further comprising operating the at least one programmed processor to perform acts of:

receiving an utterance as an input;

generating a spoken dialog hypothesis and a confidence value for said utterance;

determining a corresponding threshold for said spoken dialog hypothesis based upon said classifying; and

comparing said confidence value of said spoken dialog hypothesis to each threshold.

3. The method of claim 2, further comprising operating the at least one programmed processor to perform accepting said spoken dialog hypothesis if said confidence value exceeds said threshold.

4. The method of claim 2, further comprising operating the at least one programmed processor to perform rejecting said spoken dialog hypothesis if said confidence value does not exceed said threshold.

5. The method of claim 1, wherein said determining a threshold to each class comprises determining a confirmation threshold and a rejection threshold to each class, and

wherein the method further comprises operating the at least one programmed processor to recognize an utterance spoken by a user by:

receiving the utterance as an input;

generating a spoken dialog hypothesis and a confidence value for said utterance;

determining corresponding rejection and confirmation thresholds for said spoken dialog hypothesis based upon said classifying;

comparing said confidence value of said spoken dialog hypothesis to said thresholds;

accepting said spoken dialog hypothesis if said confidence value exceeds said confirmation threshold;

confirming said spoken dialog hypothesis if said confidence value does not exceed said confirmation threshold but exceeds said rejection threshold; and

rejecting said spoken dialog hypothesis if said confidence value does not exceed said rejection threshold and said spoken dialog hypothesis is not confirmed.

6. The method of claim 1, wherein said spoken dialog hypotheses are classified according to a difficulty of recognition.

7. A non-transitory machine-readable storage medium having encoded thereon machine-executable instructions that, when executed by a digital processing unit, causes the digital processing unit to perform the method of claim 1.

8. The method of claim 1, further comprising operating a speech recognition system to carry out a speech interaction with a user, wherein operating the speech recognition system comprises operating the at least one processor to perform the semantic interpreting, the determining, and the comparing.

9. The method of claim 1, wherein adjusting the first threshold based at least in part on the number of times the at least one first spoken dialog hypothesis was previously output by the semantic interpreter comprises applying an optimization algorithm to adjust the first threshold based at least in part on one or more indications of whether, when one of the at least one first spoken dialog hypothesis was output in each of the number of times, it was determined that the one of the at least one first spoken dialog hypothesis was a correct interpretation of input to the semantic interpreter.

10. The method of claim 1, wherein adjusting the first threshold based at least in part on a number of times the at least one first spoken dialog hypothesis was previously output by the semantic interpreter comprises adjusting the first threshold in indirect proportion to the number of times the at least one first spoken dialog hypothesis was previously output by the semantic interpreter.

11. The method of claim 10, wherein adjusting the first threshold in proportion to the number of times the at least one first spoken dialog hypothesis was previously output by the semantic interpreter comprises adjusting the first threshold in indirect proportion to the number of times the at least one first spoken dialog hypothesis was, when previously output, determined to be a correct interpretation of input to the semantic interpreter.

12. A system for spoken dialog confirmation, comprising:

at least one processor programmed to act as;

a spoken dialog hypothesis classifier that classifies a plurality of hypotheses into a plurality of classes of spoken dialog hypotheses, each of the plurality of spoken dialog hypotheses being a potential semantic interpretation that a semantic interpreter is able to output as a result of a semantic interpretation process performed on an input, the plurality of classes comprising a first class of spoken dialog hypotheses comprising at least one first spoken dialog hypothesis and a second class of spoken dialog hypotheses comprising at least one second spoken dialog hypothesis, the at least one first spoken dialog hypothesis being different from the at least one second spoken dialog hypothesis; and

a threshold assigner that determines, for each of said plurality of classes of spoken dialog hypotheses, a threshold for use in evaluating a confidence level of the semantic interpreter in an output of the semantic interpreter that comprises one of the one or more hypotheses in the class, wherein the threshold assigner determines a first threshold for the first class and a second threshold for the second class, the first threshold having a different value than the second threshold, wherein the determining comprises adjusting the first threshold based at least in part on a number of times the at least one first spoken dialog hypothesis was previously output by the semantic interpreter.

13. The system of claim 12, wherein the at least one processor is further programmed to act as a spoken dialog hypothesis generator that generates at least one of the plurality of spoken dialog hypotheses and a confidence value for an input from a user.

14. The system of claim 13, wherein the at least one processor is further programmed to act as a comparator that compares said confidence value to said threshold.

15. The system of claim 14, wherein said comparator outputs said spoken dialog hypothesis as an accepted spoken dialog hypothesis if said confidence value exceeds said threshold.

16. The system of claim 12, wherein said threshold assigner assigns said threshold based upon the classification by said spoken dialog hypothesis classifier.

17. The system of claim 12, wherein said spoken dialog hypothesis classifier classifies said plurality of spoken dialog hypotheses based upon a frequency of occurrence of said spoken dialog hypotheses.

18. The system of claim 17, wherein said threshold assigner determines a smaller value for the first threshold as the frequency of occurrence of one or more hypotheses in the first class increases.

19. The system of claim 17, wherein said threshold assigner determines a larger value for the first threshold as the frequency of occurrence of one or more hypotheses in the first class decreases.

20. A system for recognizing speech input provided by a user, the system comprising:

at least one processor programmed to;

for each class of a plurality of classes of spoken dialog hypotheses, determine a threshold for use in evaluating a confidence level of the semantic interpreter in an output of the semantic interpreter that comprises one of the one or more hypotheses in the class, each of the plurality of spoken dialog hypotheses being a potential semantic interpretation that a semantic interpreter is able to output as a result of a semantic interpretation process performed on an input, wherein the determining comprises determining a first threshold for a first class of spoken dialog hypotheses and a second threshold for a second class of spoken dialog hypotheses, the first threshold having a different value than the second threshold, the first class comprising at least one first spoken dialog hypothesis and the second class comprising at least one second spoken dialog hypothesis, the at least one first spoken dialog hypothesis being different from the at least one second spoken dialog hypothesis, wherein the determining comprises adjusting the first threshold based at least in part on a number of times the at least one first spoken dialog hypothesis was previously output by the semantic interpreter;

for a first input received from the user, semantically interpret the first input to determine a hypothesis for the first input, from the plurality of spoken dialog hypotheses, and a confidence value;

determine a class of the plurality of classes of spoken dialog hypotheses into which the hypothesis is classified and, based on the class, at least one threshold associated with the class for use in evaluating a confidence level of a hypothesis; and

compare the confidence value for the hypothesis to the at least one threshold associated with the class of the hypothesis.

21. The system of claim 20, wherein the at least one threshold in the sets of thresholds associated with the class comprises an acceptance threshold and a rejection threshold, and

wherein the at least one processor is programmed to compare the confidence value to the at least one threshold by:

accepting the speech recognition hypothesis when the confidence value exceeds the acceptance threshold;

rejecting the speech recognition hypothesis when the confidence value does not exceed the rejection threshold; and

prompting the user regarding the spoken dialog hypothesis when the confidence value exceeds the rejection threshold but does not exceed the confirmation threshold.