Random confirmation in speech based systems

• US 7,788,103 B2
• Filed: 10/18/2004
• Issued: 08/31/2010
• Est. Priority Date: 10/18/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Marcus, Jeffrey N
• Primary Examiner(s): Dorvil; Richemond
• Assistant Examiner(s): Yen; Eric
• Application Number:
  US10/683,017
• Publication Number:
  US 20060085184A1
• Time in Patent Office: 2,143 Days
• Field of Search: 704/251, 704/257, 704270-275
• US Class Current: 704/275
• CPC Class Codes: G10L 15/08 Speech classification or search
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

Method and system are provided for performing random confirmation in a speech system. When a speech recognition result is received with an associated confidence score indicating a level of confidence with respect to the speech recognition result, a confirmation decision is made in terms of whether a confirmation is to be carried out based on the confidence score. The confirmation decision may be made in a random confirmation mode. A confirmation may be performed when the confirmation decision is to carry out a confirmation on the speech recognition result.

Claims

1. A computer implemented method for speech recognition output confirmation, comprising:

receiving an automatic speech recognition result with an associated score indicating a level of confidence with respect to the speech recognition result;

accepting a mode selection specifying when an automated confirmation is to be performed, said mode selection chosen from a group consisting of a deterministic mode, a random mode and an integrated mode, wherein the random mode comprises randomly determining whether to perform the automated confirmation;

determining, based on said accepted mode selection and said score, whether a confirmation is to be performed on the speech recognition result; and

if it is determined that said confirmation is to be performed, performing an automated confirmation to verify the automatic speech recognition result by activating a confirmation construction mechanism operatively configured to carry out a speech recognition confirmation.

2. The method according to claim 1, wherein said determining step when performed for said deterministic mode comprises:

determining whether the confidence score associated with the speech recognition result is within a pre-defined range;

performing a confirmation if the confidence score associated with the speech recognition result is within the pre-defined range.

3. The method according to claim 1, wherein said determining step when performed for said random mode comprises:

generating a random number;

determining whether said generated random number is within a pre-defined range;

checking, if the generated random number falls within said pre-defined range, whether said generated random number is less than a confirmation percentage parameter value; and

performing a confirmation if the generated random number is within the pre-defined range and is less than the confirmation percentage parameter value.

4. The method according to claim 3, wherein the confirmation percentage parameter value indicates an expected percentage of times that a speech recognition result is to be confirmed.

5. The method according to claim 1, wherein said integrated mode comprises:

making a deterministic decision to perform a confirmation;

making a random decision to perform a confirmation; and

combining the deterministic decision and the random decision to generate an integrated decision to perform a confirmation.

6. The method according to claim 1, further comprising:

analyzing confirmation results collected to generate relevant statistics;

utilizing the generated relevant statistics.

7. The method according to claim 6, wherein the relevant statistics include accuracy statistics indicating the performance of a speech recognizer that produces the speech recognition result and/or performance statistics indicating the performance of the speech recognizer in different ranges of the confidence score.

8. The method according to claim 6, further comprising collecting training data based on the confirmation results and the relevant statistics, wherein the training data includes at least one of the speech data based on which the speech recognition result is obtained, the confidence score associated with the speech recognition result, and the confirmation result associated with the speech recognition result.

9. The method according to claim 8, wherein said utilizing includes

monitoring the performance of the speech recognizer based on the confirmation results and the relevant statistics;

adapting the recognition performance of the speech recognizer by conducting re-training using at least a portion of the collected training data selected according to the relevant statistics;

adjusting the speech recognizer in terms of how a confidence score is computed based on the relevant statistics; and

/or

adjusting a pre-defined range for a confidence score, which is used to determine whether a confirmation is to be performed.

10. A speech recognition output confirmation system, comprising:

a decision mechanism configured to decide whether a confirmation is to be performed, said decision mechanism being pre-configured to make said decision employing a mode chosen from the group consisting of a deterministic mode, a random mode, and an integrated mode, wherein the random mode comprises randomly determining whether to perform the automated confirmation to verify a speech recognition result; and

a confirmation mechanism, in communication with said decision mechanism, said confirmation mechanism configured to perform the automated confirmation on the speech recognition result having an associated confidence score according to the decision made by said decision mechanism, wherein a confirmation construction mechanism carries out the automated confirmation, and generates a confirmation response.

11. The system according to claim 10, wherein the confirmation mechanism performs a confirmation employing a mode chosen from a group consisting of

a manual mode in which a human operator performs the confirmation;

and

an automated mode in which a speech recognition mechanism is used to confirm the speech recognition result.

12. The system according to claim 10, wherein the decision mechanism comprises:

a random determiner that decides whether a confirmation is to be performed randomly; and

a deterministic determiner that decides that a confirmation decision is to be performed deterministically.

13. The system according to claim 12, wherein the random determiner comprises:

a random number generator capable of generating a random number;

a random decision mechanism in communication with the random number generator, that determines whether a confirmation is to be performed based on the random number generated.

14. The system according to claim 13, further comprising a storage storing a pre-defined random confirmation range specifying a range for a random number.

15. The system according to claim 14, further comprising a storage for storing a confirmation percentage parameter value specifying a maximum percentage of randomly initiated confirmations to be performed, wherein said confirmation is to be performed when the random number falls within the predefined range and is less than the confirmation percentage parameter value.

16. The system according to claim 12, further comprising an integrated determiner comprising:

a deterministic decision mechanism that makes a decision based on a pre-defined range with respect to a confidence score associated with a speech recognition result; and

a decision integrator, in communication with the random determiner and the deterministic determiner that makes a decision based on the random decision and the deterministic decision.

17. A system, in accordance with claim 10, further comprising:

a self-monitoring and adaptation mechanism capable of collecting confirmation results.

18. The system according to claim 17, wherein the self-monitoring and adaptation mechanism monitors the performance of a speech recognizer producing the speech recognition result through the collected confirmation results.

19. The system according to claim 17, wherein the self-monitoring and adaptation mechanism adapts the speech recognizer producing the speech recognition result based on collected confirmation results.

20. The system according to claim 17, wherein the self-monitoring and adaptation mechanism comprises:

a confirmation result logger capable of collecting confirmation results;

a confirmation information analyzer capable of analyzing the collected confirmation results to generated relevant statistics.

21. The system according to claim 20, further comprising a performance monitoring mechanism capable of monitoring the performance based on the relevant statistics.

22. The system according to claim 20, further comprising an adaptive training mechanism capable of performing re-training of a speech recognition mechanism from which the speech recognition result is received according to the relevant statistics.

23. The system according to claim 20, further comprising a confirmation range adjuster capable of adjusting the deterministic range according to the relevant statistics.