Random confirmation in speech based systems

• US 7,844,465 B2
• Filed: 11/30/2004
• Issued: 11/30/2010
• Est. Priority Date: 11/30/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation), ScanSoft, Inc. n/k/a Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Marcus, Jeffrey N
• Primary Examiner(s): YEN, ERIC L
• Application Number:
  US10/999,773
• Publication Number:
  US 20060116876A1
• Time in Patent Office: 2,191 Days
• Field of Search: 704/231, 704270-275
• US Class Current: 704/275
• CPC Class Codes: G10L 15/08 Speech classification or search

Abstract

Method and system are provided for performing random confirmation in a speech system. When a speech recognition result is received with an associated confidence score indicating a level of confidence with respect to the speech recognition result, a confirmation decision is made in terms of whether a confirmation is to be carried out based on the confidence score. The confirmation decision may be made in a random confirmation mode. A confirmation may be performed when the confirmation decision is to carry out a confirmation on the speech recognition result.

Claims

1. A dialogue system, comprising:

a dialogue system front end configured to interact with a user; and

a dialogue system tuning tool in communication with the dialogue system front end, wherein the dialogue system tuning tool is configured to provide feedback control to the dialogue system front end based, at least in part, on confirmation results collected from confirmations performed by the dialog system tuning tool in at least one mode including a random confirmation mode, wherein in the random confirmation mode, a determination of whether to perform a confirmation to verify a speech recognition result of speech input from the user is randomly determined in response to a determination that a confidence score associated with the speech recognition result is outside of a predetermined range.

2. The system according to claim 1, wherein the dialogue system tuning tool comprises:

a random mode enabled confirmation mechanism configured to perform a confirmation on a speech recognition result received from the dialogue system front end, wherein the confirmation is performed in a deterministic confirmation mode, the random confirmation mode, or an integrated confirmation mode; and

a self-monitoring and adaptation mechanism configured to monitor a performance of the dialogue system front end by analyzing the collected confirmation results.

3. The system according to claim 2, wherein the random mode enabled confirmation mechanism comprises:

a confirmation decision mechanism configured to make a confirmation decision; and

a confirmation construction mechanism in communication with the confirmation decision mechanism, configured to generate a confirmation on the speech recognition result received from the dialogue system front end according to the confirmation decision made by the confirmation decision mechanism.

4. The system according to claim 3, wherein the confirmation decision mechanism comprises:

a random confirmation determiner configured to make a random confirmation decision; and

a confirmation determiner configured to make a deterministic confirmation decision and/or an integrated confirmation decision.

5. The system according to claim 4, wherein the random confirmation determiner comprises:

a random number generator configured to generate a random number; and

a random confirmation decision mechanism in communication with the random number generator, wherein the random confirmation decision mechanism is configured to make the random confirmation decision based, at least in part, on the random number generated.

6. The system according to claim 4, wherein the confirmation determiner comprises:

a deterministic confirmation decision mechanism configured to make a deterministic confirmation decision based on whether a confidence score associated with the speech recognition result is within a pre-defined deterministic confirmation range; and

a confirmation decision integrator in communication with the random confirmation decision mechanism and the deterministic confirmation decision mechanism, wherein the confirmation decision integrator is configured to make a confirmation decision based, at least in part, on the random confirmation decision and the deterministic confirmation decision.

7. The system according to claim 2, wherein the self-monitoring and adaptation mechanism is further configured to adapt the dialogue system front end based on the collected confirmation results.

8. The system according to claim 7, wherein the self-monitoring and adaptation mechanism comprises:

a confirmation result logger configured to collect the confirmation results; and

a confirmation information analyzer configured to generate statistics based on the collected confirmation results.

9. The system according to claim 8, further comprising:

a performance monitoring mechanism configured to monitor the performance of the dialog system front end based on the statistics.

10. The system according to claim 8, further comprising:

an adaptive training mechanism configured to re-train a speech recognition mechanism of the dialog system front end based, at least in part, on the statistics.

11. The system according to claim 8, further comprising:

a confirmation range adjuster configured to adjust the deterministic confirmation range based, at least in part, on the statistics.

12. A call center for receiving at least one call from at least one caller, the call center comprising:

a dialogue system configured to produce a speech recognition result based on speech received from the at least one caller during the at least one call;

a call routing mechanism configured to route the at least one call from the at least one caller based on the speech recognition result, wherein the dialogue system is further configured to perform a confirmation in at least one mode including a random confirmation mode, wherein in the random confirmation mode, a determination of whether to perform a confirmation to verify the speech recognition result is randomly determined in response to a determination that a confidence score associated with the speech recognition result is outside of a predetermined range, and to collect confirmation results.

13. The call center according to claim 12, wherein the dialogue system is further configured to monitor a performance based, at least in part, on the collected confirmation results.

14. The call center according to claim 12, wherein the dialogue system is further configured to tune a performance based, at least in part, on the collected confirmation results.

15. The call center according to claim 12, wherein the dialogue system comprises:

a dialogue system front end configured to produce the speech recognition result; and

a dialogue system tuning tool in communication with the dialogue system front end, wherein the dialogue system tuning tool is configured to provide feedback control to the dialogue system front end based, at least in part, on the confirmation results collected from confirmations performed in a deterministic confirmation mode, the random confirmation mode, or an integrated confirmation mode.

16. The call center according to claim 15, wherein the dialogue system tuning tool comprises:

a random mode enabled confirmation mechanism configured to perform a confirmation on the speech recognition result in one of the deterministic confirmation mode, the random confirmation mode, and the integrated confirmation mode; and

a self-monitoring and adaptation mechanism configured to monitor a performance of the dialogue system front end by analyzing the collected confirmation results.

17. The call center according to claim 16, wherein the random mode enabled confirmation mechanism comprises:

a confirmation decision mechanism configured to make a confirmation decision; and

a confirmation construction mechanism in communication with the confirmation decision mechanism, configured to generate a confirmation on the speech recognition result received from the dialogue system front end according to the confirmation decision made by the confirmation decision mechanism.

18. The call center according to claim 17, wherein the confirmation decision mechanism comprises:

a random confirmation determiner configured to make a random confirmation decision; and

a confirmation determiner configured to make a deterministic confirmation decision and/or an integrated confirmation decision.

19. The call center according to claim 16, wherein the self-monitoring and adaptation mechanism comprises:

a confirmation result logger configured to collect the confirmation results; and

a confirmation information analyzer configured to generate statistics based on the collected confirmation results.

20. The call center according to claim 19, further comprising:

a performance monitoring mechanism configured to monitor the performance of the dialog system front end based, at least in part, on the statistics.

21. The call center according to claim 19, further comprising:

an adaptive training mechanism configured to re-train a speech recognition mechanism of the dialog system front end based, at least in part, on the statistics.

22. A method, comprising:

receiving an automatic speech recognition result from a speech recognition processor and a confidence score associated with the automatic speech recognition result;

making a confirmation decision, with at least one processor, to determine whether to perform a confirmation on the automatic speech recognition result based, at least in part, on a random confirmation mode, wherein in the random confirmation mode, the determination of whether to perform the confirmation to verify the automatic speech recognition result is randomly determined in response to a determination that a confidence score associated with the automatic speech recognition result is outside of a predetermined range; and

performing the confirmation if it is determined that the confirmation is to be performed.

23. The method of claim 22, wherein the confirmation decision is made based, at least in part, on a deterministic confirmation mode, the random confirmation mode, or an integrated confirmation mode.

24. The method of claim 22, wherein the random confirmation mode comprises:

generating a random number; and

making the confirmation decision based, at least in part, on the generated random number.

25. The method of claim 23, wherein the deterministic confirmation mode comprises:

making the confirmation decision based on whether the confidence score associated with the speech recognition result is within a pre-defined deterministic confirmation range.

26. The method of claim 22, further comprising:

collecting results of confirmations performed by the at least one processor; and

tuning an operation of the speech recognition processor based on the collected confirmation results.

27. The method of claim 26, further comprising:

generating statistics based on the collected confirmation results, wherein the operation of the speech recognition processor is tuned based on the statistics.

28. The method of claim 27, further comprising:

monitoring a performance of the speech recognition processor based on the statistics.

29. The method of claim 27, further comprising:

re-training a speech recognition model used by the speech recognition processor based, at least in part, on the statistics.

30. The method of claim 27, further comprising:

adjusting a deterministic confirmation range based, at least in part, on the statistics.

31. A non-transitory computer-readable storage medium encoded with a plurality of instructions that when executed by a computer, perform a method comprising:

receiving an automatic speech recognition result from a speech recognition processor and a confidence score associated with the automatic speech recognition result;

making a confirmation decision, to determine whether to perform a confirmation on the automatic speech recognition result based, at least in part, on a random confirmation mode, wherein in the random confirmation mode, a determination of whether to perform a confirmation to verify the automatic speech recognition result is randomly determined in response to a determination that the confidence score associated with the automatic speech recognition result is outside of a predetermined range; and

performing the confirmation if it is determined that the confirmation is to be performed.

32. The non-transitory computer-readable storage medium of claim 31, wherein the confirmation decision is made based, at least in part, on a deterministic confirmation mode, the random confirmation mode, or an integrated confirmation mode.

33. The non-transitory computer-readable storage medium of claim 31, wherein the random confirmation mode comprises:

generating a random number; and

making the confirmation decision based, at least in part, on the generated random number.

34. The non-transitory computer-readable storage medium of claim 32, wherein the deterministic confirmation mode comprises:

making the confirmation decision based on whether the confidence score associated with the speech recognition result is within a pre-defined deterministic confirmation range.

35. The non-transitory computer-readable storage medium of claim 31, wherein the method further comprises:

collecting confirmation results of performed confirmations; and

tuning an operation of the speech recognition processor based on the collected confirmation results.

36. The non-transitory computer-readable storage medium of claim 35, wherein the method further comprises:

generating statistics based on the collected confirmation results, wherein the operation of the speech recognition processor is tuned based on the statistics.

37. The non-transitory computer-readable storage medium of claim 35, wherein the method further comprises:

monitoring a performance of the speech recognition processor based on the statistics.

38. The non-transitory computer-readable storage medium of claim 35, wherein the method further comprises:

re-training a speech recognition model used by the speech recognition processor based, at least in part, on the statistics.

39. The non-transitory computer-readable storage medium of claim 35, wherein the method further comprises:

adjusting a deterministic confirmation range based, at least in part, on the statistics.