Random confirmation in speech based systems

• US 20060116876A1
• Filed: 11/30/2004
• Published: 06/01/2006
• Est. Priority Date: 11/30/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Marcus, Jeffrey N.
• Granted Patent: US 7,844,465 B2
• Time in Patent Office: Days
• US Class Current: 704/231
• CPC Class Codes: G10L 15/08 Speech classification or search

Abstract

Method and system are provided for performing random confirmation in a speech system. When a speech recognition result is received with an associated confidence score indicating a level of confidence with respect to the speech recognition result, a confirmation decision is made in terms of whether a confirmation is to be carried out based on the confidence score. The confirmation decision may be made in a random confirmation mode. A confirmation may be performed when the confirmation decision is to carry out a confirmation on the speech recognition result.

Claims

1. A method, comprising:

receiving an automatic speech recognition result with an associated score indicating a level of confidence with respect to the speech recognition result;

determining whether a confirmation is to be performed on the speech recognition result;

performing a confirmation if it is determined that a confirmation is to be carried out, wherein the confirmation can be performed in a confirmation mode including a random confirmation mode.

2. The method according to claim 1, wherein the confirmation mode is pre-configured as one of a deterministic confirmation mode, a random confirmation mode, and an integrated confirmation mode.

3. The method according to claim 1, wherein said determining comprises:

examining the pre-configured confirmation mode;

determining, in the pre-configured confirmation mode, whether a confirmation is to be performed in the pre-configured confirmation mode.

4. The method according to claim 3, wherein

the deterministic confirmation is determined based on a deterministic confirmation decision made according to a pre-defined range for a confidence score associated with a speech recognition result;

the random confirmation is determined based on a random confirmation decision made according to a pre-defined range for a random number; and

the integrated confirmation is determined based on an integrated confirmation decision according to both the deterministic confirmation decision and the random confirmation decision.

5. The method according to claim 4, wherein making a deterministic confirmation decision comprises:

determining whether the confidence score associated with the speech recognition result is within the pre-defined range for a confidence score;

determining to perform a confirmation if the confidence score associated with the speech recognition result is within the pre-defined range for a confidence score.

6. The method according to claim 4, wherein making a random confirmation decision comprises:

generating a random number;

determining whether the generated random number is within the pre-defined range for a random number;

checking, if the generated random number falls within the pre-defined range of a random number, whether the generated random number satisfies a confirmation percentage parameter; and

determining to perform a random confirmation if the generated random number is within the pre-defined range and satisfies the confirmation percentage parameter.

7. The method according to claim 6, wherein the confirmation percentage parameter indicates an expected percentage of times that a speech recognition result is to be randomly confirmed.

8. The method according to claim 4, wherein making the integrated confirmation comprises:

making a deterministic confirmation decision;

making a random confirmation decision; and

combining the deterministic confirmation decision and the random confirmation decision to produce the integrated confirmation decision on whether a confirmation is to be performed.

9. The method according to claim 4, wherein said performing a confirmation comprises:

carrying out a deterministic confirmation if the confirmation configuration corresponds to the deterministic confirmation mode and the deterministic confirmation decision is to perform a confirmation;

carrying out a random confirmation if the confirmation configuration corresponds to the random confirmation mode and the random confirmation decision is to perform a random confirmation; and

carrying out an integrated confirmation if the confirmation configuration corresponds to the integrated confirmation mode and the integrated confirmation decision is to perform a confirmation.

10. The method according to claim 1, further comprising configuring, prior to said receiving, the confirmation to be one of a deterministic confirmation mode, a random confirmation mode, and an integrated confirmation mode.

11. A method, comprising:

receiving an automatic speech recognition result with an associated score indicating a level of confidence with respect to the speech recognition result;

determining whether a confirmation is to be performed on the speech recognition result, wherein the confirmation can be performed in a confirmation mode including a random confirmation mode;

performing confirmation if it is determined that a confirmation is to be carried out; and

collecting results from the confirmation after said performing.

12. The method according to claim 11, wherein the confirmation mode is pre-configured as one of a deterministic confirmation mode, a random confirmation mode, and an integrated confirmation mode.

13. The method according to claim 11, wherein said determining comprises:

examining the pre-configured confirmation mode;

determining, in the pre-configured confirmation mode, whether a confirmation is to be performed in the pre-configured confirmation mode.

14. The method according to claim 13, wherein

the deterministic confirmation is determined based on a deterministic confirmation decision made according to a pre-defined range for a confidence score associated with a speech recognition result;

the random confirmation is determined based on a random confirmation decision made according to a pre-defined range for a random number; and

the integrated confirmation is determined based on an integrated confirmation decision according to both the deterministic confirmation decision and the random confirmation decision.

15. The method according to claim 14, wherein making a deterministic confirmation decision comprises:

determining whether the confidence score associated with the speech recognition result is within the pre-defined range for a confidence score;

determining to perform a confirmation if the confidence score associated with the speech recognition result is within the pre-defined range for a confidence score.

16. The method according to claim 14, wherein making a random confirmation decision comprises:

generating a random number;

determining whether the generated random number is within the pre-defined range for a random number;

checking, if the generated random number falls within the pre-defined range of a random number, whether the generated random number satisfies a confirmation percentage parameter; and

determining to perform a random confirmation if the generated random number is within the pre-defined range and satisfies the confirmation percentage parameter.

17. The method according to claim 14, wherein making the integrated confirmation comprises:

making a deterministic confirmation decision;

making a random confirmation decision; and

combining the deterministic confirmation decision and the random confirmation decision to produce the integrated confirmation decision on whether a confirmation is to be performed.

18. The method according to claim 14, wherein said performing a confirmation comprises:

carrying out a deterministic confirmation if the confirmation configuration corresponds to the deterministic confirmation mode and the deterministic confirmation decision is to perform a confirmation;

carrying out a random confirmation if the confirmation configuration corresponds to the random confirmation mode and the random confirmation decision is to perform a random confirmation; and

carrying out an integrated confirmation if the confirmation configuration corresponds to the integrated confirmation mode and the integrated confirmation decision is to perform a confirmation.

19. The method according to claim 11, wherein a confirmation result collected after said performing indicates one of:

the speech recognition result is correct; and

the speech recognition result is incorrect.

20. The method according to claim 11, further comprising:

analyzing confirmation results collected to generate relevant statistics;

utilizing the generated relevant statistics.

21. The method according to claim 20, wherein the relevant statistics include at least one of:

accuracy statistics indicating the performance of a speech recognizer that produces the speech recognition result; and

performance statistics indicating the performance of the speech recognizer in different ranges of the confidence score.

22. The method according to claim 20, further comprising collecting training data based on the confirmation results and the relevant statistics, wherein the training data includes at least one of:

the speech data based on which the speech recognition result is obtained;

the confidence score associated with the speech recognition result; and

the confirmation result associated with the speech recognition result.

23. The method according to claim 22, wherein said utilizing includes at least one of:

monitoring the performance of the speech recognizer based on the confirmation results and the relevant statistics;

adapting the recognition performance of the speech recognizer by conducting re-training using at least a portion of the collected training data selected according to the relevant statistics;

adjusting the speech recognizer in terms of how a confidence score is computed based on the relevant statistics; and

adjusting a pre-defined range for a confidence score, which is used to determine whether a confirmation is to be performed.

24. A system, comprising:

a confirmation decision mechanism; and

a confirmation mechanism in communication with the confirmation decision mechanism, capable of carrying out a confirmation on a speech recognition result having an associated confidence score according to a confirmation decision made by the confirmation decision mechanism, wherein the confirmation decision mechanism is capable of making the confirmation decision in terms of whether a confirmation is to be performed in a mode including a random confirmation mode.

25. The system according to claim 24, wherein the confirmation mode is pre-configured as one of a deterministic confirmation mode, a random confirmation mode, and an integrated confirmation mode.

26. The system according to claim 24, wherein the confirmation mechanism performs a confirmation via means including one of:

a manual mode in which a human operator performs the confirmation; and

an automated mode in which a text to speech mechanism is used to confirm the speech recognition result.

27. The system according to claim 24, wherein the confirmation decision mechanism comprises:

a random confirmation determiner capable of making a random confirmation decision in terms of whether a random confirmation is to be performed; and

a confirmation determiner capable of making one of a deterministic confirmation decision and an integrated confirmation decision.

28. The system according to claim 27, wherein the random confirmation determiner comprises:

a random number generator capable of generating a random number;

a random confirmation decision mechanism in communication with the random number generator, capable of determining whether a confirmation is to be performed based on the random number generated.

29. The system according to claim 28, further comprising a storage storing a pre-defined random confirmation range specifying a range for a random number.

30. The system according to claim 29, further comprising a storage for storing a confirmation percentage parameter specifying a maximum percentage of random confirmation to be performed, wherein a random confirmation is to be performed when the random number falls within the pre-defined random confirmation range and satisfies the confirmation percentage parameter.

31. The system according to claim 27, wherein the confirmation determiner comprises:

a deterministic confirmation decision mechanism capable of making a deterministic confirmation decision based on a pre-defined deterministic confirmation range with respect to a confidence score associated with a speech recognition result; and

a confirmation decision integrator in communication with the random confirmation decision mechanism and the deterministic confirmation decision mechanism, capable of making a confirmation decision based on the random confirmation decision and the deterministic conformation decision.

32. A system, comprising:

a confirmation decision mechanism;

a confirmation mechanism in communication with the confirmation decision mechanism, capable of carrying out a confirmation on a speech recognition result according to a confirmation decision made by the confirmation decision mechanism, wherein the confirmation decision mechanism is capable of making the confirmation decision in terms of whether a confirmation is to be performed in a mode including a random confirmation mode; and

a self-monitoring and adaptation mechanism capable of collecting confirmation results.

33. The system according to claim 32, wherein the confirmation mode is pre-configured as one of a deterministic confirmation mode, a random confirmation mode, and an integrated confirmation mode.

34. The system according to claim 32, wherein the speech recognition result is associated with a confidence score indicating a level of confidence with respect to the speech recognition result.

35. The system according to claim 34, wherein the confirmation decision mechanism comprises:

a random confirmation determiner capable of making a random confirmation decision in terms of whether a random confirmation is to be performed; and

a confirmation determiner capable of making one of a deterministic confirmation decision and an integrated confirmation decision.

36. The system according to claim 35, wherein the random confirmation determiner comprises:

a random number generator capable of generating a random number;

a random confirmation decision mechanism in communication with the random number generator, capable of determining whether a confirmation is to be performed based on the random number generated.

37. The system according to claim 36, further comprising a storage storing a pre-defined random confirmation range specifying a range for a random number.

38. The system according to claim 37, further comprising a storage for storing a confirmation percentage parameter specifying a maximum percentage of random confirmation to be performed, wherein a random confirmation is to be performed when the random number falls within the pre-defined random confirmation range and satisfies the confirmation percentage parameter.

39. The system according to claim 35, wherein the confirmation determiner comprises:

a deterministic confirmation decision mechanism capable of making a deterministic confirmation decision based on a pre-defined deterministic confirmation range with respect to a confidence score associated with a speech recognition result; and

a confirmation decision integrator in communication with the random confirmation decision mechanism and the deterministic confirmation decision mechanism, capable of making a confirmation decision based on the random confirmation decision and the deterministic conformation decision.

40. The system according to claim 32, wherein the self-monitoring and adaptation mechanism is further capable of monitoring the performance of a speech recognizer producing the speech recognition result through the collected confirmation results.

41. The system according to claim 32, wherein the self-monitoring and adaptation mechanism is further capable of adapting the speech recognizer producing the speech recognition result based on collected confirmation results.

42. The system according to claim 32, wherein the self-monitoring and adaptation mechanism comprises:

a confirmation result logger capable of collecting conformation results;

a confirmation information analyzer capable of analyzing the collected confirmation results to generated relevant statistics.

43. The system according to claim 42, further comprising a performance monitoring mechanism capable of monitoring the performance based on the relevant statistics.

44. The system according to claim 42, further comprising an adaptive training mechanism capable of performing re-training of a speech recognition mechanism from which the speech recognition result is received according to the relevant statistics.

45. The system according to claim 42, further comprising a confirmation range adjuster capable of adjusting the deterministic conformation range according to the relevant statistics.

46. A dialogue system, comprising:

a dialogue system front end capable of communicating with a user using speech;

a dialogue system tuning tool in communication with the dialogue system front end capable of monitoring and tuning the performance of the dialogue system front end based on confirmation results collected from confirmations performed in a mode including a random confirmation mode.

47. The system according to claim 46, wherein the dialogue system tuning tool comprises:

a random mode enabled confirmation mechanism capable of performing a confirmation on a speech recognition result produced by the dialogue system front end in one of a deterministic confirmation mode, a random confirmation mode, and an integrated confirmation mode; and

a self-monitoring and adaptation mechanism capable of monitoring the performance of the dialogue system front end through the collected confirmation results.

48. The system according to claim 47, wherein the random mode enabled confirmation mechanism comprises:

a confirmation decision mechanism configured to make a confirmation decision; and

a confirmation mechanism in communication with the confirmation decision mechanism, capable of carrying out a confirmation on a speech recognition result from the dialogue system front end according to a confirmation decision made by the confirmation decision mechanism.

49. The system according to claim 48, wherein the confirmation decision mechanism comprises:

a random confirmation determiner capable of making a random confirmation decision in terms of whether a random confirmation is to be performed; and

a confirmation determiner capable of making one of a deterministic confirmation decision and an integrated confirmation decision.

50. The system according to claim 49, wherein the random confirmation determiner comprises:

a random number generator capable of generating a random number;

a random confirmation decision mechanism in communication with the random number generator, capable of determining whether a confirmation is to be performed based on the random number generated.

51. The system according to claim 49, wherein the confirmation determiner comprises:

a deterministic confirmation decision mechanism capable of making a deterministic confirmation decision based on a pre-defined deterministic confirmation range with respect to a confidence score associated with a speech recognition result; and

a confirmation decision integrator in communication with the random confirmation decision mechanism and the deterministic confirmation decision mechanism, capable of making a confirmation decision based on the random confirmation decision and the deterministic conformation decision.

52. The system according to claim 47, wherein the self-monitoring and adaptation mechanism is further capable of adapting the dialogue system front end based on the collected confirmation results.

53. The system according to claim 52, wherein the self-monitoring and adaptation mechanism comprises:

a confirmation result logger capable of collecting conformation results;

a confirmation information analyzer capable of analyzing the collected confirmation results to generated relevant statistics.

54. The system according to claim 53, further comprising a performance monitoring mechanism capable of monitoring the performance based on the relevant statistics.

55. The system according to claim 53, further comprising an adaptive training mechanism capable of performing re-training of a speech recognition mechanism from which the speech recognition result is received according to the relevant statistics.

56. The system according to claim 53, further comprising a confirmation range adjuster capable of adjusting the deterministic conformation range according to the relevant statistics.

57. A call center, comprising:

a dialogue system capable of interfacing with a caller using speech to produce a speech recognition result based on the speech of the caller;

a call routing mechanism capable of routing the call from the caller based on the speech recognition result, wherein

the dialogue system is further capable of performing confirmations in a mode including a random confirmation mode and collecting the confirmation results generated during the confirmations.

58. The call center according to claim 57, wherein the dialogue system is further capable of monitoring the performance based on the collected confirmation results.

59. The call center according to claim 57, wherein the dialogue system is further capable of tuning the performance based on the collected confirmation results.

60. The call center according to claim 57, wherein the dialogue system comprises:

a dialogue system front end capable of producing the speech recognition result based on the speech of the caller;

a dialogue system tuning tool in communication with the dialogue system front end capable of monitoring and tuning the performance of the dialogue system front end based on confirmation results collected from confirmations performed in one of a deterministic confirmation mode, a random confirmation mode, and an integrated confirmation mode.

61. The system according to claim 60, wherein the dialogue system tuning tool comprises:

a random mode enabled confirmation mechanism capable of performing a confirmation on the speech recognition result in one of a deterministic confirmation mode, a random confirmation mode, and an integrated confirmation mode; and

a self-monitoring and adaptation mechanism capable of monitoring the performance of the dialogue system front end through the collected confirmation results.

62. The system according to claim 61, wherein the random mode enabled confirmation mechanism comprises:

a confirmation decision mechanism configured to make a confirmation decision; and

a confirmation mechanism in communication with the confirmation decision mechanism, capable of carrying out a confirmation on the speech recognition result from the dialogue system front end according to a confirmation decision made by the confirmation decision mechanism.

63. The system according to claim 62, wherein the confirmation decision mechanism comprises:

a random confirmation determiner capable of making a random confirmation decision in terms of whether a random confirmation is to be performed; and

a confirmation determiner capable of making one of a deterministic confirmation decision and an integrated confirmation decision.

64. The system according to claim 61, wherein the self-monitoring and adaptation mechanism comprises:

a confirmation result logger capable of collecting conformation results;

a confirmation information analyzer capable of analyzing the collected confirmation results to generated relevant statistics.

65. The system according to claim 64, further comprising a performance monitoring mechanism capable of monitoring the performance based on the relevant statistics.

66. The system according to claim 64, further comprising an adaptive training mechanism capable of performing re-training of a speech recognition mechanism from which the speech recognition result is received according to the relevant statistics.

67. An article comprising a storage medium having stored thereon instructions that, when executed by a machine, result in the following:

receiving an automatic speech recognition result with an associated score indicating a level of confidence with respect to the speech recognition result;

determining whether a confirmation is to be performed on the speech recognition result;

performing a confirmation if it is determined that a confirmation is to be carried out, wherein the confirmation can be performed in a confirmation mode including a random confirmation mode.

68. The article according to claim 67, wherein said determining comprises:

examining the pre-configured confirmation mode;

determining, in the pre-configured confirmation mode, whether a confirmation is to be performed in one of a pre-configured deterministic confirmation mode, a random confirmation mode, and an integrated confirmation mode.

69. The article according to claim 68, wherein a deterministic confirmation in the deterministic confirmation mode is determined based on a deterministic confirmation decision made according to a pre-defined range for a confidence score associated with a speech recognition result, said making a deterministic confirmation decision comprises:

determining whether the confidence score associated with the speech recognition result is within a pre-defined range for a confidence score;

determining to perform a confirmation if the confidence score associated with the speech recognition result is within the pre-defined range for a confidence score.

70. The article according to claim 69, wherein a random confirmation in the random confirmation mode is determined based on a random confirmation decision made according to a pre-defined range for a random number, said making a random confirmation decision comprises:

generating a random number;

determining whether the generated random number is within the pre-defined range for a random number;

checking, if the generated random number falls within the pre-defined range of a random number, whether the generated random number satisfies a confirmation percentage parameter; and

determining to perform a random confirmation if the generated random number is within the pre-defined range and satisfies the confirmation percentage parameter.

71. The article according to claim 68, wherein an integrated confirmation in the integrated confirmation mode is determined based on an integrated confirmation decision according to both the deterministic confirmation decision and the random confirmation decision, said making the integrated confirmation comprises:

making a deterministic confirmation decision;

making a random confirmation decision; and

combining the deterministic confirmation decision and the random confirmation decision to produce the integrated confirmation decision on whether a confirmation is to be performed.

72. The article according to claim 67, wherein said performing a confirmation comprises:

carrying out a deterministic confirmation if the confirmation configuration corresponds to the deterministic confirmation mode and the deterministic confirmation decision is to perform a confirmation;

carrying out a random confirmation if the confirmation configuration corresponds to the random confirmation mode and the random confirmation decision is to perform a random confirmation; and

carrying out an integrated confirmation if the confirmation configuration corresponds to the integrated confirmation mode and the integrated confirmation decision is to perform a confirmation.

73. An article comprising a storage medium having stored thereon instructions that, when executed by a machine, result in the following:

receiving an automatic speech recognition result with an associated score indicating a level of confidence with respect to the speech recognition result;

determining whether a confirmation is to be performed on the speech recognition result, wherein the confirmation can be performed in a confirmation mode including a random confirmation mode;

performing confirmation if it is determined that a confirmation is to be carried out; and

collecting results from the confirmation after said performing.

74. The article according to claim 73, wherein said determining comprises:

examining the pre-configured confirmation mode;

determining, in the pre-configured confirmation mode, whether a confirmation is to be performed in one of a pre-configured deterministic confirmation mode, a random confirmation mode, and an integrated confirmation mode.

75. The article according to claim 74, wherein a deterministic confirmation in the deterministic confirmation mode is determined based on a deterministic confirmation decision made according to a pre-defined range for a confidence score associated with a speech recognition result, said making a deterministic confirmation decision comprises:

determining whether the confidence score associated with the speech recognition result is within the pre-defined range for a confidence score;

determining to perform a confirmation if the confidence score associated with the speech recognition result is within the pre-defined range for a confidence score.

76. The article according to claim 74, wherein a random confirmation in the random confirmation mode is determined based on a random confirmation decision made according to a pre-defined range for a random number, said making a random confirmation decision comprises:

generating a random number;

determining whether the generated random number is within the pre-defined range for a random number;

checking, if the generated random number falls within the pre-defined range of a random number, whether the generated random number satisfies a confirmation percentage parameter; and

determining to perform a random confirmation if the generated random number is within the pre-defined range and satisfies the confirmation percentage parameter.

77. The article according to claim 74, wherein an integrated confirmation in the integrated confirmation mode is determined based on an integrated confirmation decision according to both the deterministic confirmation decision and the random confirmation decision, said making the integrated confirmation comprises:

making a deterministic confirmation decision;

making a random confirmation decision; and

combining the deterministic confirmation decision and the random confirmation decision to produce the integrated confirmation decision on whether a confirmation is to be performed.

78. The article according to claim 73, wherein said performing a confirmation comprises:

carrying out a deterministic confirmation if the confirmation configuration corresponds to the deterministic confirmation mode and the deterministic confirmation decision is to perform a confirmation;

carrying out a random confirmation if the confirmation configuration corresponds to the random confirmation mode and the random confirmation decision is to perform a random confirmation; and

carrying out an integrated confirmation if the confirmation configuration corresponds to the integrated confirmation mode and the integrated confirmation decision is to perform a confirmation.

79. The article according to claim 73, wherein a confirmation result collected after said performing indicates one of:

the speech recognition result is correct; and

the speech recognition result is incorrect.

80. The article according to claim 73, the instructions, when executed by a machine, further result in:

analyzing confirmation results collected to generate relevant statistics;

utilizing the generated relevant statistics.

81. The article according to claim 80, wherein the relevant statistics include at least one of:

accuracy statistics indicating the performance of a speech recognizer that produces the speech recognition result; and

performance statistics indicating the performance of the speech recognizer in different ranges of the confidence score.

82. The article according to claim 80, the instructions, when executed by a machine, further result in collecting training data based on the confirmation results and the relevant statistics, wherein the training data includes at least one of:

the speech data based on which the speech recognition result is obtained;

the confidence score associated with the speech recognition result; and

the confirmation result associated with the speech recognition result.

83. The article according to claim 82, wherein said utilizing includes at least one of:

monitoring the performance of the speech recognizer based on the confirmation results and the relevant statistics;

adapting the recognition performance of the speech recognizer by conducting re-training using at least a portion of the collected training data selected according to the relevant statistics;

adjusting the speech recognizer in terms of how a confidence score is computed based on the relevant statistics; and

adjusting a pre-defined range for a confidence score, which is used to determine whether a confirmation is to be performed.