Automatic Updating of Confidence Scoring Functionality for Speech Recognition Systems

• US 20130275135A1
• Filed: 01/07/2011
• Published: 10/17/2013
• Est. Priority Date: 01/07/2011
• Status: Active Grant

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Morales, Nicolas, Connolly, Dermot, Halberstadt, Andrew
• Granted Patent: US 9,330,665 B2
• Time in Patent Office: Days
• US Class Current: 704/236
• CPC Class Codes: G10L 15/01 Assessment or evaluation of speech recognition systems
• CPC Class Codes: G10L 15/065 Adaptation
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/0635 updating or merging of old and new templates; Mean values; Weighting
• CPC Class Codes: G10L 2015/0636 Threshold criteria for the updating

Abstract

Automatically adjusting confidence scoring functionality is described for a speech recognition engine. Operation of the speech recognition system is revised so as to change an associated receiver operating characteristic (ROC) curve describing performance of the speech recognition system with respect to rates of false acceptance (FA) versus correct acceptance (CA). Then a confidence scoring functionality related to recognition reliability for a given input utterance is automatically adjusted such that where the ROC curve is better for a given operating point after revising the operation of the speech recognition system, the adjusting reflects a double gain constraint to maintain FA and CA rates at least as good as before revising operation of the speech recognition system.

Claims

1. A method for automatically adjusting operation of a speech recognition system comprising:

revising operation of the speech recognition system so as to change an associated receiver operating characteristic (ROC) curve describing performance of the speech recognition system with respect to rates of false acceptance (FA) versus correct acceptance (CA); and

automatically adjusting a confidence scoring functionality related to recognition reliability for a given input utterance such that where the ROC curve is better for a given operating point after revising the operation of the speech recognition system, the adjusting reflects a double gain constraint to maintain FA and CA rates at least as good as before revising operation of the speech recognition system.

2. A method according to claim 1, wherein where the ROC curve is not better for a given operating point after revising the operation of the speech recognition system, the adjusting minimizes worsening of the FA and CA rates.

3. A method according to claim 1, wherein automatically adjusting the confidence scoring functionality includes establishing a mapping to maintain equivalence of the confidence scoring functionality before and after revising the operation of the speech recognition system.

4. A method according to claim 1, wherein the confidence scoring functionality includes using confidence score thresholds that define for a given set of circumstances whether to accept, reject, or confirm a given input utterance.

5. A method according to claim 4, wherein automatically adjusting the confidence scoring functionality includes updating the confidence score thresholds.

6. A method according to claim 5, wherein automatically adjusting the confidence scoring functionality is further based on a developer user input establishing a priority favoring FA or CA rate.

7. A method according to claim 5, wherein automatically adjusting the confidence scoring functionality is based on accepting and rejecting an equal percentage of input utterances as before revising the operation of the speech recognition system.

8. A method according to claim 1, wherein automatically adjusting the confidence scoring functionality is further based on a mapping to posterior probabilities.

9. A method according to claim 1, wherein automatically adjusting the confidence scoring functionality is further based on using actual application data with transcriptions.

10. A method according to claim 1, wherein automatically adjusting the confidence scoring functionality is further based on using actual application data without transcriptions.

11. A computer program product implemented in a computer readable storage medium for automatically adjusting operation of a speech recognition system, the program product comprising:

program code for revising operation of the speech recognition system so as to change an associated receiver operating characteristic (ROC) curve describing performance of the speech recognition system with respect to rates of false acceptance (FA) versus correct acceptance (CA); and

program code for automatically adjusting a confidence scoring functionality related to recognition reliability for a given input utterance, such that where the ROC curve is better for a given operating point after revising the operation of the speech recognition system, the adjusting reflects a double gain constraint to maintain FA and CA rates at least as good as before revising operation of the speech recognition system.

12. A product according to claim 11, wherein where the ROC curve is not better for a given operating point after revising the operation of the speech recognition system, the program code for adjusting minimizes worsening of the FA and CA rates.

13. A product according to claim 11, wherein the program code for automatically adjusting the confidence scoring functionality includes program code for establishing a mapping to maintain equivalence of the confidence scoring functionality before and after revising the operation of the speech recognition system.

14. A product according to claim 11, wherein the confidence scoring functionality includes using confidence score thresholds that define for a given set of circumstances whether to accept, reject, or confirm a given input utterance.

15. A product according to claim 14, wherein the program code for automatically adjusting the confidence scoring functionality includes program code for updating the confidence score thresholds.

16. A product according to claim 15, wherein the program code for automatically adjusting the confidence scoring functionality is further based on a developer user input establishing a priority favoring FA or CA rate.

17. A product according to claim 15, wherein the program code for automatically adjusting the confidence scoring functionality is based on accepting and rejecting an equal percentage of input utterances as before revising the operation of the speech recognition system.

18. A product according to claim 11, wherein the program code for automatically adjusting the confidence scoring functionality is further based on a mapping to posterior probabilities.

19. A product according to claim 11, wherein the program code for automatically adjusting the confidence scoring functionality is further based on using actual application data with transcriptions.

20. A product according to claim 11, wherein the program code for automatically adjusting the confidence scoring functionality is further based on using actual application data without transcriptions.

21. A speech recognition system implemented as a plurality of computer processes functioning in a computer processor, the speech recognition system comprising:

means for revising operation of the speech recognition system so as to change an associated receiver operating characteristic (ROC) curve describing performance of the speech recognition system with respect to rates of false acceptance (FA) versus correct acceptance (CA); and

means for automatically adjusting a confidence scoring functionality related to recognition reliability for a given input utterance, such that where the ROC curve is better for a given operating point after revising the operation of the speech recognition system, the adjusting reflects a double gain constraint to maintain FA and CA rates at least as good as before revising operation of the speech recognition system.

22. A system according to claim 21, wherein where the ROC curve is not better for a given operating point after revising the operation of the speech recognition system, the means for adjusting minimizes worsening of the FA and CA rates.

23. A system according to claim 21, wherein the means for automatically adjusting the confidence scoring functionality includes means for establishing a mapping to maintain equivalence of the confidence scoring functionality before and after revising the operation of the speech recognition system.

24. A system according to claim 21, wherein the confidence scoring functionality includes using confidence score thresholds that define for a given set of circumstances whether to accept, reject, or confirm a given input utterance.

25. A system according to claim 24, wherein the means for automatically adjusting the confidence scoring functionality includes means for updating the confidence score thresholds.

26. A system according to claim 25, wherein the means for automatically adjusting the confidence scoring functionality is further based on a developer user input establishing a priority favoring FA or CA rate.

27. A system according to claim 25, wherein the means for automatically adjusting the confidence scoring functionality is based on accepting and rejecting an equal percentage of input utterances as before revising the operation of the speech recognition system.

28. A system according to claim 21, wherein the means for automatically adjusting the confidence scoring functionality is further based on a mapping to posterior probabilities.

29. A system according to claim 21, wherein the means for automatically adjusting the confidence scoring functionality is further based on using actual application data with transcriptions.

30. A system according to claim 21, wherein the means for automatically adjusting the confidence scoring functionality is further based on using actual application data without transcriptions.