Automatic updating of confidence scoring functionality for speech recognition systems with respect to a receiver operating characteristic curve

• US 9,330,665 B2
• Filed: 01/07/2011
• Issued: 05/03/2016
• Est. Priority Date: 01/07/2011
• Status: Active Grant

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Morales, Nicolas, Connolly, Dermot, Halberstadt, Andrew
• Primary Examiner(s): Wozniak, James
• Application Number:
  US13/977,174
• Publication Number:
  US 20130275135A1
• Time in Patent Office: 1,943 Days
• Field of Search: 704/236, 704/244
• US Class Current: 1/1
• CPC Class Codes: G10L 15/01 Assessment or evaluation of speech recognition systems
• CPC Class Codes: G10L 15/065 Adaptation
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/0635 updating or merging of old and new templates; Mean values; Weighting
• CPC Class Codes: G10L 2015/0636 Threshold criteria for the updating

Abstract

Automatically adjusting confidence scoring functionality is described for a speech recognition engine. Operation of the speech recognition system is revised so as to change an associated receiver operating characteristic (ROC) curve describing performance of the speech recognition system with respect to rates of false acceptance (FA) versus correct acceptance (CA). Then a confidence scoring functionality related to recognition reliability for a given input utterance is automatically adjusted such that where the ROC curve is better for a given operating point after revising the operation of the speech recognition system, the adjusting reflects a double gain constraint to maintain FA and CA rates at least as good as before revising operation of the speech recognition system.

Claims

1. A method comprising:

determining an original receiver operating characteristic (ROC) curve describing performance of a speech recognition system with respect to an original rate of false acceptance (FA) of the speech recognition system versus an original rate of correct acceptance (CA) of the speech recognition system;

changing an algorithm used by the speech recognition system for sentence confidence scores, wherein changing the algorithm results in a new ROC curve with respect to a new rate of FA of the speech recognition system versus a new rate of CA of the speech recognition system;

receiving a user specification of relative importance of the new rate of FA versus the new rate of CA; and

based on the relative importance of the new rate of FA versus the new rate of CA, adjusting a confidence scoring functionality related to recognition reliability for a given input utterance,

wherein at or above a given operating point of the speech recognition system, the new ROC curve reflects a double gain constraint relative to the original ROC curve, such that the new rate of FA is equal to or less than the original rate of FA, and the new rate of CA is equal to or greater than the original rate of CA.

2. The method of claim 1, wherein below the given operating point of the speech recognition system, the new ROC curve minimizes worsening of the rate of FA and the rate of CA.

3. The method of claim 1, comprising:

one or more original settings of the confidence scoring functionality to corresponding one or more new settings of the confidence scoring functionality.

4. The method of claim 3, comprising:

estimating the mapping using representative labeled training data.

5. The method of claim 4, wherein estimating the mapping using representative labeled training data comprises estimating a first mapping function that maps confidence scores for the original ROC curve to first posterior probabilities, and estimating a second mapping function that maps confidence scores for the new ROC curve to second posterior probabilities.

6. The method of claim 5, comprising:

determining an equivalence relationship between the first mapping function and the second mapping function,

wherein adjusting the confidence scoring functionality comprises adjusting the confidence scoring functionality based on the equivalence relationship between the first mapping function and the second mapping function.

7. The method of claim 6, comprising:

determining a range of discrete integer confidence values for the equivalence relationship; and

performing linear interpolation to determine any missing values in the range of discrete integer confidence values for the equivalence relationship.

8. The method of claim 1, wherein the confidence scoring functionality includes confidence score thresholds that define for a given set of circumstances whether to accept, reject, or confirm a given input utterance.

9. The method of claim 8, wherein adjusting the confidence scoring functionality includes updating the confidence score thresholds.

10. The method of claim 9, wherein the user specification of relative importance of the new rate of FA versus the new rate of CA comprises input establishing a priority favoring the rate of FA or the rate of CA.

11. The method of claim 9, wherein the confidence score thresholds on the new ROC curve are determined based on accepting and rejecting an equal percentage of input utterances as the confidence score thresholds on the original ROC curve.

12. The method of claim 1, wherein receiving the user specification of relative importance of the new rate of FA versus the new rate of CA comprises receiving the user specification of relative importance via an adjustable slider of a graphical user interface of the speech recognition system, wherein a position of the adjustable slider corresponds to the relative importance of the new rate of FA versus the new rate of CA.

13. The method of claim 1,

wherein the relative importance of the new rate of CA is greater than the relative importance of the new rate of FA, and

wherein based on the relative importance of the new rate of CA being greater than the relative importance of the new rate of FA, adjusting the confidence scoring functionality related to the recognition reliability for the given input utterance comprises maximizing the new rate of CA.

14. The method of claim 1,

wherein the relative importance of the new rate of FA is greater than the relative importance of the new rate of CA, and

wherein based on the relative importance of the new rate of FA being greater than the relative importance of the new rate of CA, adjusting the confidence scoring functionality related to the recognition reliability for the given input utterance comprises minimizing the new rate of FA.

15. The method of claim 1, wherein a search space used by the speech recognition system for the given input utterance is reduced to a region that satisfies the double gain constraint.

16. The method of claim 1, comprising:

updating an acoustic model used by the speech recognition system, wherein the updated acoustic model is associated with the new ROC curve with respect to the new rate of FA of the speech recognition system versus the new rate of CA of the speech recognition system.

17. Non-transitory computer-readable media storing executable instructions that, when executed by one or more processors, cause a speech recognition system to:

determine an original receiver operating characteristic (ROC) curve describing performance of the speech recognition system with respect to an original rate of false acceptance (FA) of the speech recognition system versus an original rate of correct acceptance (CA) of the speech recognition system;

change an algorithm used by the speech recognition system for sentence confidence scores, wherein changing the algorithm results in a new ROC curve with respect to a new rate of FA of the speech recognition system versus a new rate of CA of the speech recognition system;

receive a user specification of relative importance of the new rate of FA versus the new rate of CA; and

based on the relative importance of the new rate of FA versus the new rate of CA, adjust a confidence scoring functionality related to recognition reliability for a given input utterance,

wherein at or above a given operating point of the speech recognition system, the new ROC curve reflects a double gain constraint relative to the original ROC curve, such that the new rate of FA is equal to or less than the original rate of FA, and the new rate of CA is equal to or greater than the original rate of CA.

18. The non-transitory computer-readable media of claim 17, wherein below the given operating point of the speech recognition system, the new ROC curve minimizes worsening of the rate of FA and the rate of CA.

19. The non-transitory computer-readable media of claim 17, wherein the executable instructions, when executed by the one or more processors, cause the speech recognition system to:

map one or more original settings of the confidence scoring functionality to corresponding one or more new settings of the confidence scoring functionality.

20. The non-transitory computer-readable media of claim 17, wherein the confidence scoring functionality includes confidence score thresholds that define for a given set of circumstances whether to accept, reject, or confirm a given input utterance.

21. The non-transitory computer-readable media of claim 20, wherein adjusting the confidence scoring functionality includes updating the confidence score thresholds.

22. The non-transitory computer-readable media of claim 21, wherein the user specification of relative importance of the new rate of FA versus the new rate of CA comprises input establishing a priority favoring the rate of FA or the rate of CA.

23. The non-transitory computer-readable media of claim 21, wherein the confidence score thresholds on the new ROC curve are determined based on accepting and rejecting an equal percentage of input utterances as the confidence score thresholds on the original ROC curve.

24. A system comprising:

one or more processors; and

non-transitory memory storing executable instructions that, when executed by the one or more processors, cause the system to;

determine an original receiver operating characteristic (ROC) curve describing performance of the system with respect to an original rate of false acceptance (FA) of the system versus an original rate of correct acceptance (CA) of the system;

change an algorithm used by the system for sentence confidence scores, wherein changing the algorithm results in a new ROC curve with respect to a new rate of FA of the system versus a new rate of CA of the system;

receive a user specification of relative importance of the new rate of FA versus the new rate of CA; and

based on the relative importance of the new rate of FA versus the new rate of CA, adjust a confidence scoring functionality related to recognition reliability for a given input utterance,

wherein at or above a given operating point of the system, the new ROC curve reflects a double gain constraint relative to the original ROC curve, such that the new rate of FA is equal to or less than the original rate of FA, and the new rate of CA is equal to or greater than the original rate of CA.

25. The system of claim 24, wherein below the given operating point of the system, the new ROC curve minimizes worsening of the rate of FA and the rate of CA.

26. The system of claim 24, wherein the executable instructions, when executed by the one or more processors, cause the system to:

map one or more original settings of the confidence scoring functionality to corresponding one or more new settings of the confidence scoring functionality.

27. The system of claim 24, wherein the confidence scoring functionality includes confidence score thresholds that define, for a given set of circumstances, whether to accept, reject, or confirm a given input utterance.

28. The system of claim 27, wherein adjusting the confidence scoring functionality includes updating the confidence score thresholds.

29. The system of claim 28, wherein the user specification of relative importance of the new rate of FA versus the new rate of CA comprises input establishing a priority favoring the rate of FA or the rate of CA.

30. The system of claim 28, wherein the confidence score thresholds on the new ROC curve are determined based on accepting and rejecting an equal percentage of input utterances as the confidence score thresholds on the original ROC curve.