CONFIDENCE CALIBRATION IN AUTOMATIC SPEECH RECOGNITION SYSTEMS

• US 20110144986A1
• Filed: 12/10/2009
• Published: 06/16/2011
• Est. Priority Date: 12/10/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Deng, Li, Li, Jinyu, Yu, Dong
• Granted Patent: US 9,070,360 B2
• Time in Patent Office: Days
• US Class Current: 704/232
• CPC Class Codes: G10L 15/01 Assessment or evaluation of speech recognition systems

Abstract

Described is a calibration model for use in a speech recognition system. The calibration model adjusts the confidence scores output by a speech recognition engine to thereby provide an improved calibrated confidence score for use by an application. The calibration model is one that has been trained for a specific usage scenario, e.g., for that application, based upon a calibration training set obtained from a previous similar/corresponding usage scenario or scenarios. Different calibration models may be used with different usage scenarios, e.g., during different conditions. The calibration model may comprise a maximum entropy classifier with distribution constraints, trained with continuous raw confidence scores and multi-valued word tokens, and/or other distributions and extracted features.

Claims

1. A system comprising, a calibration model, the calibration model configured to receive a confidence score and associated word from a speech recognition engine, and adjust the confidence score to provide a calibrated confidence score for use by an application, the calibration model having been trained for a usage scenario based upon a calibration training set obtained from at least one previous corresponding usage scenario.

2. The system of claim 1 wherein the usage scenario corresponds to the application, grammar, or semantic slot, or any combination of the application, grammar, or semantic slot.

3. The system of claim 1 wherein the usage scenario corresponds to a current condition, and further comprising, another calibration model that is used when a different condition exists.

4. The system of claim 1 wherein the calibration model comprises a maximum entropy classifier with distribution constraints.

5. The system of claim 4 wherein the maximum entropy classifier with distribution constraints uses continuous raw confidence scores and multi-valued word tokens.

6. The system of claim 1 wherein the calibration model adjusts the confidence score based upon training with features, including word token distribution related features and word-score information related features obtained from the previous corresponding usage scenario.

7. The system of claim 1 wherein the calibration model uses raw word confidence scores and raw semantic confidence scores to provide improved semantic confidence scores.

8. The system of claim 1 wherein the calibration model calibrates raw word confidence scores into improved word confidence scores, and uses the improved word confidence scores and raw semantic confidence scores to provide improved semantic confidence scores.

9. The system of claim 1 wherein the calibration model uses semantic data to convert at least some word labels in the calibration training set to converted word labels, wherein the converted word labels and unconverted word labels provide a set of updated word labels, and wherein the calibration model uses the updated word labels to calibrate raw word confidence scores into improved word confidence scores, and uses the improved word confidence scores and raw semantic confidence scores to provide improved semantic confidence scores.

10. The system of claim 1 wherein the calibration model is trained with features extracted from the calibration training set.

11. The system of claim 10 wherein at least one of the features is based upon sub-word unit distribution.

12. The system of claim 10 wherein at least one of the features is based upon internal information of the speech recognition engine.

13. The system of claim 10 wherein at least one of the features is based upon keyword coverage information.

14. In a computing environment, a method performed on at least one processor, comprising, training a calibration model for use in adjusting confidence scores output by a speech recognizer, including processing a calibration training set containing words, confidence scores, and labels indicating whether each word was correctly recognized, extracting features from the calibration training set including word and score distribution features, and using the features to train the calibration model.

15. The method of claim 14 further comprising, using the calibration model to adjust the confidence scores output by a speech recognizer in a usage scenario that corresponds to a usage scenario in which the words and confidence scores in the calibration training set were obtained.

16. The method of claim 14 wherein training the calibration model comprises:

a) modeling a score and a word separately for each frame, including using a label associated with the word to provide a positive score feature, a negative score feature, a positive word feature and a negative word feature;

or

b) modeling a score and a word jointly for each frame, including using a label associated with the word to provide a positive feature and a negative feature.

17. The method of claim 16 further comprising, using context information to construct features for previous and next frames.

18. The method of claim 14 wherein training the calibration model comprises using features that provide independent weights for the confidence scores and independent bias weights for different words.

19. The method of claim 14 wherein training the calibration model comprises:

a) using raw word confidence scores and raw semantic confidence scores to provide improved semantic confidence scores, or

b) calibrating raw word confidence scores into improved word confidence scores, and using the improved word confidence scores and raw semantic confidence scores to provide improved semantic confidence scores;

or

c) using semantic information to convert at least some word labels in the calibration training set to converted word labels which, along with unconverted word labels, provide a set of updated word labels, and using the updated word labels to calibrate raw word confidence scores into improved word confidence scores, and using the improved word confidence scores and raw semantic confidence scores to provide improved semantic confidence scores.

20. One or more computer-readable media having computer-executable instructions, which when executed perform steps, comprising:

training a calibration model, including processing a calibration training set containing data obtained from one or more previous usage scenarios;

receiving a confidence score from a speech recognition engine at the calibration model, and

adjusting the confidence score to output a calibrated confidence score for a usage scenario that corresponds to the one or more previous usage scenarios.