Confidence calibration in automatic speech recognition systems

• US 9,070,360 B2
• Filed: 12/10/2009
• Issued: 06/30/2015
• Est. Priority Date: 12/10/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Yu, Dong, Deng, Li, Li, Jinyu
• Primary Examiner(s): Lerner, Martin
• Application Number:
  US12/634,744
• Publication Number:
  US 20110144986A1
• Time in Patent Office: 2,028 Days
• Field of Search: 704/231, 704/235, 704/243, 704/244, 704/254, 704/255, 704/256.2, 704/257
• US Class Current: 1/1
• CPC Class Codes: G10L 15/01 Assessment or evaluation of speech recognition systems

Abstract

Described is a calibration model for use in a speech recognition system. The calibration model adjusts the confidence scores output by a speech recognition engine to thereby provide an improved calibrated confidence score for use by an application. The calibration model is one that has been trained for a specific usage scenario, e.g., for that application, based upon a calibration training set obtained from a previous similar/corresponding usage scenario or scenarios. Different calibration models may be used with different usage scenarios, e.g., during different conditions. The calibration model may comprise a maximum entropy classifier with distribution constraints, trained with continuous raw confidence scores and multi-valued word tokens, and/or other distributions and extracted features.

Claims

1. A system comprising:

one or more processors;

a memory coupled to the one or more processors;

a calibration model, dynamically selected by and implemented on the one or more processors based on a current condition, the calibration model having been trained for a usage scenario that corresponds to the current condition, the calibration model configured to receive a word confidence score and a semantic confidence score from a speech recognition engine, and configured to adjust the word confidence score to provide a calibrated word confidence score for use by an application, and further configured to adjust the semantic confidence score using the calibrated word confidence score to provide a calibrated semantic confidence score for use by the application, the calibration model having been trained for the usage scenario based upon a calibration training set obtained from at least one previous corresponding usage scenario.

2. The system of claim 1 wherein the usage scenario corresponds to at least one of the application, grammar, or semantic slot.

3. The system of claim 1 wherein the usage scenario corresponds to a current condition, and further comprising, another calibration model that is used when a different condition exists.

4. The system of claim 1 wherein the calibration model comprises a maximum entropy classifier with distribution constraints.

5. The system of claim 4 wherein the maximum entropy classifier with distribution constraints is trained using continuous raw confidence scores and multi-valued word tokens.

6. The system of claim 1 wherein the calibration model is configured to adjust the word confidence score based upon training with features, including word token distribution related features and word-score information related features obtained from the previous corresponding usage scenario.

7. The system of claim 1 wherein the calibration model uses raw word confidence scores and raw semantic confidence scores to provide improved semantic confidence scores.

8. The system of claim 1 wherein the calibration model is configured to calibrate raw word confidence scores into improved word confidence scores, and uses the improved word confidence scores and raw semantic confidence scores to provide improved semantic confidence scores.

9. The system of claim 1 wherein the calibration model uses semantic data to convert at least some word labels in the calibration training set to converted word labels, wherein the converted word labels and unconverted word labels provide a set of updated word labels, and wherein the calibration model uses the updated word labels to calibrate raw word confidence scores into improved word confidence scores, and uses the improved word confidence scores and raw semantic confidence scores to provide improved semantic confidence scores.

10. The system of claim 1 wherein the calibration model is trained with features extracted from the calibration training set.

11. The system of claim 10 wherein at least one of the features is based upon sub-word unit distribution.

12. The system of claim 10 wherein at least one of the features is based upon internal information of the speech recognition engine.

13. The system of claim 10 wherein at least one of the features is based upon keyword coverage information.

14. In a computing environment, a method comprising:

training a calibration model, implemented on one or more processors, for use in adjusting confidence scores output by a speech recognizer in a usage scenario, including processing a calibration training set corresponding to the usage scenario containing words, confidence scores, and labels indicating whether each word was correctly recognized, extracting features from the calibration training set corresponding to the usage scenario, the features including word and score distribution features, keyword coverage values, and at least one of sub-word units, semantics, or sentences, and using the features and continuous confidence scores to train the calibration model for the usage scenario, wherein the calibration model is dynamically selected based upon a current condition that corresponds to the usage scenario.

15. The method of claim 14 further comprising:

using the calibration model to adjust the confidence scores output by the speech recognizer in the usage scenario that corresponds to the usage scenario in which the words and confidence scores in the calibration training set were obtained.

16. The method of claim 14 wherein training the calibration model comprises:

a) modeling a score and a word separately for each frame, including using a label associated with the word to provide a positive score feature, a negative score feature, a positive word feature and a negative word feature;

or

b) modeling a score and a word jointly for each frame, including using a label associated with the word to provide a positive feature and a negative feature.

17. The method of claim 16 further comprising:

using context information to construct features for previous and next frames.

18. The method of claim 14 wherein training the calibration model comprises using features that provide independent weights for the confidence scores and independent bias weights for different words.

19. The method of claim 14 wherein training the calibration model comprises:

a) using raw word confidence scores and raw semantic confidence scores to provide improved semantic confidence scores, or

b) calibrating raw word confidence scores into improved word confidence scores, and using the improved word confidence scores and raw semantic confidence scores to provide improved semantic confidence scores;

or

c) using semantic information to convert at least some word labels in the calibration training set to converted word labels which, along with unconverted word labels, provide a set of updated word labels, and using the updated word labels to calibrate raw word confidence scores into improved word confidence scores, and using the improved word confidence scores and raw semantic confidence scores to provide improved semantic confidence scores.

20. One or more computer storage devices having computer-executable instructions, which in response to execution by a computer, cause the computer to perform steps comprising:

dynamically selecting a calibration model based on a current usage scenario, the calibration model having been trained using data obtained from one or more previous usage scenarios corresponding to the current usage scenario;

receiving a raw word confidence score and a raw semantic confidence score from a speech recognition engine at the calibration model;

adjusting the raw word confidence score using continuous confidence scores to output a calibrated word confidence score for the current usage scenario; and

adjusting the raw semantic confidence score using the calibrated word confidence score to output a calibrated semantic confidence score for the current usage scenario.