Method for estimating a confidence measure for a speech recognition system

• US 6,735,562 B1
• Filed: 06/05/2000
• Issued: 05/11/2004
• Est. Priority Date: 06/05/2000
• Status: Active Grant

• Current Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Original Assignee: Motorola, Inc. (Motorola Solutions, Inc.)
• Inventors: Choi, Ho Chuen, Song, Jian Ming, Zhang, Yaxin
• Primary Examiner(s): Dorvil, Richemond
• Assistant Examiner(s): Storm, Donald L.
• Application Number:
  US09/588,163
• Time in Patent Office: 1,436 Days
• Field of Search: 704/240, 704/252, 704/236, 704/238, 704/239
• US Class Current: 704/240
• CPC Class Codes: G10L 15/01 Assessment or evaluation of speech recognition systems

Abstract

A method of estimating a confidence measure for a speech recognition system, involves comparing an input speech signal with a number of predetermined models of possible speech signals. Best scores indicating the degree of similarity between the input speech signal and each of the predetermined models are then used to determine a normalized variance, which is used as the Confidence Measure, in order to determine whether the input speech signal has been correctly recognized, the Confidence Measure is compared to a threshold value. The threshold value is weighted according to the Signal to Noise Ratio of the input speech signal and according to the number of predetermined models used.

Claims

1. A method of estimating a confidence measure for a speech recognition system, the method comprising the steps of:

receiving an input utterance;

comparing the input utterance with a plurality of predetermined models of possible utterances to provide a plurality of scores indicating a degree of similarity between the input utterance and the plurality of predetermining models;

determining a variance of a predetermined number of the plurality of scores; and

normalizing the variance to provide a confidence measure for a likely recognition result for the in put utterance, wherein the confidence measure (CM) is calculated by;

$\mathrm{CM}=\frac{1}{N}<br><br>\underset{i=1}{\overset{N}{\sum <br><br>}}<br><br>\text{\hspace{1em}<br><br>}<br><br>{\left(\frac{S_i-\mathrm{\mu <br><br>}}{\mathrm{\mu <br><br>}}\right)}^2$

where CM is the confidence measure, N is a predetermined number of N-Best scores, Si is the i-th best score, and μ

is the mean calculated by;

$\mathrm{\mu <br><br>}=\frac{1}{N}<br><br>\underset{i=1}{\overset{N}{\sum <br><br>}}<br><br>\text{\hspace{1em}<br><br>}<br><br>S_i.$

2. A method of determining whether an input utterance to a speech recognition system is correctly recognized by the system the method comprising the steps of:

determining a likely recognition result for an input utterance;

comparing the input utterance with a plurality of predetermined models of possible utterances to provide a plurality of scores indicating a degree of similarity between the input utterance and the plurality of predetermined models;

determining a variance of a predetermined number of the plurality of scores;

normalizing the variance to provide an estimated confidence measure for a likely recognition result for the input utterance;

determining a threshold comprising weighting the threshold depending on the noise level in an input signal containing the input utterance;

comparing the threshold with the confidence measure; and

accepting or rejecting the recognition result according to whether the confidence measure is above or below the threshold.

3. A method according to claim 2, wherein the threshold is weighted according to a signal to noise ratio of the input signal.

4. A method according to claim 2, wherein the weighting has a first value at low noise levels, a second value at high noise levels, and varies between the first and second values at intermediate noise levels.

5. A method according to claim 4, wherein the weighting has a value of 1 when the signal to noise ratio of the input signal is greater then approximately 15.

6. A method according to claim 4, wherein the weighting has a value of 0 when the signal to noise ratio of the input signal is smaller than approximately 8.

7. A method according to claim 4, wherein the weighting (W) is given by

8. A method according to claim 2, wherein the step of determining a threshold comprises weighting the threshold depending on the number of predetermined models that the input utterance is compared with.

9. A method according to claim 8, wherein the weighting (W) is given by $W^{\prime }$

=α

+β

×



-VS/γ

where the number of predetermined models (VS) is 2 or more.

10. A method according to claim 9, wherein

α

=0.6;

β

=1.08; and

γ

=10.0.