Error prediction in spoken dialog systems

• US 20060149544A1
• Filed: 01/05/2005
• Published: 07/06/2006
• Est. Priority Date: 01/05/2005
• Status: Abandoned Application

• Current Assignee: AT&T Corporation (AT&T, Inc.)
• Original Assignee: AT&T Corporation (AT&T, Inc.)
• Inventors: Riccardi, Giuseppe, Hakkani-Tur, Dilek Z., Tur, Gokhan
• Application Number:
  US11/029,278
• Publication Number:
  US 20060149544A1
• Time in Patent Office: Days
• US Class Current: 704/236
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

A spoken dialog system configured to use a combined confidence score. A first confidence score, indicating a confidence level in a speech recognition result of recognizing an utterance, is provided. A second confidence level, indicating a confidence level of mapping the speech recognition result to an intent, is provided. The first confidence score and the second confidence score are combined to form a combined confidence score. A determination is made, with respect to whether to accept the intent, based on the combined confidence score.

Claims

1. A method in a spoken dialog system, the method comprising:

providing a first confidence score indicating a confidence level in a speech recognition result of recognizing an utterance;

providing a second confidence score indicating a confidence level of mapping the speech recognition result to an intent;

combining the first confidence score with the second confidence score to form a combined confidence score; and

determining whether to accept the intent based on the combined confidence score.

2. The method of claim 1, wherein the determining whether to accept the intent based on the combined confidence score further comprises:

comparing the combined confidence score to a first threshold; and

accepting the intent when the combined confidence score is greater than the first threshold.

3. The method of claim 1, wherein the determining whether to accept the intent based on the combined confidence score further comprises:

comparing the combined confidence score to a second threshold; and

rejecting the intent when the combined confidence score is less than or equal to the second threshold.

4. The method of claim 3, wherein the determining whether to accept the intent based on the combined confidence score further comprises:

re-prompting a user when the combined confidence score is less than or equal to the second threshold.

5. The method of claim 1, wherein the determining whether to accept the intent based on the combined confidence score further comprises:

comparing the combined confidence score to a first threshold and a second threshold; and

asking a user to confirm a hypothesized utterance from the speech recognition result and an estimated intent from the mapping the speech recognition result to an intent when the second threshold is less than the combined confidence score which is less than or equal to the first threshold.

6. The method of claim 1, wherein the combining the first confidence score with the second confidence score to form a combined confidence score further comprises:

computing the combined confidence score according to;

ns=e(ŵ

)<sup>α

1</sup>×

e(ĉ

)<sup>α

2</sup>, where ns is the combined confidence level, e(ŵ

)is the first confidence score, e(ĉ

) is the second confidence score, and α

₁ and α

₂ are scaling factors.

7. The method of claim 1, wherein the combining the first confidence score with the second confidence score to form a combined confidence score further comprises:

using a linear regression technique to compute the combined score.

8. The method of claim 1, further comprising:

providing a word length of a hypothesized utterance from the speech recognition result, wherein;

the combining the first confidence score with the second confidence score to form a combined confidence score further comprises;

computing the combined score according to;

ns=β

₁+β

₂×

e(ŵ

)+β

₃×

e(ĉ

)+β

₄×

|ŵ

|, where ns is the combined confidence score, β

₁, β

₂, β

₃, and β

₄ are regression parameters, e(ŵ

) is the first confidence score, e(ĉ

) is the second confidence score, and |ŵ

| is the word length of the hypothesized utterance.

9. The method of claim 1, wherein the combining the first confidence score with the second confidence score to form a combined confidence score further comprises:

using a logistic regression technique to compute the combined score.

10. The method of claim 1, further comprising:

providing a word length of a hypothesized utterance from the speech recognition result, wherein;

the combining the first confidence score with the second confidence score to form a combined confidence score further comprises;

computing the combined confidence score according to;

$\mathrm{ns}=\frac{1}{1+{\mathrm{e<br><br>}}^{{\mathrm{\gamma <br><br>}}_1+{\mathrm{\gamma <br><br>}}_2*e<br><br>\left(\hat{w}\right)+{\mathrm{\gamma <br><br>}}_3*e<br><br>\left(\hat{c}\right)+{\mathrm{\gamma <br><br>}}_4*<br><br>\hat{w}<br><br>}},$

where ns is the combined confidence level, γ

₁, γ

₂, γ

₃, and γ

₄ are regression parameters, e(ŵ

) is the first confidence score, e(ĉ

) is the second confidence score, and |ŵ

| is the word length of the hypothesized utterance.

11. The method of claim 1, wherein the combining the first confidence score with the second confidence score to form a combined confidence score further comprises:

using a decision tree technique to determine the combined confidence score.

12. The method of claim 11, wherein the using a decision tree technique further comprises:

following a set of rules to sort down a tree from a root to a leaf node; and

computing the combined confidence score based on a probability that the intent of the utterance is correctly classified at the leaf node.

13. A spoken dialog system comprising:

a first component configured to provide a first confidence score indicating a confidence level in a speech recognition result of recognizing an utterance;

a second component configured to provide a second confidence score indicating a confidence level of mapping the speech recognition result to an intent;

a third component configured to combine the first confidence score with the second confidence score to form a combined confidence score; and

a fourth component configured to determine whether to accept the intent based on the combined confidence score.

14. The spoken dialog system of claim 13, wherein the third component is included in one of the second component or the fourth component.

15. The spoken dialog system of claim 13, wherein the fourth component is further configured to:

compare the combined confidence score to a first threshold; and

accept the intent when the combined confidence score is greater than the first threshold.

16. The spoken dialog system of claim 13, wherein the fourth component is further configured to:

compare the combined confidence score to a second threshold; and

reject the intent when the combined confidence score is less than or equal to the second threshold.

17. The spoken dialog system of claim 16, wherein the fourth component is further configured to:

re-prompt a user when the combined confidence score is less than or equal to the second threshold.

18. The spoken dialog system of claim 13, wherein the fourth component is further configured to:

compare the combined confidence score to a first threshold and a second threshold; and

ask a user to confirm a hypothesized utterance from the speech recognition result and an estimated intent from the mapping the speech recognition result to an intent when the second threshold is less than the combined confidence score which is less than or equal to the first threshold.

19. The spoken dialog system of claim 13, wherein the third component is further configured to:

compute the combined confidence score according to;

ns=e(ŵ

)<sup>α

1</sup>×

e(ĉ

)<sup>α

2</sup>, where ns is the combined confidence level, e(ŵ

) is the first confidence score, e(ĉ

) is the second confidence score, and α

₁ and α

₂ are scaling factors.

20. The spoken dialog system of claim 13, wherein:

the first component is further configured to provide a word length of a hypothesized utterance from the speech recognition result, and

the third component is further configured to compute the combined confidence score according to;

ns=β

₁+β

₂×

e(ŵ

)+β

₃×

e(ĉ

)+β

₄×

|ŵ

|, where ns is the combined confidence score, β

₁, β

₂, β

₃, and β

₄ are regression parameters, e(ŵ

) is the first confidence score, e(ĉ

) is the second confidence score, and |ŵ

| is the word length of the hypothesized utterance.

21. The spoken dialog system of claim 13, wherein

the first component is further configured to provide a word length of a hypothesized utterance from the speech recognition result, and

the third component is further configured to compute the combined confidence score according to:

$\mathrm{ns}=\frac{1}{1+{\mathrm{e<br><br>}}^{{\mathrm{\gamma <br><br>}}_1+{\mathrm{\gamma <br><br>}}_2⨯<br><br>e<br><br>\left(\hat{w}\right)+{\mathrm{\gamma <br><br>}}_3⨯<br><br>e<br><br>\left(\hat{c}\right)+{\mathrm{\gamma <br><br>}}_4⨯<br><br><br><br>\hat{w}<br><br>}},$

where ns is the combined confidence score, γ

₁, γ

₂, γ

₃, and γ

₄ are regression parameters, e(ŵ

) is the first confidence score, e(ĉ

) is the second confidence score, and |ŵ

| is the word length of the hypothesized utterance.

22. The spoken dialog system of claim 13, wherein the third component is further configured to:

use a decision tree technique to determine the combined confidence score.

23. A machine-readable medium having a plurality of instructions included therein, the plurality of instructions comprising:

instructions for providing a first confidence score indicating a confidence level in a speech recognition result of recognizing an utterance;

instructions for providing a second confidence score indicating a confidence level of mapping the speech recognition result to an intent;

instructions for combining the first confidence score with the second confidence score to form a combined confidence score; and

instructions for determining whether to accept the intent based on the combined confidence score.

24. The machine-readable medium of claim 23, further comprising:

instructions for comparing the combined confidence score to a first threshold; and

instructions for accepting the intent when the combined confidence score is greater than the first threshold.

25. The machine-readable medium of claim 23, further comprising:

instructions for comparing the combined confidence score to a second threshold; and

instructions for rejecting the intent when the combined confidence score is less than or equal to the second threshold.

26. The machine-readable medium of claim 25, further comprising:

instructions for re-prompting a user when the combined confidence score is less than or equal to the second threshold.

27. The machine-readable medium of claim 23, further comprising:

instructions for comparing the combined confidence score to a first threshold and a second threshold; and

instructions for asking a user to confirm a hypothesized utterance from the speech recognition result and an estimated intent from the mapping the speech recognition result to an intent when the second threshold is less than the combined confidence score which is less than or equal to the first threshold.

28. The machine-readable medium of claim 23, further comprising:

instructions for computing the combined confidence score according to;

ns=e(ŵ

)<sup>α

1</sup>×

e(ĉ

)<sup>α

2</sup>, where ns is the combined confidence level, e(ŵ

) is the first confidence score, e(ĉ

) is the second confidence score, and α

₁ and α

₂ are scaling factors.

29. The machine-readable medium of claim 23, further comprising:

instructions for providing a word length of a hypothesized utterance from the speech recognition result, and

instructions for computing the combined confidence score according to;

ns=β

₁+β

₂×

e(ŵ

)+β

₃×

e(ĉ

)+β

₄×

|ŵ

|, where ns is the combined confidence score, β

₁, β

₂, β

₃, and β

₄ are regression parameters, e(ŵ

) is the first confidence score, e(ĉ

) is the second confidence score, and |ŵ

| is the word length of the hypothesized utterance.

30. The machine-readable medium of claim 23, further comprising:

instructions for providing a word length of a hypothesized utterance from the speech recognition result, and

instructions for computing the combined confidence score according to;

$\mathrm{ns}=\frac{1}{1+{\mathrm{e<br><br>}}^{{\mathrm{\gamma <br><br>}}_1+{\mathrm{\gamma <br><br>}}_2⨯<br><br>e<br><br>\left(\hat{w}\right)+{\mathrm{\gamma <br><br>}}_3⨯<br><br>e<br><br>\left(\hat{c}\right)+{\mathrm{\gamma <br><br>}}_4⨯<br><br><br><br>\hat{w}<br><br>}},$

where ns is the combined confidence score, γ

₁, γ

₂, γ

₃, and γ

₄ are regression parameters, e(ŵ

) is the first confidence score, e(ĉ

) is the second confidence score, and |ŵ

| is the word length of the hypothesized utterance.

31. The machine-readable medium of claim 23, further comprising:

instructions for using a decision tree technique to determine the combined score.

32. An apparatus comprising:

means for providing a first confidence score indicating a confidence level in a speech recognition result of recognizing an utterance;

means for providing a second confidence score indicating a confidence level of mapping a speech recognition result to an intent;

means for combining a first confidence score with a second confidence score to form a combined confidence score; and

means for determining whether to accept an intent based on a combined confidence score.