Methods and systems of routing utterances based on confidence estimates

• US 7,003,456 B2
• Filed: 06/12/2001
• Issued: 02/21/2006
• Est. Priority Date: 06/12/2000
• Status: Expired due to Term

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: ScanSoft, Inc. n/k/a Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Roth, Robert, Gillick, Laurence S., Rao, Ashwin, Manganaro, Linda, Petty, David C., Peskin, Barbara R.
• Primary Examiner(s): Chawan, Vijay B.
• Application Number:
  US09/878,173
• Publication Number:
  US 20020133341A1
• Time in Patent Office: 1,715 Days
• Field of Search: 204/235, 204/246, 204/239, 204/251, 204/256, 204/270, 204/252, 204/240
• US Class Current: 704/235
• CPC Class Codes: G10L 15/10 using distance or distortion measures between unknown speech and reference templates
• CPC Class Codes: G10L 15/32 Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems

Abstract

A computer-based method of routing a message to a system includes receiving a message, and processing the message using large-vocabulary continuous speech recognition to generate a string of text corresponding to the message. The method includes generating a confidence estimate of the string of text corresponding to the message and comparing the confidence estimate to a predetermined threshold. If the confidence estimate satisfies the predetermined threshold, the string of text is forwarded to the system. If the confidence estimate does not satisfy the predetermined threshold, the information relating to the message is forwarded to a transcriptionist. The message may include one or more utterances. Each utterance in the message may be separately or jointly processed. In this way, a confidence estimate may be generated and evaluated for each utterance or for the whole message. Information relating to each utterance may be separately or jointly forwarded based on the results of the generation and evaluation.

Claims

1. A computer-based method of routing an utterance to a system, the method comprising:

receiving an utterance including two or more words;

processing the utterance using large-vocabulary continuous speech recognition to generate a string of text corresponding to the utterance;

generating a confidence estimate of the string of text corresponding to the utterance, the generating including;

selecting one or more predictors relating to the large-vocabulary continuous speech recognition, and

training a confidence model using the one or more predictors;

comparing the confidence estimate to a predetermined threshold;

if the confidence estimate satisfies the predetermined threshold, forwarding the string of text to the system; and

if the confidence estimate does not satisfy the predetermined threshold, forwarding information relating to the utterance to a transcriptionist.

2. The method of claim 1 further comprising if the confidence estimate does not satisfy the predetermined threshold, having the transcriptionist determine an acceptable string of text and forwarding the acceptable string of text to the system.

3. The method of claim 1 in which the confidence estimate indicates a probability that the string of text is an acceptable representation of the utterance.

4. The method of claim 1 in which the information relating to the utterance comprises one or more of:

the utterance, the string of text corresponding to the utterance, and the generated confidence estimate of the string of text.

5. The method of claim 1 in which generating the confidence estimate comprises:

extracting values of the one or more predictors based on the received utterance; and

providing the extracted values to the confidence model to generate the confidence estimate.

6. The method of claim 1 further comprising:

comparing the confidence estimate to a second predetermined threshold; and

if the confidence estimate does not satisfy the first predetermined threshold and does satisfy the second predetermined threshold level;

forwarding the information relating to the utterance to the user who spoke the utterance, and

permitting the user to act in response to the forwarded information.

7. The method of claim 1 in which the transcriptionist is a human transcriptionist.

8. The method of claim 1 in which the system includes a human recipient.

9. A computer-based method of routing a message to a system, the method comprising:

receiving a message including utterances;

processing each utterance in the message using large-vocabulary continuous speech recognition to generate a string of text corresponding to that utterance;

generating a confidence estimate for each string of text that corresponds to an utterance, the generating includ ing;

selecting one or more predictors for the string of text based on the large-vocabulary continuous speech recognition; and

training a confidence model for the string of text using the one or more predictors;

comparing each confidence estimate to a predetermined threshold;

if all of the confidence estimates satisfy the predetermined threshold, forwarding the string of text to the system; and

if any of the confidence estimates does not satisfy the predetermined threshold level, forwarding the message to the transcriptionist.

10. The method of claim 9 in which a confidence estimate for a string of text indicates a probability that the string of text is an acceptable representation of the corresponding utterance.

11. The method of claim 9 further comprising, if any one of the confidence estimates does not satisfy the predetermined threshold, having the transcriptionist determine acceptable strings of text for the message and forwarding the acceptable strings of text for the message to the system.

12. The method of claim 9 in which generating the confidence estimate for a string of text comprises:

extracting values of the one or more predictors for the string of text based on the corresponding utterance; and

providing the extracted values for the utterance to the confidence model to generate the confidence estimate for the string of text.

13. The method of claim 9 further comprising:

comparing each confidence estimate to a second predetermined threshold; and

if a confidence estimate does not satisfy the first predetermined threshold level and satisfies the second predetermined threshold level;

forwarding information relating to the message to the user who spoke the message, and

permitting the user to act in response to the forwarded information.

14. The method of claim 13 in which the information relating to the message comprises one or more of:

the message, the string of text for the confidence estimate, and the confidence estimate for the string of text.

15. The method of claim 9 in which the transcriptionist is a human transcriptionist.

16. The method of claim 9 in which the system includes a human recipient.

17. The method of claim 9 in which at least one utterance in the message comprises two or more words.

18. A computer-based method of routing a message to a system, the method comprising:

receiving a message including utterances;

processing each utterance in the message using large-vocabulary continuous speech recognition to generate a string of text corresponding to that utterance;

generating a confidence estimate for each string of text that corresponds to an utterance, the generating including;

selecting one or more predictors for the string of text based on the large-vocabulary continuous speech recognition; and

training a confidence model for the string of text using the one or more predictors;

comparing each confidence estimate to a predetermined threshold;

if all of the confidence estimates satisfy the predetermined threshold, forwarding the strings of text to the system; and

if one of the confidence estimates does not satisfy the predetermined threshold, forwarding information relating to the utterance corresponding to that confidence estimate to a transcriptionist.

19. The method of claim 18 in which a confidence estimate for a string of text indicates a probability that the string of text is an acceptable representation of the corresponding utterance.

20. The method of claim 18 further comprising, if one of the confidence estimates does not satisfy the predetermined threshold, after the transcriptionist determines an acceptable string of text for the utterance, forwarding the acceptable string of text for the message to the system.

21. The method of claim 18 in which generating the confidence estimate for a string of text comprises:

extracting values of the one or more predictors for the string of text based on the corresponding utterance; and

providing the extracted values for the utterance to the confidence model to generate the confidence estimate for the string of text.

22. The method of claim 18 in which the information relating to the utterance corresponding to the confidence estimate comprises one or more of:

the message, the string of text for the confidence estimate, and the confidence estimate.

23. The method of claim 18 further comprising:

comparing each confidence estimate to a second predetermined threshold; and

if any one of the confidence estimates does not satisfy the first predetermined threshold level and satisfies the second predetermined threshold level;

forwarding the information relating to the utterance that generated that confidence estimate to the user who spoke the message, and

permitting the user to act in response to the forwarded information.

24. The method of claim 18 in which the transcriptionist is a human transcriptionist.

25. The method of claim 18 in which the system includes a human recipient.

26. The method of claim 18 in which at least one utterance in the message comprises two or more words.

27. A computer-based method of routing a message to a system, the method comprising:

receiving a message including utterances;

processing each utterance in the message using large-vocabulary continuous speech recognition to generate a string of text for each utterance in the message;

generating a confidence estimate for each string of text that corresponds to an utterance, the generating including;

selecting one or more predictors for the string of text based on the large-vocabulary continuous speech recognition; and

training a confidence model for the string of text using the one or more predictors;

calculating a message confidence estimate based on the confidence estimates for the strings of text generated for the message;

comparing the message confidence estimate to a predetermined threshold;

if the message confidence estimate satisfies the predetermined threshold, forwarding the strings of text generated for the message to the system; and

if the message confidence estimate does not satisfy the predetermined threshold, forwarding information relating to the message to a transcriptionist.

28. The method of claim 27 in which a confidence estimate for a string of text indicates a probability that the string of text is an acceptable representation of the corresponding utterance.

29. The method of claim 27 further comprising, if the message confidence estimate does not satisfy the predetermined threshold, having the transcriptionist determine one or more acceptable strings of text and for the utterances of the message, forwarding the acceptable strings of text for the message to the system.

30. The method of claim 27 in which information relating to the message comprises one or more of:

the message, the string of text for each utterance in the message, and the message confidence estimate.

31. The method of claim 27 in which generating the confidence estimate for a string of text comprises:

extracting values of the one or more predictors for the string of text based on the corresponding utterance; and

providing the extracted values for the utterance to the confidence model to generate the confidence estimate for the string of text.

32. The method of claim 27 further comprising:

comparing the message confidence estimate to a second predetermined threshold; and

if the message confidence estimate does not satisfy the first predetermined threshold and it satisfies the second predetermined threshold;

forwarding the information relating to the message to the user who spoke the message, and

permitting the user to act in response to the forwarded information.

33. The method of claim 27 in which the transcriptionist is a human transcriptionist.

34. The method of claim 27 in which the system includes a human recipient.

35. The method of claim 27 in which at least one utterance in the message comprises two or more words.