Methods and systems for assessing and improving the performance of a speech recognition system

• US 7,949,533 B2
• Filed: 03/21/2007
• Issued: 05/24/2011
• Est. Priority Date: 02/04/2005
• Status: Active Grant

• Current Assignee: Vocollect, Inc. (Honeywell International Inc.)
• Original Assignee: Vococollect Incorporated
• Inventors: Pike, Jeffrey, Braho, Keith, El-Jaroudi, Amro, Laughery, Michael, Pike, Lori
• Primary Examiner(s): MCFADDEN, SUSAN IRIS
• Application Number:
  US11/688,916
• Publication Number:
  US 20070198269A1
• Time in Patent Office: 1,525 Days
• Field of Search: 704/270
• US Class Current: 704/270
• CPC Class Codes: G10L 15/01 Assessment or evaluation of speech recognition systems
• CPC Class Codes: G10L 15/06 Creation of reference templates; Training of speech recognition systems, e.g. adaptation to the characteristics of the speaker's voice G10L15/14 takes precedence

Abstract

A method for assessing a performance of a speech recognition system may include determining a grade, corresponding to either recognition of instances of a word or recognition of instances of various words among a set of words, wherein the grade indicates a level of the performance of the system and the grade is based on a recognition rate and at least one recognition factor. An apparatus for assessing a performance of a speech recognition system may include a processor that determines a grade, corresponding to either recognition of instances of a word or recognition of instances of various words among a set of words, wherein the grade indicates a level of the performance of the system and wherein the grade is based on a recognition rate and at least one recognition factor.

Claims

1. A method for assessing a performance of a speech recognition system, comprising:

using at least one processor, determining a grade, corresponding to either recognition of instances of a word or recognition of instances of various words among a set of words, wherein the grade indicates a level of the performance of the system; and

the grade is based on a recognition rate determined from a plurality of recognition observations and at least one recognition factor.

2. The method of claim 1, wherein the grade is a letter, score, number, color, bar having a certain length, or symbol, indicating a rating.

3. The method of claim 1, the grade is based on a numerical score for the recognition of a word.

4. The method of claim 3, further comprising:

determining the numerical score;

comparing the numerical score to a set of predetermined scores that indicate possible performance levels; and

assigning to the numerical score the grade.

5. The method of claim 1, wherein the recognition rate is an error rate based on estimated errors estimated from evaluating system behavior.

6. The method of claim 1, wherein the recognition rate is an accuracy rate based on estimated correct observations by the system estimated from evaluating system behavior.

7. The method of claim 5, wherein the recognition rate is determined, in part, using:

a count of occurrences for which instances of the word or instances of the various words are recognized within a certain confidence factor range.

8. The method of claim 5, wherein the recognition rate is determined, in part, using:

a count of occurrences for which the system rejects a first hypothesis of a first utterance, followed by a user repeating the first utterance in a second utterance, followed by the system accepting a second hypothesis of the second utterance.

9. The method of claim 5, wherein the recognition rate is determined, in part, using:

a count of occurrences when instances of the word or one of the various words are recognized incorrectly in the system'"'"'s hypothesis for a first utterance, followed by a user repeating the first utterance in a second utterance, followed by the system recognizing and accepting the hypothesis for the second utterance.

10. The method of claim 5, wherein the recognition rate is determined, in part, using:

a count of occurrences when a hypothesis generated by the system does not match an expected response word-for-word, but the hypothesis mostly matches the expected response word-for-word.

11. The method of claim 5, wherein the recognition rate is determined, in part, using:

a count of occurrences when a user provides feedback to the system.

12. The method of claim 1, wherein the recognition factor is a number of words in a vocabulary of the system.

13. The method of claim 1, wherein the recognition factor is an average number of words per utterance for utterances used in the determination of the grade.

14. The method of claim 1, wherein the word or the various words is a digit or digits.

15. The method of claim 1, wherein the recognition factor is an environmental factor.

16. The method of claim 15, wherein the environmental factor is a measurement of the user'"'"'s speech level.

17. The method of claim 15, wherein the environmental factor is a background noise level.

18. The method of claim 1, wherein the recognition factor is a measure of difficulty of recognition of the word or various words.

19. The method of claim 1, wherein the grade is a measure of performance of a system or systems used by one or more users.

20. The method of claim 1, wherein the recognition factor is based on performance of a system or systems used by one or more users.

21. The method of claim 1, further comprising:

determining if the grade for a word or various words from a set of words fails to meet a predetermined standard and improvement is therefore needed; and

if the improvement is needed, determining appropriate corrective action.

22. The method of claim 21, further comprising:

creating an alert to perform the corrective action through an alert mechanism.

23. The method of claim 1, further comprising:

determining if the grade of the system is not as good as other like-generated grades of other like systems used by other users, wherein the other like systems are operated in an environment similar to the system'"'"'s environment and improvement is therefore achievable; and

if the improvement is achievable, determining appropriate corrective action.

24. The method of claim 21, further comprising creating an alert to communicate the corrective action to the user.

25. The method of claim 1, further comprising:

generating a report of the performance of the system; and

displaying the report.

26. The method of claim 25, wherein the report includes information about performances of multiple systems used by multiple users.

27. The method of claim 25, further comprising displaying the report in a location other than on the speech recognition system.

28. The method of claim 1, wherein the method is performed as the system is being used by a user.

29. The method of claim 1, wherein the method is performed after the system is used by a user for a period of time, using information from a log of system activity.

40. The method of claim 5, wherein the error rate comprises:

a substitution rate at which an instance of the word or one of the various words is incorrectly recognized as another word, when a hypothesis generated by the system is compared to an expected response.

30. An apparatus for assessing a performance of a speech recognition system, comprising:

a processor that is configured to determine a grade, corresponding to either recognition of instances of a word or recognition of instances of various words among a set of words, wherein the grade indicates a level of the performance of the system; and

the grade is based on a recognition rate determined from a plurality of recognition observations and at least one recognition factor.

31. The apparatus of claim 30, further comprising:

a report generator that generates a report of the performance of the system; and

a display for displaying the report.

32. The apparatus of claim 31, wherein the display is located at the system.

33. The apparatus of claim 31, wherein the report includes information about performances of multiple systems used by multiple users.

34. The apparatus of claim 31, wherein the display is at a location other than where the system and the other like systems are located.

35. The apparatus of claim 31, wherein the display is a management console.

36. The apparatus of claim 31, wherein the display is a web browser.

37. The apparatus of claim 30, wherein the system is located on a portable terminal.

38. A method for assessing a performance of a speech recognition system, comprising:

using at least one processor, determining a grade, corresponding to either recognition of instances of a word or recognition of instances of various words among a set of words, wherein the grade indicates a level of the performance of the system; and

the grade is based on a count of a plurality of observations with errors or a count of a plurality of correct observations and at least one recognition factor.

39. An apparatus for assessing a performance of a speech recognition system, comprising:

a processor that is configured to determine a grade, corresponding to either recognition of instances of a word or recognition of instances of various words among a set of words, wherein the grade indicates a level of the performance of the system; and

the grade is based on a count of a plurality of observations with errors or a count of a plurality of correct observations and at least one recognition factor.