Method and apparatus for determining a corrective action for a speech recognition system based on the performance of the system

• US 8,255,219 B2
• Filed: 03/09/2011
• Issued: 08/28/2012
• Est. Priority Date: 02/04/2005
• Status: Active Grant

• Current Assignee: Vocollect, Inc. (Honeywell International Inc.)
• Original Assignee: Vocollect, Inc. (Honeywell International Inc.)
• Inventors: Braho, Keith, Pike, Jeffrey, El-Jaroudi, Amro, Pike, Lori, Laughery, Michael
• Primary Examiner(s): MCFADDEN, SUSAN IRIS
• Application Number:
  US13/044,076
• Publication Number:
  US 20110161083A1
• Time in Patent Office: 538 Days
• Field of Search: 704/251
• US Class Current: 704/251
• CPC Class Codes: G10L 15/01 Assessment or evaluation of speech recognition systems
• CPC Class Codes: G10L 15/06 Creation of reference templates; Training of speech recognition systems, e.g. adaptation to the characteristics of the speaker's voice G10L15/14 takes precedence

Abstract

A method for assessing a performance of a speech recognition system may include determining a grade, corresponding to either recognition of instances of a word or recognition of instances of various words among a set of words, wherein the grade indicates a level of the performance of the system and the grade is based on a recognition rate and at least one recognition factor. An apparatus for assessing a performance of a speech recognition system may include a processor that determines a grade, corresponding to either recognition of instances of a word or recognition of instances of various words among a set of words, wherein the grade indicates a level of the performance of the system and wherein the grade is based on a recognition rate and at least one recognition factor.

Claims

1. A method for improving performance of a speech recognition system comprising:

operating a processor to determine a performance of the system based on either recognition of instances of a word or recognition of instances of various words among a set of words, and

determining a corrective action based on the performance, to improve the performance.

2. The method of claim 1, further comprising communicating the corrective action to a user.

3. The method of claim 1, further comprising initiating the corrective action automatically.

4. The method of claim 1, wherein the method is performed as the system is being used by a user.

5. The method of claim 1, wherein the method is performed after the system is used by a user for a period of time, using information from a log of system activity.

6. The method of claim 1, wherein the corrective action is for the user to retrain a word or set of words.

7. The method of claim 1, wherein the corrective action is for the user to perform an action that causes a model or set of models to be modified.

8. The method of claim 1, wherein the corrective action is for the user to speak louder.

9. The method of claim 1, wherein the corrective action is for the user to move the microphone.

10. The method of claim 1, wherein the corrective action is for the user to replace a microphone.

11. The method of claim 1, wherein determining the performance is based on estimated errors, which are estimated from evaluating system behavior.

12. The method of claim 11, wherein the estimated errors estimated from evaluating system behavior correspond to proximate utterances input to the system by the user.

13. The method of claim 1, wherein determining the performance is based on estimated correct observations by the system estimated from evaluating system behavior.

14. The method of claim 1, wherein determining the performance is based, at least in part, on identifying occurrences for which instances of the word or instances of the various words are recognized within a certain confidence factor range.

15. The method of claim 1, wherein determining the performance uses, at least in part, identifying occurrences for which the system rejects a first hypothesis of a first utterance, followed by the system accepting a second hypothesis of the second utterance, wherein the first and second hypotheses substantially match word-for-word.

16. The method of claim 1, wherein determining the performance uses, at least in part:

identifying occurrences when the system generates first and second hypotheses of two utterances and the system accepts the second hypothesis, wherein the two hypotheses do not match word-for-word, but the hypotheses mostly match word-for-word.

17. The method of claim 1, wherein determining the performance uses, at least in part:

identifying occurrences when a hypothesis generated by the system does not match an expected response word-for-word, but the hypothesis mostly matches the expected response word-for-word.

18. The method of claim 1, wherein determining the performance uses, at least in part:

identifying occurrences when a user provides feedback to the system.

19. The method of claim 1, wherein determining the corrective action is based on a recognition factor.

20. The method of claim 1, wherein determining the corrective action is based on an environmental factor.

21. The method of claim 1, wherein determining the corrective action is based on performance of a system or systems used by one or more users.

22. The method of claim 1, wherein the performance is based on an error rate comprising:

a rate at which the system rejects a first utterance, followed by a user repeating the first utterance in a second utterance, followed by the system accepting the second utterance.

23. The method of claim 1, wherein the performance is based on an error rate comprising:

a rate at which the system substitutes an instance of the word or one of the various words in the system'"'"'s hypothesis for a first utterance, followed by a user repeating the first utterance in a second utterance, followed by the system recognizing and accepting the second utterance.

24. The method of claim 1, wherein the performance is based on an error rate comprising:

a substitution rate at which the word an instance of the word or one of the various words is incorrectly recognized as another word, when comparing a hypothesis generated by the system to an expected response.

25. An apparatus for improving performance of a speech recognition system comprising:

a processor that is configured to determine a performance of the speech recognition system based on either recognition of instances of a word or recognition of instances of various words among a set of words;

the processor further configured to determine a corrective action based on the previously determined performance to improve the performance of the speech recognition system.

26. The apparatus of claim 25, wherein the processor is adapted to communicate the corrective action to a user.

27. The apparatus of claim 25, wherein the processor is adapted to initiate the corrective action automatically.

28. The apparatus of claim 25, wherein the apparatus determines the performance and determines the corrective action as the system, is being used by a user.

29. The apparatus of claim 25, wherein the apparatus determines the performance and determines the corrective action after the system is used by a user for a period of time, using information from a log of system activity.

30. The apparatus of claim 25, wherein the performance is determined using estimated errors, which are estimated from evaluating system behavior.

31. The apparatus of claim 25, wherein the performance is determined using estimated correct observations, which are estimated from evaluating system behavior.

32. The apparatus of claim 30, wherein the estimated errors include occurrences for which instances of the word or instances of the various words are recognized within a certain confidence factor range.

33. The apparatus of claim 30, wherein the estimated errors include occurrences for which the system rejects a first hypothesis of a first utterance, followed by the system accepting a second hypothesis of the second utterance, wherein the first and second hypotheses substantially match word-for-word.

34. The apparatus of claim 30, wherein the estimated errors include occurrences when the system generates first and second hypotheses of two utterances, and the system accepts the second hypothesis, wherein the two hypotheses do not match word-for-word, but the hypotheses mostly match word-for-word.

35. The apparatus of claim 30, wherein the estimated errors include occurrences when a hypothesis generated by the system does not match an expected response word-for-word, but the hypothesis mostly matches the expected response word-for-word.

36. The apparatus of claim 30, wherein the estimated errors include occurrences of when a user provides feedback to the system.

37. The apparatus of claim 25, wherein determining the corrective action is based on a recognition factor.

38. The apparatus of claim 25, wherein determining the corrective action is based on an environmental factor.

39. The apparatus of claim 25, wherein determining the corrective action is based on performance of a system or systems used by one or more users.

40. The apparatus of claim 25, wherein the performance is based on an error rate comprising:

a rate at which the system rejects a first utterance, followed by a user repeating the first utterance in a second utterance, followed by the system accepting the second utterance.

41. The apparatus of claim 25, wherein the performance is based on an error rate comprising:

a rate at which the system substitutes an instance of the word or one of he various words in the system'"'"'s hypothesis for a first utterance, followed by a user repeating the first utterance in a second utterance, followed by the system recognizing and accepting the second utterance.

42. The apparatus of claim 25, wherein the performance is based on an error rate comprising:

a substitution rate at which the word an instance of the word or one of the various words is incorrectly recognized as another word, when comparing a hypothesis generated by the system to an expected response.