Automatically improving a voice recognition system

• US 20030125945A1
• Filed: 12/14/2001
• Published: 07/03/2003
• Est. Priority Date: 12/14/2001
• Status: Active Grant

• Current Assignee: Intellectual Ventures I LLC (Intellectual Ventures LLC)
• Original Assignee: Ben Franklin Patent Holding LLC (Intellectual Ventures LLC)
• Inventors: Doyle, Sean
• Granted Patent: US 7,103,542 B2
• Time in Patent Office: Days
• US Class Current: 704/246
• CPC Class Codes: G10L 15/01 Assessment or evaluation of speech recognition systems

Abstract

Systems and methods for automatically improving a voice recognition system are provided. In one embodiment, the systems and methods retrieve voice recognition information produced by a voice recognition system in response to recognizing a user utterance. The voice recognition information comprises a recognized voice command associated with the user utterance and a reference to an audio file that includes the user utterance. The audio file is played and it is determined if the recognized voice command matches the user utterance included in the audio file. The user utterance is then transcribed to create a transcribed utterance, if the recognized voice command does not match the user utterance. The transcribed utterance is then recorded in association with the recognized voice command to monitor recognition accuracy.

Claims

1. An automated method of monitoring recognition accuracy in a voice recognition system comprising:

retrieving voice recognition information produced by a voice recognition system in response to recognizing a user utterance, the voice recognition information comprising a recognized voice command associated with the user utterance and a reference to an audio file that includes the user utterance;

playing the audio file;

determining if the recognized voice command matches the user utterance included in the audio file;

transcribing the user utterance to create a transcribed utterance, if the recognized voice command does not match the user utterance; and

recording the transcribed utterance in association with the recognized voice command to monitor recognition accuracy of the voice recognition system.

2. The method of claim 1, wherein the voice recognition information further comprises information for a recognition grammar used to recognize the user utterance.

3. The method of claim 1, wherein the voice recognition information further comprises a confidence threshold associated with the recognized voice command.

4. The method of claim 1, wherein the voice recognition information further comprises a recognition delay associated with the recognized voice command.

5. The method of claim 1, wherein the transcribed utterance is recorded in a transcription log in association with the recognized voice command and the corresponding voice recognition information.

6. The method of claim 5, wherein the voice recognition information includes the audio file that includes the user utterance.

7. The method of claim 5, further comprising:

analyzing information recorded in the transcription log to detect a type of recognition error.

8. The method of claim 7, further comprising:

determining the source of the detected recognition error based on the type of recognition error.

9. The method of claim 8, further comprising:

hypothesizing a solution to eliminate the source of error.

10. The method of claim 5, further comprising:

determining recognition accuracy based on the information recorded in the transcription log.

11. A system for monitoring recognition accuracy of a voice recognition system having at least one recognition grammar comprising:

software for retrieving voice recognition information stored in a call log by a voice recognition system in response to recognizing a user utterance, the voice recognition information comprising a recognized voice command associated with the user utterance and a reference to an audio file that includes the user utterance;

a transcription log produced by the transcription software, the transcription log comprising an entry associated with the user utterance, the entry comprising;

the recognized voice command; and

a transcribed utterance produced from transcribing the user utterance, if the user utterance included in the audio file does not match the recognized voice command associated with it;

wherein the software determines a recognition accuracy for the voice recognition system by analyzing the entry recorded in the transcription log.

12. The system of claim 11, wherein the voice recognition information further comprises information for a recognition grammar used to recognize the user utterance.

13. The system of claim 11, wherein the voice recognition information further comprises a confidence threshold associated with the recognized voice command.

14. The system of claim 11, wherein the voice recognition information further comprises:

a recognition delay associated with the recognized voice command.

15. The system of claim 11, wherein the transcribed utterance is compared with the recognized voice command to determine if an error in recognition has occurred.

16. The system of claim 15, wherein the software records in an error log a type of recognition error associated with a recognized user utterance, if the transcribed utterance does not match the recognized voice command.

17. The system of claim 16, wherein the software records an in-grammar false reject (IGFR) type error, if the transcribed utterance is part of the recognition grammar but the user utterance is not.

18. The system of claim 16, wherein the software records an in-grammar false accept (IGFA) type error, if the transcribed utterance is part of the recognition grammar and the recognized voice command does not match the transcribed utterance.

19. The system of claim 16, wherein the software records an out-of-grammar false accept (OGFA) type error, if the transcribed utterance is not part of the recognition grammar and the recognized voice command is mistakenly matched with the transcribed utterance.

20. The system of claim 16, wherein the software records an out-of-grammar correct reject (OGCR) type error if the transcribed utterance is not part of the recognition grammar but the transcribed utterance is frequently recorded in the transcription log.

21. The system of claim 17, wherein:

a source of error is determined to be a low confidence threshold, if an IGFA type error is detected.

22. The system of claim 17, wherein:

a source of error is determined to be acoustic similarity between the recognized voice command and the user utterance, if an IGFA type error is detected.

23. The system of claim 17, wherein the recognition grammar includes one or more acoustic models for a recognized voice command, wherein:

a source of error is determined to be lack of sufficient acoustic models in the recognition grammar, if an IGFA type error is detected.

24. The system of claim 18, wherein a source of error is determined to be a high confidence threshold, if an IGFR type error is detected.

25. The system of claim 18, wherein the recognition grammar includes one or more acoustic models for a recognized voice command, wherein:

a source of error is determined to be lack of sufficient acoustic models in the recognition grammar, if an IGFR type error is detected.

26. The system of claim 18, wherein the voice recognition system is configured with a first recognition sensitivity threshold, wherein:

a source of error is determined to be a low sensitivity threshold, if an IGFR type error is detected.

27. The system of claim 19, wherein:

a source of error is determined to be acoustic similarity between the recognized voice command and the user utterance, if an OGFA type error is detected.

28. The system of claim 19, wherein the voice recognition system is configured with a first recognition sensitivity threshold, wherein:

a source of error is determined to be a high sensitivity threshold, if an OGFA type error is detected.

29. The system of claim 19, wherein a source of error is determined to be a low confidence threshold, if an OGFA type error is detected.

30. The system of claim 20, wherein the recognition grammar includes one or more acoustic models for the recognized voice command, wherein:

a source of error is determined to be lack of sufficient acoustic models in the recognition grammar, if an OGCR type error is detected.

31. A method for automatically improving a voice recognition system comprising:

determining voice recognition accuracy by analyzing voice recognition information produced by a voice recognition system in response to receiving a user utterance, the voice recognition system comprising a recognition grammar and a first configuration, the voice recognition information comprising a recognized voice command associated with the user utterance;

detecting a recognition error, if the recognized voice command does not match a transcribed utterance for the user utterance;

determining at least one source of error for the detected recognition error;

hypothesizing at least one solution to eliminate the source of error; and

modifying the first configuration based on the hypothesized solution to implement a second configuration for the voice recognition system with better recognition accuracy.

32. The method of claim 31, further comprising:

testing the voice recognition system to detect improvements in voice recognition accuracy; and

restoring the first configuration, if voice recognition accuracy is not improved.

33. The method of claim 32, wherein the voice recognition information further comprises information for a recognition grammar used to recognize the user utterance.

34. The method of claim 33, wherein the voice recognition information further comprises a confidence threshold associated with the recognized voice command.

35. The method of claim 34, wherein the voice recognition information further comprises a recognition delay associated with the recognized voice command.

36. The method of claim 35, wherein an in-grammar false accept (IGFA) type error is detected, if the transcribed utterance is part of the recognition grammar and the recognized voice command does not match the transcribed utterance.

37. The method of claim 35, wherein an in-grammar false reject (IGFR) type error is detected, if the transcribed utterance is part of the recognition grammar but the user utterance is not.

38. The method of claim 35, wherein an out-of-grammar false accept (OGFA) type error is detected, if the transcribed utterance is not part of the recognition grammar and the recognized voice command is mistakenly matched with the transcribed utterance.

39. The method of claim 35, wherein an out-of-grammar correct reject (OGCR) type error is detected, if the transcribed utterance is not part of the recognition grammar but the transcribed utterance is frequently recorded in the transcription log.

40. The method of claim 36, wherein a source of error is determined to be a low confidence threshold, if an IGFA type error is detected.

41. The method of claim 36, wherein a source of error is determined to be acoustic similarity between the recognized voice command and the user utterance, if an IGFA type error is detected.

42. The method of claim 36, wherein the recognition grammar includes one or more acoustic models for a recognized voice command, wherein a source of error is determined to be lack of sufficient acoustic models in the recognition grammar, if an IGFA type error is detected.

43. The method of claim 37, wherein a source of error is determined to be a high confidence threshold, if an IGFR type error is detected.

44. The method of claim 37, wherein the recognition grammar includes one or more acoustic models for a recognized voice command, wherein a source of error is determined to be lack of sufficient acoustic models in the recognition grammar, if an IGFR type error is detected.

45. The method of claim 37, wherein the voice recognition system is configured with a first recognition sensitivity threshold, wherein a source of error is determined to be a low sensitivity threshold, if an IGFR type error is detected.

46. The method of claim 38, wherein a source of error is determined to be acoustic similarity between the recognized voice command and the user utterance, if an OGFA type error is detected.

47. The method of claim 38, wherein the voice recognition system is configured with a first recognition sensitivity threshold, wherein a source of error is determined to be a high sensitivity threshold, if an OGFA type error is detected.

48. The method of claim 38, wherein a source of error is determined to be a low confidence threshold, if an OGFA type error is detected.

49. The method of claim 39, wherein the recognition grammar includes one or more acoustic models for a recognized voice command, wherein a source of error is determined to be lack of sufficient acoustic models in the recognition grammar, if an OGFR type error is detected.

50. A method of improving recognition accuracy and efficiency in a voice recognition system having a grammar, wherein a received utterance is accepted if the utterance is an acoustic match with at least one term included in the grammar and rejected otherwise, the method comprising:

determining if the utterance has been falsely accepted by the voice recognition system; and

removing a term from the grammar, if the utterance is part of the grammar but is falsely accepted as another term included in the grammar due to acoustic similarity.

51. The method of claim 50, further comprising:

adding to the grammar one or more synonyms which are acoustically dissimilar to the utterance if it has been falsely accepted.

52. The method of claim 50, wherein the voice recognition system comprises at least one acoustic model for the utterance, the method further comprising:

adding to the voice recognition system an additional acoustic model for the utterance, if the utterance is part of the grammar, but is falsely rejected due to lack of sufficient alternative acoustic models.

53. The method of claim 52, wherein the additional acoustic model is one that is associated with the rejected utterance.

54. The method of claim 50, wherein the voice recognition system has a confidence threshold for adjusting recognition accuracy based on degree of acoustic similarity between the utterance and one or more acoustic models included in the voice recognition system, wherein the recognition accuracy is improved if the confidence threshold is higher, the method further comprising:

lowering the confidence threshold, if the utterance is part of the grammar, but is falsely rejected due to a high confidence threshold.

55. The method of claim 50, the voice recognition system having a sensitivity threshold for accepting utterances, wherein a higher sensitivity allows for a larger number of utterances to be considered by the voice recognition system, the method further comprising:

lowering the sensitivity threshold if a predetermined amount of noise is undesirably considered by the voice recognition system.

56. The method of claim 55, further comprising:

increasing the sensitivity threshold, if the utterance is part of the grammar, but is falsely rejected due to a low sensitivity threshold.

57. The method of claim 50, the voice recognition system having a timeout threshold and an end-of-speech threshold for accepting an utterance having a certain audio length or audio delay, the method further comprising:

adjusting the timeout and the end-of-speech thresholds, if a predetermined amount of noise is undesirably considered for acceptance by the system.

58. The method of claim 50, wherein the grammar has a perplexity associated with a number of decision levels and terms included in the grammar, the method further comprising:

reducing grammar perplexity, if recognition of the utterance is delayed beyond a threshold value.

59. The method of claim 50, further comprising:

monitoring frequently received utterances that are rejected because they are not part of the grammar; and

adding the frequently received utterances to the grammar.

60. A method performed on a computer for monitoring recognition accuracy in a voice recognition system comprising:

recording a user utterance in an audio file;

storing a voice command which was recognized by the voice recognition system for the user utterance;

presenting the audio file and the voice command to an operator for selection;

playing the user utterance from the audio file for the operator upon selection by the operator;

receiving a transcription of the user utterance; and

associating the transcription with the recognized voice command.