Methods and systems for adapting a model for a speech recognition system

• US 20070192095A1
• Filed: 10/06/2006
• Published: 08/16/2007
• Est. Priority Date: 02/04/2005
• Status: Active Grant

• Current Assignee: Vocollect, Inc. (Honeywell International Inc.)
• Original Assignee: Vocollect, Inc. (Honeywell International Inc.)
• Inventors: Pike, Jeffrey, Braho, Keith, Pike, Lori
• Granted Patent: US 7,827,032 B2
• Time in Patent Office: Days
• US Class Current: 704/232
• CPC Class Codes: G10L 15/01 Assessment or evaluation of speech recognition systems
• CPC Class Codes: G10L 15/065 Adaptation

Abstract

Methods are disclosed for identifying possible errors made by a speech recognition system without using a transcript of words input to the system. A method for model adaptation for a speech recognition system includes determining an error rate, corresponding to either recognition of instances of a word or recognition of instances of various words, without using a transcript of words input to the system. The method may further include adjusting an adaptation, of the model for the word or various models for the various words, based on the error rate. Apparatus are disclosed for identifying possible errors made by a speech recognition system without using a transcript of words input to the system. An apparatus for model adaptation for a speech recognition system includes a processor adapted to estimate an error rate, corresponding to either recognition of instances of a word or recognition of instances of various words, without using a transcript of words input to the system. The apparatus may further include a controller adapted to adjust an adaptation of the model for the word or various models for the various words, based on the error rate.

Claims

1. A method for model adaptation for a speech recognition system comprising:

determining an error rate, corresponding to either recognition of instances of a word or recognition of instances of various words among a set of words; and

adjusting an adaptation of a model for the word or various models for the various words, based on the error rate.

2. The method of claim 1, wherein the error rate is based on actual errors determined from comparing a transcript of words input to the system with corresponding hypotheses generated by the system.

3. The method of claim 1, wherein the error rate is based on estimated errors estimated from evaluating system and user behavior corresponding to proximate utterances input to the system by the user.

4. The method of claim 1, wherein adjusting an adaptation comprises adapting the model or the various models or withholding adapting the model or the various models based on the error rate.

5. The method of claim 1, wherein adjusting the adaptation comprises:

making a comparison of the error rate to an error rate threshold; and

adapting the model or the various models or withholding adapting the model or the various models based on the comparison.

6. The method of claim 1, wherein adjusting the adaptation comprises:

withholding adapting the model or the various models until the error rate meets an error rate threshold.

7. The method of claim 5, wherein the error rate threshold is a predetermined value.

8. The method of claim 5, wherein the error rate threshold is a value settable by a user.

9. The method of claim 5, wherein the error rate threshold is a dynamic value.

10. The method of claim 5, wherein the error rate threshold is based on a number of words in a hypothesis of an utterance input to the system.

11. The method of claim 5, wherein the error rate threshold is based on an environmental factor.

12. The method of claim 5, wherein the error rate threshold is based on a measure of difficulty of the recognition of the word.

13. A method for identifying a possible error made by a speech recognition system, comprising:

identifying an instance of a word that was recognized by the system within a certain confidence factor range.

14. The method of claim 13, wherein a count of occurrences of the possible error is used in adjusting an adaptation a model for the word associated with the possible error.

15. The method of claim 13, wherein the word is from a hypothesis generated by the

system that matches an expected response.

16. The method of claim 13, wherein the range comprises values corresponding to low confidence that the system recognized the instances of the word correctly for an application in which the system is used.

17. The method of claim 13, wherein the range is a predetermined set of values in between a low confidence threshold and a high confidence threshold.

18. The method of claim 17, wherein the low confidence threshold is an adjusted acceptance threshold, which is a value lower than the acceptance threshold.

19. The method of claim 17, wherein the high confidence threshold is a value equal to or higher than an acceptance threshold for accepting a hypothesis generated by the system, in which a confidence value associated with the hypothesis must exceed the acceptance threshold in order for the hypothesis to be accepted.

20. A method for identifying a possible error made by a speech recognition system comprising:

identifying an instance where the system rejects a first hypothesis of a first utterance, followed by the system accepting a second hypothesis of a second utterance, wherein the first and second hypotheses substantially match word-for-word.

21. The method of claim 20, wherein the first and second hypotheses substantially match word-for-word by matching word-for-word, except that one of the first and second hypotheses includes at least one additional recognized model that is negligible for purposes of identifying the possible error.

22. The method of claim 20, wherein a count of occurrences of the possible error is used in adjusting an adaptation of a model for a word associated with the possible error.

23. The method of claim 20, wherein the system rejects the first hypothesis due to a confidence factor of the first hypothesis not exceeding an acceptance threshold.

24. The method of claim 20, wherein the system rejects at least one word in the first hypothesis.

25. The method of claim 20, wherein the first and second utterances are spoken consecutively, proximately or within a predetermined amount of time.

26. A method for identifying a possible error made by a speech recognition system comprising:

identifying when the system generates first and second hypotheses of two utterances and the system accepts the second hypothesis, wherein the two hypotheses do not match word-for-word, but the hypotheses mostly match word-for-word.

27. The method of claim 26, wherein the hypotheses mostly match word-for-word by matching word-for-word except for a predetermined number of words.

28. The method of claim 26, wherein a count of occurrences of the possible error is used in adjusting an adaptation a model for a word associated with the possible error.

29. The method of claim 26, wherein the two utterances are spoken consecutively, proximately, or within a predetermined amount of time.

30. The method of claim 26, wherein there are no intervening utterances that indicate that the first of the two utterances was correctly recognized by the system.

31. The method of claim 26, wherein the two hypotheses differ in that a word in the second hypothesis is substituted with a word in the first hypothesis.

32. The method of claim 26, wherein the hypotheses differ in that a word in the second hypothesis is substituted with garbage in the first hypothesis.

33. A method for identifying a possible error made by a speech recognition system, comprising:

identifying when a hypothesis generated by the system does not match an expected response word-for-word, but the hypothesis mostly matches the expected response word-for-word.

34. The method of claim 33, wherein a count of occurrences of the possible errors is used in adjusting adaptation of a model for a word associated with the possible errors.

35. The method of claim 33, wherein the hypothesis mostly matches the expected response word-for-word by matching word-for-word except for one word.

36. A method for adapting a model for a speech recognition system comprising:

generating a count of occurrences of when a user provides feedback to the system; and

adjusting an adaptation of the model based on the count.

37. The method of claim 36, wherein the count is determined without using a transcript of words input to the system.

38. The method of claim 36, wherein the count of occurrences includes a count of occurrences of when the user indicates that the system made a mistake.

39. The method of claim 36, wherein the user provides the feedback in response to the system requesting the feedback.

40. The method of claim 39, wherein the feedback includes a word or other indication of aggravation by the user.

41. The method of claim 39, wherein the feedback is a correction command to the system.

42. An apparatus for model adaptation for a speech recognition system comprising:

a processor adapted to determine an error rate, corresponding to either recognition of instances of a word or recognition of instances of various words among a set of words; and

a controller adapted to adjust an adaptation of a model for the word or various models for the various words, based on the error rate.

43. The apparatus of claim 42, wherein the error rate is based on actual errors determined from comparing a transcript of words input to the system with corresponding hypotheses generated by the system.

44. The apparatus of claim 42, wherein the error rate is based on estimated errors estimated from evaluating system and user behavior corresponding to proximate utterances input to the system by the user.

45. The apparatus of claim 42, wherein the controller adjusts an adaptation by adapting the model or the various models or withholding adapting the model or the various models based on the error rate.

46. The apparatus of claim 42, wherein the controller adjusts the adaptation by making a comparison of the error rate to an error rate threshold;

and

adapting the model or the various models or withholding adapting the model or the various models based on the comparison.

47. The apparatus of claim 42, wherein the controller adjusts the adaptation by withholding adapting the model or the various models until the error rate meets an error rate threshold.

48. The apparatus of claim 42, wherein the error rate threshold is a predetermined value.

49. The apparatus of claim 42, wherein the error rate threshold is a value settable by a user.

50. The apparatus of claim 42, wherein the error rate threshold is a dynamic value.

51. The apparatus of claim 42, wherein the error rate threshold is based on a number of words in a hypothesis of an utterance input to the system.

52. The apparatus of claim 42, wherein the error rate threshold is based on an environmental factor.

53. The apparatus of claim 42, wherein the error rate threshold is based a measure of difficulty of the recognition of the word.

54. An apparatus for identifying a possible error made by a speech recognition system, comprising:

a processor adapted to identify an instance of a word that was recognized by the system within a certain confidence factor range.

55. The apparatus of claim 54, wherein a count of occurrences of the possible error is used in adjusting adaptation of a model for the word associated with the possible error.

56. The apparatus of claim 54, wherein the word is from a hypothesis generated by the system that matches an expected response.

57. The apparatus of claim 54, wherein the range comprises values corresponding to low confidence that the system recognized the instances of the word correctly for an application in which the system is used.

58. The apparatus of claim 54, wherein the range is a predetermined set of values in between a low confidence threshold and a high confidence threshold.

59. The apparatus of claim 58, wherein the high confidence threshold is an acceptance threshold for accepting a hypothesis generated by the system, in which a confidence threshold associated with the hypothesis must exceed the acceptance threshold in order for the hypothesis to be accepted.

60. The apparatus of claim 58, wherein the low confidence threshold is an adjusted acceptance threshold, which is a value lower than the acceptance threshold.

61. The apparatus of claim 58, wherein the high confidence threshold is a value equal to or higher than an acceptance threshold for accepting a hypothesis generated by the system, in which a confidence value associated with the hypothesis must exceed the acceptance threshold in order for the hypothesis to be accepted.

62. An apparatus for identifying a possible error made by a speech recognition system comprising:

a processor adapted to identify an instance where the system rejects a first hypothesis of a first utterance, followed by the system accepting a second hypothesis of a second utterance, wherein the first and second hypotheses substantially match word-for-word.

63. The apparatus of claim 62, wherein the first and second hypotheses substantially match word-for-word by matching word-for-word, except that one of the first and second hypotheses includes at least one additional recognized model that is negligible for purposes of identifying the possible error.

64. The apparatus of claim 62, wherein a count of occurrences of the possible error is used in adjusting an adaptation of a model for a word associated with the possible error.

65. The apparatus of claim 62, wherein the system rejects the first hypothesis due to a confidence factor of the first hypothesis not exceeding an acceptance threshold.

66. The apparatus of claim 62, wherein the system rejects at least one word in the first hypothesis.

67. The apparatus of claim 62, wherein the first and second utterances are spoken consecutively, proximately or within a predetermined amount of time.

68. An apparatus for identifying a possible error made by a speech recognition system comprising:

a processor adapted to identify when the system generates first and second hypotheses of two utterances and the system accepts the second hypothesis, wherein the two hypotheses do not match word-for-word, but the hypotheses mostly match word-for-word.

69. The apparatus of claim 68, wherein the hypotheses mostly match word-for-word by matching word-for-word except for a predetermined number of words.

70. The apparatus of claim 68, wherein a count of occurrences of the possible error is used in adjusting an adaptation a model for a word associated with the possible error.

71. The apparatus of claim 68, wherein the two utterances are spoken consecutively, proximately, or within a predetermined amount of time.

72. The apparatus of claim 68, wherein there are no intervening utterances that indicate that the first of the two utterances was correctly recognized by the system.

73. The apparatus of claim 68, wherein the two hypotheses differ in that a word in the second hypothesis is substituted with a word in the first hypothesis.

74. The apparatus of claim 68, wherein the hypothesis differ in that a word in the second hypothesis is substituted with garbage in the first hypothesis.

75. An apparatus for identifying a possible error rate made by a speech recognition system, comprising:

a processor adapted to identify when a hypothesis generated by the system does not match an expected response word-for-word, but the hypothesis mostly matches the expected response word-for-word.

76. The apparatus of claim 75, wherein a count of occurrences of the possible error is used in adjusting an adaptation of a model for a word associated with the possible error.

77. The apparatus of claim 75, wherein the hypothesis mostly matches the expected response word-for-word by matching word-for-word except for a predetermined number of words.

78. An apparatus for adapting a model for a speech recognition system comprising:

a processor adapted to generate a count of occurrences of when a user provides feedback to the system; and

a controller adapted to adjust an adaptation of the model based on the count.

79. The apparatus of claim 78, wherein the count is determined without using a transcript of words input to the system.

80. The apparatus of claim 78, wherein the transcript is compared with corresponding hypotheses generated by the system.

81. The apparatus of claim 78, wherein the count includes a count of occurrences of when the user indicates that the system made a mistake.

82. The apparatus of claim 78, wherein the user provides the feedback in response to the system requesting the feedback.

83. The apparatus of claim 82, wherein the feedback includes a word indicating aggravation by the user.

84. The apparatus of claim 82, wherein the feedback is a correction command to the system.