Method and system for dynamically adjusted training for speech recognition

• US 5,963,903 A
• Filed: 06/28/1996
• Issued: 10/05/1999
• Est. Priority Date: 06/28/1996
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Mahajan, Milind V., Ju, Yun-Cheng, Jiang, Li, Huang, Xuedong D., Hwang, Mei-Yuh, Hon, Hsiao-Wuen, Rozak, Michael J.
• Primary Examiner(s): Dorvil, Richemond
• Application Number:
  US08/673,435
• Time in Patent Office: 1,194 Days
• Field of Search: 704/277, 704/238, 704/200, 704/243, 704/222, 704/256, 704/244, 704/254, 704/257, 704/239, 704/240, 704/251, 704/249, 704/246, 704/234, 704/231
• US Class Current: 704/254
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 2015/0635 updating or merging of old and new templates; Mean values; Weighting

Abstract

A method and system for dynamically selecting words for training a speech recognition system. The speech recognition system models each phoneme using a hidden Markov model and represents each word as a sequence of phonemes. The training system ranks each phoneme for each frame according to the probability that the corresponding codeword will be spoken as part of the phoneme. The training system collects spoken utterances for which the corresponding word is known. The training system then aligns the codewords of each utterance with the phoneme that it is recognized to be part of. The training system then calculates an average rank for each phoneme using the aligned codewords for the aligned frames. Finally, the training system selects words for training that contain phonemes with a low rank.

Claims

1. A method in a computer system for dynamically selecting words for training a speech recognition system, the speech recognition system for recognizing a plurality of words, the speech recognition system having indications of phonemes that compose each word, the speech recognition system representing each spoken utterance by a sequence of codewords, the speech recognition system having a model for each phoneme, each model for generating a probability that each possible sequence of codewords corresponds to the modeled phoneme, the method comprising:

for each codeword, ranking the phonemes according to the probability that the codeword will be spoken as part of the phoneme;

collecting a plurality of spoken utterances for which the corresponding word is known;

for each collected utterance,

converting the collected utterance to a sequence of codewords; and

aligning each codeword in the sequence of codewords with a phoneme of the known word to which the collected utterance corresponds based on the phoneme models;

for each phoneme,

accumulating the ranks of that phoneme with all codewords with which it is aligned in each of the collected utterances; and

calculating an average rank of the phoneme by dividing the accumulated rank by a total number of codewords that are aligned with that phoneme in the collected utterances;

identifying a phoneme with low average rank; and

selecting words that contain the identified phoneme as words for training the speech recognition system.

2. The method of claim 1 wherein the models are hidden Markov models.

3. The method of claim 1 wherein the aligning uses a Viterbi-based alignment algorithm.

4. The method of claim 1 including presenting the selected words to a speaker for training.

5. The method of claim 4 wherein the probability that the word will be spoken by a speaker wherein the selected words are preferentially selected.

6. The method of claim 1 wherein the identifying of a phoneme includes identifying more than one phoneme with a low average rank and wherein the selecting selects words that contain more than one of the identified phonemes.

7. The method of claim 6 including presenting the selected words to a speaker for training.

8. The method of claim 7 wherein each word has a language model probability indicating the probability that the word will be spoken by a speaker and wherein the selected words are presented to the speaker in an order based on the language model probability of the selected words.

9. A method in a computer system for dynamically selecting words for training a speech recognition system, the speech recognition system for recognizing a plurality of words, the speech recognition system having indications of phonetic units that compose each word, the speech recognition system representing speech as a sequence of feature vectors, the speech recognition system having model for each phonetic unit, each model for generating a probability that each possible sequence of feature vectors corresponds to the modeled phonetic unit, the method comprising:

collecting a plurality of spoken utterances for which the corresponding word is known;

for each collected utterance,

converting the collected utterance to a sequence of feature vectors; and

aligning each feature vector in the sequence of feature vectors with a phonetic unit of the known word to which the collected utterance corresponds based on the models of the phonetic units of the known word;

identifying from the feature vectors aligned with each phonetic unit a phonetic unit that is not accurately modeled; and

selecting words that contain the identified phonetic unit as words for training the speech recognition system.

10. The method of claim 9 wherein identifying which phonetic units are least accurately modeled includes calculating a frame accuracy measure based on the aligned phonetic unit and feature vector, and calculating a combined accuracy measure for each unique phonetic unit by combining the frame accuracy measures that are based on that phonetic unit.

11. The method of claim 10 wherein the frame accuracy measure is a ratio of the probability that the feature vector is contained within the phonetic unit with which it is aligned to a highest probability that the feature vector is contained within any phonetic unit.

12. The method of claim 10 wherein the frame accuracy measure is a difference of the probability that the feature vector is contained within the phonetic unit with which it is aligned to a highest probability that the feature vector is contained within any phonetic unit.

13. The method of claim 10 wherein the frame accuracy measure is a rank of each phonetic unit based on the probability that those vector frames aligned with the phonetic unit will be spoken as part of the phonetic unit.

14. The method of claim 13 wherein the probability that those feature vectors aligned with the phonetic unit will be spoken as part of the phonetic unit is an acoustic model probability.

15. The method of claim 13 wherein the combined accuracy measure is an average of the probability that each feature vector that is aligned with that phonetic unit is contained within that phonetic unit.

16. The method of claim 13 wherein the combined accuracy measure is a maximum of the probabilities that each feature vector that is aligned with that phonetic unit is contained within that phonetic unit.

17. The method of claim 13 wherein the combined accuracy measure is a minimum of the probabilities that each feature vector that is aligned with that phonetic unit is contained within that phonetic unit.

18. The method of claim 13 wherein the combined accuracy measure is a weighted average of the probabilities that each feature vector that is aligned with that phonetic unit is contained within that phonetic unit.

19. The method of claim 9 wherein the identifying of which phonetic units are least accurately modeled is based on analysis of words misrecognized during recognition.

20. The method of claim 9 wherein the phonetic unit is a phoneme.

21. The method of claim 9 wherein the phonetic unit is a senone.

22. The method of claim 9 wherein the phonetic unit is a context-dependent phoneme state.

23. The method of claim 9 wherein the phonetic unit is a word.

24. The method of claim 9 wherein the feature vectors are vector quantized.

25. The method of claim 9 including training the speech recognition system based on the selected words.

26. The method of claim 25 wherein the training includes generating an error profile indicating a probability for each feature vector for each phonetic unit that reflects differences between the models and words spoken during training.

27. The method of claim 26 wherein the probabilities of the error profile are factored into the probabilities of the model during speech recognition.

28. A method in a computer system for tutoring a speaker on pronunciation of words, each word being phonetically represented by phonetic units, each spoken utterance being represented by a sequence of feature vectors, each phonetic unit having a model for generating a probability that various sequences of feature vectors corresponds to the modeled phonetic unit, the method comprising:

collecting a plurality of spoken utterances from the speaker for which the corresponding word is known;

for each collected utterance,

converting the collected utterance into a sequence of feature vectors; and

aligning each feature vector in the sequence of feature vectors with a phonetic unit of the known word to which the collected utterance corresponds based on the model of the phonetic units of the known word;

identifying, from the feature vectors aligned with each phonetic unit, a phonetic unit that is inaccurately spoken by the speaker; and

selecting words that contain the identified phonetic units as words for tutoring the speaker.

29. The method of claim 28 including presenting the selected words to the speaker.

30. The method of claim 29 including receiving a speech utterance corresponding to each spoken word and evaluating the accuracy of the received speech utterances.

31. The method of claim 28 wherein identifying which phonetic units are least accurately modeled includes calculating a frame accuracy measure based on the aligned phonetic unit and feature vector, and calculating a combined accuracy measure for each unique phonetic unit by combining the frame accuracy measures that are based on that phonetic unit.

32. The method of claim 31 wherein the frame accuracy measure is a ratio of the probability that the feature vector is contained within the phonetic unit with which it is aligned to a highest probability that the feature vector is contained within any phonetic unit.

33. The method of claim 31 wherein the frame accuracy measure is a difference of the probability that the feature vector is contained within the phonetic unit with which it is aligned to a highest probability that the feature vector is contained within any phonetic unit.

34. The method of claim 31 wherein the frame accuracy measure is a rank of each phonetic unit based on the probability that those vector frames aligned with the phonetic unit will be spoken as part of the phonetic unit.

35. The method of claim 34 wherein the probability that those feature vectors aligned with the phonetic unit will be spoken as part of the phonetic unit is an acoustic model probability.

36. The method of claim 34 wherein the combined accuracy measure is an average of the probability that each feature vector that is aligned with that phonetic unit is contained within that phonetic unit.

37. The method of claim 36 wherein the phonetic unit is a phoneme.

38. The method of claim 36 wherein the phonetic unit is a senone.

39. The method of claim 36 wherein the phonetic unit is a context-dependent phoneme state.

40. The method of claim 36 wherein the phonetic unit is a word.

41. The method of claim 34 wherein the combined accuracy measure is a maximum of the probabilities that each feature vector that is aligned with that phonetic unit is contained within that phonetic unit.

42. The method of claim 34 wherein the combined accuracy measure is a minimum of the probabilities that each feature vector that is aligned with that phonetic unit is contained within that phonetic unit.

43. The method of claim 34 wherein the combined accuracy measure is a weighted average of the probabilities that each feature vector that is aligned with that phonetic unit is contained within that phonetic unit.

44. The method of claim 28 wherein the phonetic unit is a phoneme.

45. The method of claim 28 wherein the phonetic unit is a senone.

46. The method of claim 28 wherein the feature vectors are vector quantized.

47. A method in a computer system for selecting words for training a speech recognition system, the speech recognition system for recognizing a plurality of words, each word being spoken with phonetic units, the method comprising:

receiving a plurality of spoken utterances for which the corresponding word is determined;

for each phonetic unit of each of the determined words for which a spoken utterance is received, determining a context-dependent accuracy of the speech recognition system at recognizing the phonetic unit within the determined word;

for each phonetic unit, determining a context-independent accuracy of the speech recognition system at recognizing the phonetic unit based on the context-dependent accuracies; and

selecting words that contain phonetic units that are determined to have a lowest context-independent accuracy for training the speech recognition system.

48. The method of claim 47 wherein the words are selected based on context-dependent accuracy for training the speech recognition system.

49. The method of claim 47 wherein the speech recognition system has a model for each phonetic unit indicating a probability that a sequence of quantized vectors corresponds to the phonetic unit and wherein the determining of the context-dependent accuracy includes:

for each received utterance,

converting the spoken utterance into a sequence of quantized vectors;

aligning each quantized vector in the sequence with a phonetic unit of the determined word based on the model for the phonetic units of the determined word; and

identifying a probability that each aligned quantized vector is spoken as part of the phonetic unit with which it is aligned, wherein the identified probability is used to determine the context-dependent accuracy.

50. The method of claim 47 including presenting the selected words to a speaker for training.

51. The method of claim 50 wherein each word has a language model probability indicating the probability that the word will be spoken and wherein the selected words are presented to the speaker in an order based on the language model probability of the words.

52. The method of claim 47 wherein the selecting of word includes selecting words with more than one phonetic unit with a context-independent accuracy that is determined to be low.

53. The method of claim 52 including presenting the selected words to a speaker for training.

54. The method of claim 53 wherein each word has a language model probability indicating the probability that the word will be spoken and wherein the selected words are presented to the speaker in an order based on the language model probability of the selected words.

55. The method of claim 47 wherein the receiving of the plurality of spoken utterances occurs during a recognition process and wherein the recognition process periodically asks a speaker to train on the selected words.

56. The method of claim 55 wherein the training and recognizing are repeatedly performed.

57. The method of claim 47 wherein the receiving of the plurality of spoken utterances occurs during a recognition process and wherein when the recognition process identifies when the recognition process is incorrectly recognizing a certain portion of spoken utterances, and automatically prompting for training on the selected words.

58. The method of claim 47 wherein the receiving of the plurality of spoken utterances occurs during a recognition process and wherein the received spoken utterances were misrecognized during the recognition process.

59. A computer-readable medium containing instructions for causing a computer system to tutor a speaker on pronunciation of words, each word being spoken with phonetic units, by performing the steps of:

receiving from the speaker a plurality of spoken utterances for which the corresponding word is known;

identifying from the spoken utterances which phonetic units are inaccurately spoken by the speaker; and

selecting words that contain an identified phonetic unit for tutoring the speaker.

60. The computer-readable medium of claim 59 including the step of presenting the selected words to the speaker.

61. The computer-readable medium of claim 60 including the step of receiving a speech utterance corresponding to each spoken word presented to the speaker and evaluating the accuracy of the received speech utterances.

62. The computer-readable medium of claim 59 wherein the phonetic unit is a phoneme.

63. The computer-readable medium of claim 59 wherein the phonetic unit is a senone.

64. A computer-readable medium containing instructions for causing a computer system to select words for training a speech recognition system, the speech recognition system for recognizing a plurality of words, the speech recognition system having a indication of phonetic units that compose each word, the speech recognition system having a model for each phonetic unit, the speech recognition system representing speech as a sequence of feature vectors, each model for indicating a probability that each possible sequence of feature vectors corresponds to the modeled phonetic unit, by performing the steps of:

receiving a plurality of spoken utterances for which the corresponding word is determined;

for each collected utterance,

converting the collected utterance to a sequence of feature vectors; and

aligning each feature vector in the sequence of feature vectors with a phonetic unit of the determined word to which the collected utterance corresponds based on the model of the phonetic units of the determined word;

identifying from the feature vectors aligned with each phonetic unit which phonetic units are least accurately modeled; and

selecting words that contain the identified phonetic units as words for training the speech recognition system.

65. The computer-readable medium of claim 64 wherein the identifying which phonetic units are least accurately modeled includes the step of calculating a rank of each phonetic unit based on the probability that those vectors aligned with the phonetic unit will be spoken as part of the phonetic unit.

66. The computer-readable medium of claim 65 wherein the probability that those feature vectors aligned with the phonetic unit will be spoken as part of the phonetic unit is an acoustic model probability.

67. The computer-readable medium of claim 64 including the step of training the speech recognition system based on the selected words.

68. The computer-readable medium of claim 67 wherein the training includes the step of generating an error profile indicating a probability for each feature vector for each phonetic unit that reflects differences between the models and words spoken during training.

69. The computer-readable medium of claim 68 wherein the probabilities of the error profile are factored into the probabilities of the model during speech recognition.

70. A method in a computer system for dynamically selecting words for training a speech recognition system, each word comprising phonetic units, the method comprising:

collecting a plurality of spoken utterances for which the corresponding word is known;

identifying from the spoken utterances which phonetic units are inaccurately modeled by the speech recognition system; and

selecting words that contain an identified phonetic unit for training of the speech recognition system.

71. The method of claim 70 wherein the phonetic unit is a phoneme.

72. The method of claim 70 wherein the phonetic unit is a senone.

73. The method of claim 70 wherein the phonetic unit is a context-dependent phoneme state.

74. The method of claim 70 wherein the phonetic unit is a word.

75. The method of claim 70 wherein the speech recognition system has a model for each phonetic unit indicating a probability that a sequence of quantized vectors corresponds to the phonetic unit and wherein the identifying includes:

for each collected utterance,

converting the collected utterance into a sequence of quantized vectors;

aligning each quantized vector in the sequence with a phonetic unit of the known word based on the model for the phonetic units of the determined word; and

identifying a probability that each aligned quantized vector is spoken as part of the phonetic unit with which it is aligned.

76. The method of claim 70 including presenting the selected words to a speaker for training.

77. The method of claim 76 wherein each word has a language model probability indicating the probability that the word will be spoken and wherein the selected words are presented to the speaker in an order based on the language model probability of the words.

78. The method of claim 70 wherein the selecting of words includes selecting words with more than one phonetic unit that is inaccurately modeled.

79. The method of claim 78 including presenting the selected words to a speaker for training.

80. The method of claim 79 wherein each word has a language model probability indicating the probability that the word will be spoken and wherein the selected words are presented to the speaker in an order based on the language model probability of the selected words.

81. The method of claim 70 wherein the receiving of the plurality of spoken utterances occurs during a recognition process and wherein the recognition process periodically asks a speaker to train on the selected words.

82. The method of claim 81 wherein the training and recognizing are repeatedly performed.

83. The method of claim 70 wherein the receiving of the plurality of spoken utterances occurs during a recognition process and wherein when the recognition process identifies when the recognition process is incorrectly recognizing a certain portion of spoken utterances, training on the selected words.

84. The method of claim 70 wherein the receiving of the plurality of spoken utterances occurs during a recognition process and wherein the receive spoken utterance were misrecognized during the recognition process.

85. The method of claim 70 wherein the identifying of which phonetic units are inaccurately modeled includes:

aligning the feature vectors of the spoken utterance with the phonetic units;

calculating a frame accuracy measure for each feature vector based on the aligned phonetic unit and feature vector; and

calculating a combined accuracy measure for each unique phonetic unit by combining the frame accuracy measures that are based on that phonetic unit.

86. The method of claim 85 wherein the frame accuracy measure is a ratio of the probability that the spoken utterance of that frame is contained within the phonetic unit with which it is aligned to a highest probability that the spoken utterance of that frame is contained with any phonetic unit.

87. The method of claim 85 wherein the frame accuracy measure is a difference of the probability that the spoken utterance of that frame is contained within the phonetic unit with which it is aligned to a highest probability that the spoken utterance of that frame is contained with any phonetic unit.

88. The method of claim 85 wherein the frame accuracy measure is an average of the probability that the spoken utterance of each frame that is aligned with that phonetic unit is contained within that phonetic unit.

89. The method of claim 85 wherein the frame accuracy measure is an average of a maximum of the probabilities that the spoken utterance of each frame that is aligned with that phonetic unit is contained within that phonetic unit.

90. The method of claim 85 wherein the frame accuracy measure is a minimum of the probabilities that the spoken utterance of each frame that is aligned with that phonetic unit is contained within that phonetic unit.

91. The method of claim 85 wherein the frame accuracy measure is a sum of a weighted average of the probabilities that the spoken utterance of each frame that is aligned with that phonetic unit is contained within that phonetic unit.

92. The method of claim 70 wherein the identifying of which phonetic units are inaccurately modeled includes counting the number of times that a phonetic unit is recognized in a misrecognized word during recognition.

93. The method of claim 70 wherein the identifying of which phonetic units are inaccurately modeled includes counting the number of times that a phonetic unit was not recognized in a correct word during recognition.

94. The method of claim 70 wherein the identifying of which phonetic units are inaccurately modeled is based on differences between probability values of correct and incorrect phoneme models.

95. The method of claim 70 wherein the phonetic unit is a context-dependent phoneme.

96. The method of claim 70 wherein the phonetic unit is a word.

97. The method of claim 70 wherein the identifying of which phonetic units are inaccurately modeled includes aligning a sequence of frames of the spoken utterances with phonetic units and calculating a rank of a phonetic unit based on a probability that the aligned sequences of frames is contained within that phonetic unit.

98. A computer system for dynamically selecting words for training a speech recognition system, each word comprising phonetic units, comprising:

a sample collecting component that collects a plurality of spoken utterances for which the corresponding word is known and converts the spoken utterances to codewords;

an aligning component that aligns codewords with phonetic units of each word;

a phonetic unit ranking component that identifies from the spoken utterances which phonetic units are inaccurately modeled by the speech recognition system based on the aligned codewords; and

a word selection component that selects words that contain a phonetic unit identified by the phonetic unit ranking component for training of the speech recognition system.

99. The computer system of claim 98 wherein the speech recognition system has a model for each phonetic unit indicating a probability that a sequence of quantized vectors corresponds to the phonetic unit, wherein the aligning component aligns each codeword with a phonetic unit of the determined word based on the model for the phonetic units of the determined word, and wherein the phonetic unit ranking component identifies a probability that each aligned codeword is spoken as part of the phonetic unit with which it is aligned.

100. The computer system of claim 98 wherein each word has a language model probability indicating the probability that the word will be spoken, and including a presenting component for presenting selected words to the speaker in an order based on the language model probability of the words.

101. The computer system of claim 98 wherein the word selection component selects words with more than one phonetic unit that is inaccurately modeled.

102. A method in a computer recognition system for evaluating accuracy of the recognition system at recognizing words, each word comprising phonetic units, the method comprising:

collecting a plurality of spoken utterances for which the corresponding word is known; and

identifying an accuracy of each phonetic unit by aligning frames of the spoken utterances with phonetic units and calculating a frame accuracy measure for each frame based on a probability that the spoken utterance of that frame is contained within the phonetic unit with which it is aligned.

103. The method of claim 102 wherein the frame accuracy measure is a ratio of the probability that the spoken utterance of that frame is contained within the phonetic unit with which it is aligned to a highest probability that the spoken utterance of that frame is contained within any phonetic unit.

104. The method of claim 102 wherein the frame accuracy measure is a difference of the probability that the spoken utterance of that frame is contained within the phonetic unit with which it is aligned to a highest probability that the spoken utterance of that frame is contained within any phonetic unit.

105. The method of claim 102 wherein the frame accuracy measure is an average of the probability that the spoken utterance of each frame that is aligned with that phonetic unit is contained within that phonetic unit.

106. The method of claim 102 wherein the frame accuracy measure is a maximum of the probabilities that the spoken utterance of each frame that is aligned with that phonetic unit is contained within that phonetic unit.

107. The method of claim 102 wherein the frame accuracy measure is a minimum of the probabilities that the spoken utterance of each frame that is aligned with that phonetic unit is contained within that phonetic unit.

108. The method of claim 102 wherein the frame accuracy measure is a weighted average of the probabilities that the spoken utterance of each frame that is aligned with that phonetic unit is contained within that phonetic unit.

109. The method of claim 102 wherein the calculating includes counting the number of times that a phonetic unit is not recognized in a misrecognized word during recognition.

110. The method of claim 102 wherein the calculating includes counting the number of times that a phonetic unit was not recognized in a correct word during recognition.

111. The method of claim 102 wherein the phonetic unit is a context-dependent phoneme.

112. The method of claim 102 wherein the phonetic unit is a word.

113. The method of claim 102 wherein the phonetic unit is a state of a phoneme.

114. The method of claim 102 wherein the phonetic unit is a senone.

115. A method in a computer recognition system for evaluating accuracy of the recognition system at recognizing words, each word comprising phonetic units, the method comprising:

collecting a plurality of spoken utterances for which the corresponding word is known; and

identifying an accuracy of each phonetic unit by aligning frames of the spoken utterances with phonetic units and counting the number of times that a phonetic unit was not recognized in a correct word during recognition.

116. A method in a computer recognition system for evaluating accuracy of the recognition system at recognizing words, each word comprising phonetic units, the method comprising:

collecting a plurality of spoken utterances for which the corresponding word is known; and

identifying an accuracy of each phonetic unit by aligning frames of the spoken utterances with phonetic units and counting the number of times that a phonetic unit is not recognized in a misrecognized word during recognition.