Continuous speech pattern recognizer

• US 4,400,788 A
• Filed: 03/27/1981
• Issued: 08/23/1983
• Est. Priority Date: 03/27/1981
• Status: Expired due to Term

• Current Assignee: Bell Telephone Laboratories, Inc. (Nokia Corporation)
• Original Assignee: Bell Telephone Laboratories, Inc. (Nokia Corporation)
• Inventors: Rabiner, Lawrence R., Pirz, Frank C., Myers, Cory S.
• Primary Examiner(s): KEMENY, EMANUEL
• Application Number:
  US06/248,570
• Time in Patent Office: 879 Days
• Field of Search: 179/1 SB, 179/1 SD, 364/513, 364/156, 340/146.3 AQ, 340/146.3 WD
• US Class Current: 704/241
• CPC Class Codes: G10L 15/00 Speech recognition G10L17/00 takes precedence
• CPC Class Codes: G10L 15/12 using dynamic programming techniques, e.g. dynamic time warping [DTW]

Abstract

This speech recognizer concatenates a string of reference isolated-words for comparison with the unknown string of connected-words. The invention includes a level-building (LB) algorithm, "level" implying a location in a sequence of words. A constrained endpoint dynamic-time-warp algorithm, in which the slope of the warping function is restricted between 1/2 and 2, is used to find the best alignment between an unknown continuous-word test pattern, and a concatenated sequence of L reference patterns. Properties of the LB algorithm include: modification of the references; back-track decision logic; heuristic selection of multiple candidates, and syntax constraints. As a result, the processing required is less than two-level dynamic-program-matching and sampling algorithms.

Claims

1. Apparatus for recognizing a speech pattern as a string of predetermined reference words comprising:

means for storing a set of signals representative of the time frame sequence of acoustic features of each reference word, said time frame sequence having a beginning frame and an ending frame;

means for producing a set of signals representative of the time frame sequence of acoustic features of said speech pattern, means responsive to the speech pattern acoustic feature signals and the reference word acoustic feature signals for generating a plurality of reference word strings, and means for identifying the speech pattern as one of the generated reference word strings, characterized in that said reference word string generating means comprises;

means for generating a set of signals for identifying successive reference word levels, means for assigning a segment of said speech pattern to each successive level, means operative in each successive level for time registering the level speech pattern segment feature signals with the reference word feature signals to produce level time registration speech pattern segment endframe signals and time registration correspondence signals for said reference words, and means responsive to the time registration endframe and time registration correspondence signals of the levels for selecting reference word strings.

2. Apparatus for recognizing a speech pattern as a string of predetermined reference words according to claim 1 further characterized in that said time registering means comprises:

means responsive to the time registration speech pattern endframes of the immediately preceding level for restricting the range of time registration starting frames for the current level, and means responsive to said time registration correspondence signals, and the time registration speech pattern endframe signals for storing the best correspondence signal for each time registration endframe, a signal representative of the best corresponding reference word for each endframe and a signal representative of the time registration speech pattern starting frame corresponding to the best reference word for each endframe.

3. Apparatus for recognizing a speech pattern as a string of predetermined reference words according to claim 2 further characterized in that said speech pattern starting frame restricting means comprises:

means responsive to the time registration best corresponding signals of the preceding level for selecting the minimum preceding level best correspondence signal, and means responsive to the preceding level time registration best correspondence signals and said selected minimum best correspondence signal for selecting time registration speech pattern starting frames of the current level.

4. Apparatus for recognizing a speech pattern as a string of predetermined words according to claim 3 further characterized in that said reference word string generating means further comprises:

means responsive to the speech pattern feature signals for generating a signal representative of the speech pattern final frame;

means operative at each level responsive to said minimum best correspondence signal for generating a signal representative of the speech pattern segment endframe having the minimum best correspondence signal; and

means responsive to the minimum best correspondence endframe signal being within a prescribed range of the speech pattern final frame signal for initiating the operation of said reference word string selecting means.

5. Apparatus for recognizing a speech pattern as a string of predetermined reference words according to claim 4 further characterized in that said reference word string selecting means comprises:

means responsive to the stored time registration speech pattern segment endframe signals, starting frame signals and reference word identification signals and said speech pattern final frame signal for producing signals representative of each reference word string having a time registration speech pattern endframe within said prescribed range of said speech pattern final frame.

6. Apparatus for recognizing a speech pattern as a string of predetermined reference words according to claim 1 further characterized in that said time registering means comprises:

means for selecting a range of reference word frames for time registration with each level speech pattern segment frame including means responsive to the current reference word range lower limit frame being less that a first predetermined number for extending the lower reference word range limit frame to the reference word beginning frame, and means responsive to the current reference word range upper limit frame being greater than a second predetermined number for extending the upper reference word range limit frame to the reference word ending frame whereby coarticulation effects are reduced.

7. Apparatus for recognizing a speech pattern as a string of predetermined reference words according to claim 1 further characterized in that said time registering means comprises:

means responsive to the feature signals of said reference word and said level speech segment feature signals for selecting a range of reference word frames for time registration with each level speech pattern segment frame, and means responsive to the reference word range lower limit frame being equal to or greater than the reference word ending frame for generating a level speech pattern segment time registration endframe.

8. Apparatus for recognizing a speech pattern as a string of predetermined reference words according to claim 1 further characterized in that said time registering means comprises:

means operative in each reference word time registration of the current level responsive to the reference word feature signal and the speech pattern segment feature signals for selecting a range of reference word frames for each speech pattern segment frame, and means operative at each speech pattern segment frame responsive to the reference word registration lower limit frame being equal to or exceeding the reference word ending frame for terminating the time registration of said reference word.

9. Apparatus for recognizing a speech pattern as a string of predetermined reference words according to claim 8 further characterized in that said time registering means comprises:

means operative in each level reference word time registration responsive to the reference word feature signals and the level speech pattern segment feature signals for generating a time registration correspondence signal for each level speech pattern segment frame, means responsive to the level speech pattern segment frame for generating a maximum allowable correspondence signal for said speech pattern segment frame and means responsive to said time registration correspondence signal of a speech pattern segment frame exceeding the maximum allowable correspondence signal for said speech pattern segment frame for terminating the time registration of said level reference word.

10. Apparatus for recognizing a speech pattern as a string of predetermined reference words according to claims 1, 2, 3, 4, 5, 6, 7, 8, or 9 further characterized in that said reference word generating means further comprises:

means responsive to said level identification signals for selecting a prescribed set of reference words for time registration in each reference word level.

45. Apparatus for recognizing a speech pattern as a string of predetermined reference words according to claim 10 further characterized in that said speech pattern and reference word acoustic feature signals are spectrally derived feature signals.

46. Apparatus for recognizing a speech pattern as a string of predetermined reference words according to claim 10 further characterized in that said speech pattern and reference word acoustic feature signals are linear prediction feature signals.

47. Apparatus for recognizing a speech pattern as a string of predetermined reference words according to claim 10 further characterized in that said speech pattern and reference word acoustic feature signals are Markov derived feature signals.

11. A method for recognizing a speech pattern as a string of predetermined reference words comprising the steps of:

storing a set of signals representative of the time frame sequence of acoustic features of each reference word, said sequence having a beginning frame and an ending frame;

producing a set of signals representative of the time frame sequence of acoustic features of the speech pattern;

generating at least one reference word string responsive to the speech pattern acoustic feature signals and the reference word acoustic feature signals; and

identifying the speech pattern as one of the generated reference word strings characterized in that said reference word string generation comprises;

producing a set of signals identifying successive reference word levels;

assigning a segment of the speech pattern to each successive level;

for each level, time registering the level speech pattern segment feature signals with the feature signals of the reference words to produce level time registration speech pattern segment endframe signals and time registration correspondence signals for said reference words, and selecting reference word strings responsive to the time registration speech pattern endframe and time registration correspondence signals of the levels.

12. A method for recognizing a speech pattern as a string of predetermined reference words according to claim 11 further characterized in that said time registering comprises:

restricting the range of time registration starting frames for the current level responsive to the time registration speech segment endframes of the immediately preceding level; and

storing for each time registration speech segment endframe the best correspondence signal for said endframe, a signal representative of the best correspondence signal reference word for said endframe, and a signal representative of the time registration speech pattern starting frame corresponding to the best reference word for said endframe.

13. A method for recognizing a speech pattern as a string of predetermined reference words according to claim 12 further characterized in that the restriction of said level starting frames comprises:

detecting the minimum of the preceding level best correspondence signals responsive to the time registration best correspondence signals of the preceding level, and selecting time registration speech segment starting frames for the current level responsive to the best correspondence signals of the preceding level and the detected minimum best correspondence signal of the preceding level.

14. A method for recognizing a speech pattern as a string of predetermined reference words according to claim 13 further characterized in that said reference word string generating step further comprises:

generating a signal representative of the speech pattern final frame responsive to the speech pattern feature signals;

at each level, generating a signal representative of the speech pattern segment endframe having the minimum best correspondence signal; and

initiating the reference word string selection responsive to the minimum best correspondence endframe signal being within a prescribed range of the speech pattern final frame signal.

15. A method for recognizing a speech pattern as a string of predetermined reference words according to claim 14 further characterized in that said reference word selecting step comprises:

producing signals representative of each reference word string having a time registration speech pattern endframe within a prescribed range of said speech pattern final frame responsive to the stored time registration speech pattern endframe signals, starting frame signals and reference word identification signals.

16. A method for recognizing a speech pattern as a string of predetermined reference words according to claim 11 further characterized in that said time registering comprises:

selecting a range of reference word frames for registration with each level speech pattern segment frame responsive to the reference word feature signals and the speech pattern segment feature signals;

including extending the reference word range lower limit frame to the reference word beginning frame responsive to the selected reference word range lower limit frame being less than a first predetermined number and extending the selected reference word range upper limit frame to the reference word ending frame responsive to the selected reference word range upper limit frame being greater than a second predetermined number.

17. A method for recognizing a speech pattern as a string of predetermined reference words according to claim 11 further characterized in that said time registering comprises:

selecting a range of reference word frames for registration with each level speech pattern segment frame responsive to the reference word feature signals and the level speech segment feature signals and generating level speech pattern segment time registration endframes responsive to the selected reference word range upper limit frame being equal to or exceeding the reference word ending frame.

18. A method for recognizing a speech pattern as a string of predetermined reference words according to claim 11 further characterized in that said time registering comprises:

selecting a range of reference word frames for each speech pattern segment frame at each level responsive to the reference word feature signals and the speech pattern feature signals and terminating the time registration of said level reference word responsive to the selected reference word registration range lower limit frame being equal to or exceeding the reference word ending frame.

19. A method for recognizing a speech pattern as a string of predetermined reference words according to claim 18 further characterized in that said time registering comprises:

generating a time registration correspondence signal for each level speech pattern segment frame responsive to the reference word feature signals and the level speech pattern feature signals;

generating a maximum allowable correspondence signal for said speech pattern segment frame responsive to the level speech pattern segment frame, and terminating the time registration of said level reference word responsive to the time registration correspondence signal of a speech pattern segment frame exceeding the maximum allowable correspondence signal for said frame.

20. A method for recognizing a speech pattern as a string of predetermined reference words according to claims 11, 12, 13, 14, 15, 16, 17, 18 or 19 further characterized in that said reference word string generation further comprises selecting a prescribed set of reference words for time registration in each reference word level responsive to the level identification signals.

48. A method for recognizing a speech pattern as a string of predetermined reference words according to claim 20 further characterized in that said speech pattern and reference word acoustic feature signals are spectrally derived feature signals.

49. A method for recognizing a speech pattern as a string of predetermined reference words according to claim 20 further characterized in that said speech pattern and reference word acoustic feature signals are linear prediction feature signals.

50. A method for recognizing a speech pattern as a string of predetermined reference words according to claim 20 further characterized in that said speech pattern and reference word acoustic feature signals are Markov derived feature signals.

21. A speech analyzer for recognizing a speech pattern as a string of predetermined reference words comprising:

means for storing a set of signals representative of the time frame sequence of acoustic features of each reference word from a beginning frame to an ending frame, means for producing a set of signals representative of the time frame sequence of acoustic features of the speech pattern from a beginning frame to a final frame;

means responsive to the feature signals of said reference words and said speech pattern for generating at least one reference word string; and

means for identifying the speech pattern as one of said generated reference word strings, said reference word string generating means comprising;

means for generating a set of signals to identify successive levels of said reference words;

means for assigning a segment of said speech pattern to each successive level, means operative at each successive level responsive to the reference word and speech pattern segment feature signals for dynamically time warping feature signals of each reference word with the feature signals of the speech pattern segment assigned to the level to produce signals representative of time registration path speech pattern endframes for said reference words and signals representative of the correspondence of the reference word and speech pattern segment feature signals on said time registration path; and

means responsive to the time registration path speech pattern endframe and correspondence signals of the levels for selecting strings of reference words.

22. A speech analyzer according to claim 21 wherein said dynamic time warping means further comprises:

means operative at each successive level responsive to said reference word time registration speech pattern endframe and correspondence signal for selecting for each speech pattern endframe signal the time registration path having the minimum correspondence signal to said enframe; and

means for storing, for each speech pattern endframe signal, a signal identifying the reference word for said minimum correspondence signal time registration path, the minimum correspondence signal, and a signal representative of the starting frame of the minimum correspondence time registration path.

23. A speech analyzer according to claim 22 wherein said dynamic time warping means further comprises:

means responsive to the speech pattern endframe signals of the preceding level for restricting the speech pattern starting frames of the current level time registration paths to the speech pattern endframes of the preceding level.

24. A speech analyzer according to claim 23 wherein said current level starting frame restricting means comprises:

means for detecting the smallest of the minimum speech pattern endframe correspondence signals of a level and means responsive to said detected smallest minimum correspondence signal for limiting the range of speech pattern starting frames for the succeeding level.

25. A speech analyzer according to claim 24 further comprising means for generating a signal representative of the speech pattern final frames, means operative at each level responsive to said endframe smallest minimum correspondence signal for generating a signal representative of the endframe having the smallest minimum correspondence signals;

means responsive to said smallest minimum correspondence signal endframe being within a prescribed range of said speech pattern final frame for initiating the operation of said reference word storing selecting means.

26. A speech analyzer according to claim 25 wherein said reference string selecting means comprises:

means responsive to the stored minimum correspondence signals, starting frame signals, and reference word identification signals stored for endframes of said levels for producing signals representative of each reference word string having a time registration endframe within a prescribed range of the speech pattern final frame.

27. A speech analyzer according to claim 24 wherein said starting frame range limiting means comprises:

means responsive to a multiple of said smallest minimum correspondence signal for selecting the highest speech pattern endframe of the level having a minimum correspondence signal less than said multiple of said smallest minimum correspondence signal as the range upper limit speech pattern starting frame of the next level and selecting the lowest speech pattern endframe of the level having a minimum correspondence signal less than said multiple of said smallest minimum correspondence signal as the range lower limit starting frame of the next level.

28. A speech analyzer according to claim 21 wherein said dynamic time warping means further comprises:

means responsive to the reference word and speech pattern segment feature signals for selecting a range of reference word frames for registration with each speech pattern segment frame including means responsive to the reference word range lower limit frame being less than a first predetermined number for extending the reference word lower limit frame to the reference word beginning frame and means responsive to the reference word range upper limit frame being greater than a second predetermined number for extending the reference word range upper limit frame to the reference word ending frame.

29. A speech analyzer according to claim 28 wherein said dynamic time warping means further comprises:

means responsive to said reference word range upper limit frame being equal to or exceeding said reference word ending frame for detecting the level speech segment endframes of said reference words.

30. A speech analyzer according to claim 29 wherein said dynamic time warping means further comprises:

means responsive to the selected reference word range lower limit frame being equal to or exceeding the reference word ending frame for terminating the dynamic time warping of said reference word.

31. A speech analyzer according to claim 30 wherein said dynamic time warping means further comprises:

means responsive to the feature signals of selected reference word frames and speech pattern segment frame feature signals for generating a signal representative of the correspondence between the feature signals of the selected reference word frames and the speech pattern segment frame feature signals for each speech pattern segment frame, means for storing a maximum allowable correspondence signal for each speech pattern segment frame, and means responsive to said frame correspondence signal exceeding said frame maximum allowable correspondence signal for terminating the dynamic time warping of said level reference word.

32. A speech analyzer according to claims 21, 22, 23, 24, 25, 26, 27, 28, 29, 30 or 31 wherein said reference word generating means further comprises means responsive to the level identification signals for selecting a prescribed set of reference words for dynamic time warping in each successive level.

33. A method for recognizing a speech pattern as a string of predetermined reference words comprising the steps of:

storing a set of signals representative of the time frame sequence of acoustic features of each reference word from a beginning frame to an ending frame;

producing a set of signals representative of the time frame sequence of acoustic features of the speech from a beginning frame to a final frame;

generating at least one reference word string responsive to the feature signals of the reference words and the feature signals of the speech pattern; and

identifying the speech pattern as one of said generated reference word strings;

the reference word string generating step comprising;

generating a set of signals to identify successive levels of said reference words, assigning a segment of the speech pattern to each successive reference word level, at each reference word level dynamically time warping the feature signals of each reference word with the feature signals of the speech pattern segment assigned to the level to produce signals representative of time registration path speech pattern endframes for said reference word and signals representative of the correspondence of the reference word and speech pattern segment feature signals along the time registration paths, and selecting strings of reference words responsive to the time registration path speech pattern endframe and correspondence signals of the levels.

34. A method for recognizing a speech pattern as a string of predetermined reference words according to claim 33 wherein said dynamic time warping step further comprises:

selecting for each speech pattern endframe signal the time registration path having the minimum correspondence signal at said endframe responsive to the reference word time registration endframe and correspondence signals of each level;

storing for each speech pattern endframe signal, a signal identifying the reference word with the minimum correspondence signal time registration path, the minimum correspondence signal and a signal representative of the starting frame of the minimum correspondence time registration path.

35. A method for recognizing a speech pattern as a string of predetermined reference words according to claim 34 wherein said dynamic time warping step further comprises:

restricting the speech pattern starting frames of the current level time registration paths responsive to the speech pattern endframe signals of the preceding level.

36. A method for recognizing a speech pattern as a string of predetermined reference words according to claim 35 wherein the current level speech pattern starting frame restricting step comprises:

detecting the smallest of the minimum speech pattern endframe correspondence signals of a level and limiting the range of speech pattern starting frames for the succeeding level responsive to said detected smallest minimum correspondence signal and the minimum speech pattern endframe correspondence signals of the level.

37. A method for recognizing a speech pattern as a string of predetermined reference words according to claim 36 wherein said reference word string generation step further comprises:

generating a signal representative of the speech pattern final frame responsive to the speech pattern feature signals;

at each level, generating a signal representative of the time registration path speech pattern segment endframe having the smallest minimum correspondence signal responsive to the minimum corespondence signals; and

initiating the reference word selection responsive to the smallest minimum correspondence endframe signal being within a prescribed range of the speech pattern final frame signal.

38. A method for recognizing a speech pattern as a string of predetermined reference words according to claim 37 wherein said reference word string selecting step comprises:

producing signals representative of each reference word string having a time registration path speech pattern endframe within a prescribed range of said speech pattern final frame responsive to the stored time registration speech pattern endframe signals, starting frame signals, and reference word identification signals.

39. A method for recognizing a speech pattern as a string of predetermined reference words according to claim 36 wherein the speech pattern starting frame range limiting step comprises:

selecting the level highest endframe having a minimum correspondence signal less than a multiple of the smallest minimum correspondence signal as the range upper limit starting frame of the next level and selecting the lowest endframe of the level having a minimum correspondence signal less than said multiple of the smallest minimum correspondence signal as the range lower limit starting frame of the next level.

40. A method for recognizing a speech pattern as a string of predetermined reference words according to claim 33 wherein the dynamic time warping step further comprises:

selecting a range of reference word frames for time warping with each speech pattern segment frame responsive to the reference word and level speech pattern segment feature signals including extending the reference word range lower limit frame to the reference word beginning frame responsive to the reference word range lower limit frame being less than a first predetermined number, and extending the reference word range upper limit frame to the reference word ending frame responsive to the reference word range upper limit frame being greater than a second predetermined number.

41. A method for recognizing a speech pattern as a string of predetermined reference words according to claim 40 wherein said dynamic time warping step further comprises:

detecting the level speech segment endframes for said reference words responsive to the reference word range upper limit frame being equal to or exceeding the reference word ending frame.

42. A method for recognizing a speech pattern as a string of predetermined reference words according to claim 41 wherein the dynamic time warping step further comprises:

for each reference word dynamic time warping of the current level terminating the dynamic time warping of said reference word responsive to the selected reference word range lower limit frame being equal to or exceeding the reference word ending frame.

43. A method for recognizing a speech pattern as a string of predetermined reference words according to claim 42 wherein the dynamic time warping step further comprises:

generating a signal representative of the correspondence between the selected reference word range frame feature signals and the speech pattern segment feature signals for each speech pattern segment frame responsive to the feature signals of the selected reference word range frames and the level speech segment frame feature signals, storing a maximum allowable correspondence signal for each level speech pattern segment frame responsive to the level speech pattern segment frame and terminating the dynamic time warping of the level reference word responsive to said level speech pattern segment frame correspondence signal exceeding the level speech pattern segment frame maximum allowable correspondence signal.

44. A method for recognizing a speech pattern as a string of predetermined reference words according to claims 33, 34, 35, 36, 37, 38, 39, 40, 41, 42 or 43 wherein said reference word generating step further comprises:

selecting a prescribed set of said reference words for dynamic time warping in each successive level responsive to the level identification signals.