Method and apparatus for generating speech pattern templates

• US 4,454,586 A
• Filed: 11/19/1981
• Issued: 06/12/1984
• Est. Priority Date: 11/19/1981
• Status: Expired due to Term

• Current Assignee: Bell Telephone Laboratories, Inc. (Nokia Corporation)
• Original Assignee: AT&T, Inc.
• Inventors: Rabiner, Lawrence R., Wilpon, Jay G., Pirz, Frank C.
• Primary Examiner(s): KEMENY, EMANUEL
• Application Number:
  US06/322,748
• Time in Patent Office: 936 Days
• Field of Search: 381/41, 381/43, 381/42, 381/44, 381/45, 381/51-53, 364/513.5
• US Class Current: 704/245
• CPC Class Codes: G10L 15/12 using dynamic programming techniques, e.g. dynamic time warping [DTW]

Abstract

A system for generating speech pattern templates for use with either speech recognition or speech synthesis. Reference demisyllable templates are first generated from a reference first speaker using both manual and automatic analysis. The analysis for a second speaker is simplified and automated by comparing with the first speaker'"'"'s templates. The second speaker speaks the same words at a rate time-warped to match the first speakers rate and template. We define a demisyllable as each of the two halves of a syllable, assuming a syllable starts and ends with a noisy consonant, and the syllable is split at its vowel center, thereby simplifying concatenation and comparison. Key features of the invention include generating a set of signals representative of the time alignment between the first and second speaker'"'"'s templates, and the time-of-occurence boundaries of each syllable in a word.

Claims

1. A method for producing subunit speech patterns comprising the steps of:

storing a plurality of reference speech pattern templates each comprising a time frame sequence of acoustic feature signals representative of a prescribed spoken reference speech pattern;

storing a set of signals each representative of the time of occurrence of at least one predetermined subunit in the reference acoustic feature signal sequence for each reference speech pattern;

analyzing an utterance of one of said stored reference speech patterns to generate a time frame sequence of acoustic feature signals representative of the utterance;

generating signals representative of the time alignment of said utterance feature signal sequence and said stored reference speech pattern feature signal sequence; and

determining the sequence of utterance feature signals corresponding to the predetermined subunit in said stored reference speech pattern template responsive to the time alignment signals and said reference subunit time of occurrence representative signals.

2. A method for producing subunit speech patterns according to claim 1 wherein said subunit time of occurrence signal storing step comprises storing a first signal representative of the time frame of the reference speech pattern template at which said subunit begins and storing a second signal representative of the time frame at which said subunit terminates;

and said determining step comprises detecting a first time frame of the utterance corresponding to said first signal and a second time frame of the utterance corresponding to said second signal, and storing the utterance feature signals from said first time frame and to said second time frame.

3. A method for producing subunit speech patterns according to claim 2 wherein said time aligning step comprises modifying said utterance feature signals to match the number of time frames in said utterance to the number of time frames in said stored reference speech pattern jointly responsive to the utterance feature signals and the stored reference speech pattern feature signals.

4. A method for producing subword speech patterns according to claim 3 wherein said time aligning step further comprises producing a set of signals representative of the best matching time frame path between said utterance feature signals and reference speech pattern feature signals jointly responsive to the modified utterance feature signals and the reference speech pattern feature signals.

5. A method for producing subunit speech patterns according to claim 4 wherein said utterance time frame detecting step comprises comparing said first signal to the sequence of reference speech pattern time frames to generate a subunit begin frame signal, comparing said second signal to the sequence of reference speech pattern time frames to generate a subunit endframe signal, producing a signal representative of the utterance time frame corresponding to the reference speech pattern subunit begin frame responsive to said reference speech pattern subunit begin frame signal and said best matching time frame path signals, and producing a signal representative of the utterance time frame corresponding to the reference speech pattern subunit endframe responsive to said reference speech pattern subunit endframe signal and said best matching time frame path signals.

6. A method for producing subunit speech patterns according to claim 5 wherein the step of producing an utterance time frame signal corresponding to the reference speech pattern subunit endframe comprises determining the time of occurrence of the reference speech pattern subunit endframe in the sequence of reference speech pattern time frames responsive to said second signal, and selecting the utterance time frame corresponding to the best time alignment at said time of occurrence responsive to the best matching time frame path signals.

7. A method for producing subunit speech patterns according to claim 5 wherein the step of producing an utterance time frame signal corresponding to the reference speech pattern subunit begin frame comprises determining the time of occurrence of the reference speech pattern subunit begin frame in the sequence of reference speech pattern time frames responsive to said first signal and selecting the utterance time frame corresponding to the best time alignment at said time of occurrence responsive to the best matching time frame path signals.

8. A method for producing subunit speech patterns according to claim 6 wherein the utterance modifying step comprises generating a scale factor signal representative of the ratio of the number of frames in the utterance to the number of frames in the reference speech pattern, and wherein the step of selecting the utterance time frame having the best time alignment with the determined reference speech pattern subunit endframe comprises detecting the modified utterance time frame corresponding to said determined reference speech pattern subunit endframe responsive to said best matching time frame path signals, and multiplying the detected modified utterance endframe signal by said scale factor signal.

9. A method for producing subunit speech patterns according to claim 7 wherein the utterance modifying step comprises generating a scale factor signal representative of the ratio of the number of utterance frames to the number of reference speech pattern frames, and wherein the step of selecting the utterance time frame having the best time alignment with the determined reference speech pattern subunit begin frame comprises detecting the modified utterance time frame corresponding to said determined reference speech pattern subunit begin frame responsive to said best matching time frame path signals, and multiplying the detected modified utterance begin frame signal by said scale factor signal.

10. A method for producing subunit speech patterns according to claims 1, 2, 3, 4, 5, 6, 7, 8 or 9 wherein each reference speech pattern is a reference word speech pattern and each subunit speech pattern is a subword unit speech pattern.

11. Apparatus for producing subunit speech patterns comprising means for storing a plurality of reference speech pattern templates, each template comprising a time frame sequence of acoustic feature signals representative of a prescribed spoken reference speech pattern and for storing a set of signals representative of the time of occurrence of at least one predetermined subunit in the reference speech pattern template for each reference speech pattern;

means for analyzing an utterance of one of said stored reference speech patterns to generate a sequence of acoustic speech signals representative of the utterance;

means for generating signals representative of the time alignment of the utterance feature signal sequence and the stored reference speech pattern feature signal sequence; and

means for determining the sequence of utterance feature signals coresponding to the predetermined subunit in said stored reference speech pattern template responsive to the time alignment signals and the reference subunit time of occurrence representative signals.

12. Apparatus for producing subunit speech patterns according to claim 11 wherein said subunit time of occurrence signal storing means comprises means for storing a first signal representative of the time frame of the reference speech pattern template at which said subunit begins and for storing a second signal representative of the time frame at which said subunit terminates;

and said selecting means comprises means responsive to said time alignment signals for detecting a first time frame of the utterance corresponding to said first signal and a second time frame of the utterance corresponding to said second signal, and means for storing the utterance feature signals of the interval from said first time frame to said second time frame.

13. Apparatus for producing subunit speech patterns according to claim 12 wherein said means for generating time alignment signals comprises means for modifying said utterance feature signals to match the number of time frames in said utterance to the number of time frames in said stored reference speech pattern template jointly responsive to the utterance feature signals and the stored reference speech pattern feature signals.

14. Apparatus for producing subunit speech patterns according to claim 13 wherein said means for generating time alignment signals further comprises means for producing a set of signals representative of the best matching time frame path between the sequence of modified utterance feature signals and the sequence of reference speech pattern feature signals jointly responsive to the modified utterance feature signals and the reference speech pattern feature signals.

15. Apparatus for producing subunit speech patterns according to claim 14 wherein said means for detecting the first and second utterance time frames comprises means for producing a signal representative of the utterance time frame corresponding to the reference subunit begin frame responsive to said first signal and said best matching time frame path signals, and means for producing a signal representative of the utterance time frame corresponding to the reference subunit endframe responsive to said second signal and said best matching time frame path signals.

16. Apparatus for producing subunit speech patterns according to claim 15 wherein said utterance time frame detecting means comprises means for comparing said first signal to the sequence of reference speech pattern time frames to generate a subunit begin frame signal, means for comparing said second signal to the sequence of reference speech pattern time frames to generate a subunit endframe signal, means responsive to said reference subunit begin frame signal and said best matching time frame path signals for producing a signal representative of the utterance time frame corresponding to the reference subunit begin frame, and means responsive to said reference word subunit endframe signal and said best matching time frame path signals for producing a signal representative of the utterance frame corresponding to the reference subunit endframe.

17. Apparatus for producing subunit speech patterns according to claim 16 wherein the means for producing an utterance time frame signal corresponding to the reference word subunit endframe comprises means responsive to said second signal for determining the time of occurrence of the reference word subunit endframe, and means responsive to the best matching time frame path signals for selecting the utterance time frame corresponding to the best time alignment at said time of occurrence.

18. Apparatus for producing subunit speech patterns according to claim 16 wherein the means for producing an utterance time frame signal corresponding to the reference speech pattern subunit begin frame comprises means responsive to said first signal for determining the time of occurrence of reference speech pattern subunit begin frame, and means responsive to the best matching time frame path signals for selecting the utterance time frame corresponding to the best time alignment at said time of occurrence.

19. Apparatus for producing subword unit speech patterns according to claim 17 wherein the utterance modifying means comprises means for generating a scale factor signal representative of the ratio of the number of frames in the utterance to the number of frames in the reference speech pattern;

and wherein the means for selecting the utterance time frame corresponding to the determined reference word subunit endframe comprises means responsive to said best matching time frame path signals for detecting the modified utterance time frame corresponding to said determined reference speech pattern subunit endframe and means for multiplying the detected modified utterance endframe signal by said scale factor signal.

20. Apparatus for producing subunit speech patterns according to claim 18 wherein the utterance modifying means comprises means for generating a scale factor signal representative of the ratio of the number of utterance frames to the number of reference speech pattern frames;

and wherein the means for selecting the utterance time frame corresponding to the determined reference speech pattern subunit begin frame comprises means responsive to said best matching time frame path signals for detecting the modified utterance time frame corresponding to said determined reference speech pattern subunit begin frame, and means for multiplying the detected modified utterance begin frame signal by said scale factor signal.

21. Apparatus for producing subunit speech patterns according to claims 11, 12, 13, 14, 15, 16, 17, 18, 19 or 20 wherein each reference speech pattern is a reference word speech pattern and each subunit speech pattern is a subword unit speech pattern.