Automatic determination of labels and Markov word models in a speech recognition system

• US 5,072,452 A
• Filed: 11/02/1989
• Issued: 12/10/1991
• Est. Priority Date: 10/30/1987
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Nahomoo, David, Picheny, Michael A., Brown, Peter F., De Souza, Peter V.
• Primary Examiner(s): NOT, DEFINED
• Assistant Examiner(s): Merecki, John
• Application Number:
  US07/431,720
• Time in Patent Office: 768 Days
• Field of Search: 381/41-46, 364/513, 364/513.5, 382/14-15
• US Class Current: 704/256.4
• CPC Class Codes: G10L 15/14 using statistical models, e.g. Hidden Markov Models [HMMs] G10L15/18 takes precedence

Abstract

In a Markov model speech recognition system, an acoustic processor generates one label after another selected from an alphabet of labels. Each vocabulary word is represented as a baseform constructed of a sequence of Markov models. Each Markov model is stored in a computer memory as (a) a plurality of states; (b) a plurality of arcs, each extending from a state to a state with a respective stored probability; and (c) stored label output probabilities, each indicating the likelihood of a given label being produced at a certain arc. Word likelihood based on acoustic characteristics is determined by matching a string of labels generated by the acoustic processor against the probabilities stored for each word baseform. Improved models of words are obtained by specifying label parameters and constructing word baseforms interdependently and iteratively.

Claims

1. Speech processing apparatus comprising:

an acoustic processor for producing as a first output, in response to speech input, one label after another at successive time intervals, each label being selected from an alphabet of labels, each label having parameter values;

dictionary means for storing statistical data for each of a plurality of vocabulary words as Markov model word baseforms, wherein each baseform is characterized by a sequence of Markov models, at least one word baseform containing at least one Markov model at different locations in the sequence, each Markov model having a plurality of arcs, wherein the dictionary means includes storage for (i) the respective probability of each arc in each Markov model, and (ii) a respective probability of producing each label in the alphabet at each of some arcs in each Markov model

means, coupled to said acoustic processor, for re-specifying the parameter values of the labels in the alphabet which can be produced as outputs of the acoustic processor; and

baseform constructor means, coupled to said dictionary means, for up-dating the stored data for the Markov model word baseforms from labels generated by the acoustic processor based on the re-specified parameter values;

wherein said label re-specifying means re-specifies the parameter values of labels based on the up-dated stored data for the Markov model word baseforms;

wherein said acoustic processor produces as a second output one feature vector after another at the successive time intervals;

wherein each different Markov model corresponds to one respective label; and

wherein said label re-specifying means includes;

alignment processor means for aligning a string of labels generated by the acoustic processor against a word baseform stored in the dictionary means, said alignment processor means aligning successive substrings in the string with successive Markov models in the word baseform; and

estimator means for receiving as input from the acoustic processor the feature vectors corresponding to the labels aligned with a given Markov model and computing means and covariance values of the feature vectors received for the given Markov model; and

label specifier means, coupled to the estimator means, for storing (i) the mean and covariance values of the feature vectors corresponding to the labels aligned with each Markov model, as (ii) the parameter values of the label corresponding to the Markov model.

2. Apparatus as in claim 1 wherein said acoustic processor includes:

means for comparing each successive feature vector generated by the acoustic processor against the mean and covariance values associated with each label and determining which means and covariance values are closets to the feature vector; and

means for assigning to each successive interval the label with associated mean and covariance values closet to the feature vector at said each interval.

3. Apparatus as in claim 2 further comprising:

training means for computing Markov model data based on a string of labels generated by the acoustic processor in response to the utterance of a known text during the training session;

said training means being connected to enter the Markov model data into said baseform constructor means.

4. Apparatus as in claim 3, further comprising:

acoustic match means for (a) storing the word baseforms and the Markov model data computed therefor by said training means and (b) computing the measure of match between a string of labels generated by said acoustic processor and the sequence of Markov models for each word baseform.

5. Apparatus as in claim 4 further comprising:

language model processor means for storing occurrence data of word sequences derived from sample database text and for computing a likelihood score of each word based on the stored occurrence data; and

stack decoder means, connected to receive as inputs (i) language model likelihood scores for words as computed by said language model processor means and (ii) acoustic match scores for words as computed by said acoustic match means, for selecting likely next words for each of a plurality of word paths based on the acoustic match scores and language model scores computed for the words;

said stack decoder means including means for computing, for each word path having a likelihood above a prescribed threshold, the likelihood of said each path after selected words are appended thereto.

6. A computerized method of processing speech for speech recognition comprising the steps of:

(a) generating, in an acoustic processor, one feature vector after another for one time interval after another in response to uttered speech, each feature vector having a feature value;

(b) for each time interval, assigning one of an alphabet of stored labels thereto which corresponds to one prototype vector of an alphabet of prototype vectors, each prototype vector having parameter values, the parameter values of said one assigned prototype vector being the closest to the feature value of the feature vector generated for a given time interval;

(c) storing each word of a vocabulary in a computer memory as a sequence of Markov models, at least one word containing at least one Markov model at different locations in the sequence, which includes the steps of;

selecting a set of Markov models wherein each Markov model corresponds to a label; and

storing, for each Markov model, a plurality of arc probabilities and label probabilities, wherein each label probability corresponds to the likelihood of a respective label being produced at a given Markov model arc;

(d) for an uttered known word sequence, aligning labels which are generated according to step (a) with each successive Markov model included in the known word sequence; and

(e) for a subject Markov model, re-specifying the prototype vector based solely on the feature vectors corresponding to each label aligned with the subject Markov model and associating the re-specified prototype vector with the label corresponding to the subject Markov model.

7. The method of claim 6 comprising the further step of:

(f) repeating step (e) for each Markov model as the subject Markov model.

8. The method of claim 6 wherein step (e) includes the steps of:

(g) computing and storing the mean and covariance values over all the feature vectors corresponding to the labels aligned with the subject Markov model; and

(h) re-specifying the parameters of the label corresponding to the subject Markov model as the mean and covariance values computed in step (g) for the subject Markov model.

9. The method of claim 8 comprising the further step of:

(j) repeating step (e) for each Markov model as the subject Markov model.

10. The method of claim 9 comprising the further steps of:

(k) for an uttered known word sequence, (i) generating successive feature vectors, (ii) comparing each feature vector against the mean value and covariance value computed in step

(g) for each label, and (iii) assigning to each interval the re-specified label having the closest mean value and covariance value to the feature vector corresponding to the interval; and

(l) re-constructing the word baseforms based on the respecified labels.

11. The method of claim 10 wherein step (l) includes the steps of:

(m) transforming each of multiple utterances of the word segment into a respective string of labels;

(n) determining the best single Markov model P1 for producing the multiple label strings;

(p) determining the best two-model baseform of the form P1P2 or P2P1 for producing the multiple label strings;

(q) aligning the best two-model baseform against each label string;

(r) splitting each label string into a left portion and a right portion with the left portion corresponding to the first model of the two-model baseform and the right portion corresponding to the second model of the two-model baseform;

(s) identifying each left portion as a left substring and each right portion as a right substring;

(t) processing the set of left substrings in the same manner as a set of label strings corresponding to the multiple utterances including the further step of inhibiting further splitting of a substring when the single model baseform thereof has a higher probability of producing the substring than does the best two-model baseform;

(u) processing the set of right substrings in the same manner as the set of label strings corresponding to the multiple utterances, including the further step of inhibiting further splitting of a substring when the single model baseform thereof has a higher probability of producing the substring than does the best two-model baseform; and

(v) concatenating the unsplit single models in an order corresponding to the order of the label substrings to which they correspond.

12. The method of claim 11 wherein step (l) includes the further steps of:

(w) aligning a concatenated baseform against each of the label strings and identifying, for each model in the concatenated baseform, the substring in each label string which corresponds thereto, the substrings corresponding to a given model being a set of common substrings;

(x) for each set of common substrings, determining the model having the highest joint probability of producing the common substrings; and

(y) for each common substring, replacing the model therefor in the concatenated baseform by the determined model of highest joint probability;

the baseform resulting from the replacing of models being a refined baseform.

13. The method of claim 12 wherein step (l) further includes the step of:

(z) repeating steps (w) through (y) until no models are replaced.

14. A speech processing apparatus comprising:

means for measuring the value of at least one feature of a speech input, said speech input occurring over a series of successive time intervals, said means measuring the feature value of the speech input during each time interval to produce a series of feature vector signals representing the feature values;

means for storing a plurality of label vector signals, each label vector signal having at least one parameter value and having a unique identification value;

means for storing a baseform of a first word, said baseform comprising a sequence of baseform segments, each baseform segment being assigned a label vector signal identification value, at least two separate baseform segments being assigned a first label vector signal identification value of a first label vector signal;

means for sorting a series of feature vector signals produced by the measuring means as a result of one or more utterances of the first word into groups of one or more feature vector signals, one group of feature vector signals corresponding to the two or more segments of the baseform of the first word which are assigned the first label vector signal identification value; and

means for modifying the stored parameter value of the first label vector signal as a function solely of the feature values of the feature vectors which correspond to the baseform segments which are assigned the identification value of the first label vector signal.

15. A speech processing apparatus as claimed in claim 14, characterized in that each baseform segment is assigned a Markov model.

16. A speech processing apparatus as claimed in claim 15, characterized in that the means for modifying the stored parameter value of the first label vector signal calculates the arithmetic mean and covariance of the feature values of the feature vectors which correspond to the baseform segments which are assigned the identification value of the first label vector signal.

17. A speech processing apparatus as claimed in claim 14, characterized in that the sorting means comprises:

means for comparing the feature value, of each feature vector signal in the series of feature vector signals produced by the measuring means as a result of the utterance of the first word, to the parameter of the label vector signals to determine, for each feature vector, the closest associated label vector signal to produce a series of closest label vector signals; and

means for determining, for each label vector signal in the series of closest label vector signals, the baseform segment of the uttered word which most likely generated the feature value associated with the label vector signal.

18. A speech processing apparatus comprising:

means for measuring the value of at least one feature of a speech input, said speech input occurring over a series of successive time intervals, said means measuring the feature value of the speech input during each time interval to produce a series of feature vector signals representing the feature values;

means for storing a plurality of label vector signals, each label vector signal having at least one parameter value and having a unique identification value;

means for storing baseforms of first and second words, each baseform comprising a sequence of baseform segments, each baseform segment being assigned a label vector signal identification value, at least one baseform segment from the baseform of the first word being assigned a first label vector signal identification value of a first label vector signal, at least one baseform segment from the baseform of the second word being assigned the first label vector signal identification value;

means for sorting a series of feature vector signals produced by the measuring means as a result of one or more utterances of the first and second words into groups of one or more feature vector signals, one group of feature vector signals corresponding to the two or more segments of the baseforms of the first and second words which are assigned the first label vector signal identification value; and

means for modifying the stored parameter value of the first label vector signal as a function solely of the feature values of the feature vectors which correspond to the baseform segments which are assigned the identification value of the first label vector signal.

19. A speech processing apparatus as claimed in claim 18, characterized in that each baseform segment is assigned a Markov model.

20. A speech processing apparatus as claimed in claim 19, characterized in that the means for modifying the stored parameter value of the first label vector signal calculates the arithmetic mean and covariance of the feature values of the feature vectors which correspond to the baseform segments which are assigned the identification value of the first label vector signal.

21. A speech processing apparatus as claimed in claim 18, characterized in that the sorting means comprises:

means for comparing the feature value, of each feature vector signal in the series of feature vector signals produced by the measuring means as a result of the utterance of the first and second words, to the parameter values of the label vector signals to determine, for each feature vector, the closest associated label vector signal to produce a series of closest label vector signals; and

means for determining, for each label vector signal in the series of closest label vector signals, the baseform segment of the uttered words which most likely generated the feature value associated with the label vector signal.