Apparatuses and methods for developing and using models for speech recognition

• US 5,715,367 A
• Filed: 01/23/1995
• Issued: 02/03/1998
• Est. Priority Date: 01/23/1995
• Status: Expired due to Term

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: Dragon Systems, Inc. (Microsoft Corporation)
• Inventors: Gillick, Laurence S., Scattone, Francesco
• Primary Examiner(s): Zele, Krista M.
• Assistant Examiner(s): Hunter, Daniel S.
• Application Number:
  US08/376,348
• Time in Patent Office: 1,107 Days
• Field of Search: 395/2.52, 395/2.54, 395/2.58, 395/2.59, 395/2.63, 395/2.64, 395/2.65, 395/2.12, 395/2.53
• US Class Current: 704/254
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 2015/022 Demisyllables, biphones or triphones being the recognition units
• CPC Class Codes: G10L 2015/0631 Creating reference templates; Clustering

Abstract

A computerized system time aligns frames of spoken training data against models of the speech sounds; automatically selects different sets of phonetic context classifications which divide the speech sound models into speech sound groups aligned against acoustically similar frames; creates model components from the frames aligned against speech sound groups with related classifications; and uses these model components to build a separate model for each related speech sound group. A decision tree classifies speech sounds into such groups, and related speech sound groups descend from common tree nodes. New speech samples time aligned against a given speech sound group'"'"'s model update models of related speech sound groups, decreasing the training data required to adapt the system. The phonetic context classifications can be based on knowledge of which contextual features are associated with acoustic similarity. The computerized system samples speech sounds using a first, larger, parameter set; automatically selects combinations of phonetic context classifications which divide the speech sounds into groups whose frames are acoustically similar, such as by use of a decision tree; selects a second, smaller, set of parameters based on that set'"'"'s ability to separate the frames aligned with each speech sound group, such as by used of linear discriminant analysis; and then uses these new parameters to represent frames and speech sound models. Then, using the new parameters, a decision tree classifier can be used to re-classify the speech sounds and to calculate new acoustic models for the resulting groups of speech sounds.

Claims

1. A computerized method for automatically creating models of speech sounds to be used in speech recognition comprising the steps of:

receiving training signals representing the sound of spoken words;

storing a plurality of phonetic context units, each representing a speech sound in a phonetic context defined by one or more phonetic features, and associating with each phonetic context unit an initial acoustic model to represent its associated speech sound;

time aligning successive time frames of the training signals against the initial models of the phonetic context units of the words corresponding to those training signals, to associate each frame with the phonetic context unit whose sound it represents;

storing a set of classifications, each representing a possible set of one or more of the phonetic features which can be associated with one of said phonetic context units;

using an automatic classification routine to select a plurality of sub-sets of said classifications which divide the phonetic context units into phonetic context groups, such that the phonetic context units in each such phonetic context group tend to be time aligned against acoustically similar frames;

developing shared acoustic model components for a plurality of phonetic context groups whose associated sub-sets of classifications share a sub-sub-set of classifications and whose frames have a certain acoustic similarity, which shared acoustic model components contain statistical information derived from frames time aligned against the phonetic context units in different ones of said plurality of phonetic context groups; and

developing an acoustic model for each given phonetic context group in said plurality of phonetic context groups which contains a combination of said statistical information contained in the model components shared by said plurality of groups and more specific statistical information representing the frames time aligned against the phonetic context units in the given individual phonetic context group.

2. A computerized method as in claim 1 wherein:

said classification routine builds a decision tree to select the plurality of sub-sets of classifications used to divide the phonetic context units into phonetic context groups; and

said plurality of phonetic context groups which share a sub-sub-set of classifications are descendants from a common ancestor node in a common decision tree.

3. A computerized method for automatically creating models of speech sounds to be used in speech recognition comprising the steps of:

receiving training signals representing the sound of spoken words;

storing a plurality of phonetic context units, each representing a speech sound in a phonetic context defined by one or more phonetic features, and associating with each phonetic context unit an initial acoustic model to represent its associated speech sound;

time aligning successive time frames of the training signals against the initial models of the phonetic context units of the words corresponding to those training signals, to associate each frame with the phonetic context unit whose sound it represents;

storing a set of classifications, each representing a possible set of one or more of the phonetic features which can be associated with one of said phonetic context units;

using an automatic classification routine to select a plurality of sub-sets of said classifications which divide the phonetic context units into phonetic context groups, such that the phonetic context units in each such phonetic context group tend to be time aligned against acoustically similar frames;

developing shared acoustic model components for a plurality of phonetic context groups whose associated sub-sets of classifications share a sub-sub-set of classifications, and whose frames have a certain acoustic similarity; and

developing an acoustic model for each phonetic context group in said plurality of phonetic context groups based both on the model components shared by said plurality of groups and on the frames associated with the phonetic context units in the individual phonetic context group;

wherein;

said classification routine builds a decision tree to select the plurality of sub-sets of classifications used to divide the phonetic context units into phonetic context groups;

said plurality of phonetic context groups which share a sub-sub-set of classifications are descendants from a common ancestor node in a common decision tree;

the acoustic model components shared by a set of phonetic context groups having a common ancestor node are a set of probability distribution models, each representing a possible distribution of multi-dimensional acoustic values associated with frames time aligned against the phonetic context units of said set of phonetic context groups; and

the acoustic model developed for an individual phonetic context group is a mixture model made up of a weighted sum of such a set of distribution models, with each distribution model in the set being weighted so the sum of such models better represents the distribution of frame values associated with the individual phonetic context group.

4. A computerized method as in claim 3 further including the steps of:

creating a mixture model for each common ancestor node by weighting the set of distribution models associated with such a common ancestor node to better represent the distribution of frame values of all the phonetic context groups sharing that common ancestor node;

using the mixture models for a common ancestor node as a temporary model for each of its associated phonetic context groups; and

repeating the following succession of steps one or more times;

time aligning the successive frames of training signals against the phonetic context group models of the phonetic context units of the words corresponding to those training signals;

using the frames time aligned against all the phonetic context groups sharing a common ancestor node to update each of the common ancestor nodes'"'"'s set of associated distribution models; and

using the frames time aligned against each phonetic context group to update the weighting associated with each of the probability distributions associated with the mixture model for that phonetic context group.

5. A computerized method as in claim 3 further including the steps of:

receiving additional signals representing additional sounds of spoken words;

time aligning successive frames of the additional signals against the models of phonetic context groups corresponding to the phonetic context units of the words corresponding to those additional signals, to associate each additional frame with the phonetic context group whose sound it represent; and

using acoustic data from one or more of said additional frames associated with a given phonetic context group to update the acoustic models of a phonetic context group other than the given phonetic context group as a function of the closeness of common ancestry in said decision tree, and the acoustic similarity of frames, of the given and the other phonetic context group.

6. A computerized method as in claim 3 wherein said phonetic context units each represent one of a sequence of nodes in a multi-node phoneme-in-context model representing a given phoneme in a given context defined by at least its preceding and/or following phonemes, where the sequence of phonetic context units in the multi-node model represent a sequence of speech sounds associated with the phoneme in said given context.

7. A computerized method as in claim 6 wherein said classification routine creates a separate decision tree for each node of a generalized phoneme-in-context model created for each phoneme representing that phoneme in multiple contexts represented by a plurality of its phoneme-in-context models.

8. A computerized method as in claim 3 wherein said set of classifications used by said decision tree classification routine represents a sub-set of all possible sets of phonetic features, which sub-set is selected by a human based on linguistic knowledge of which phonetic features are likely to be associated with acoustical similarity.

9. A computerized method for automatically creating models of speech sounds to be used in speech recognition comprising the steps of:

receiving training signals representing the sound of spoken words;

storing a plurality of phonetic context units, each representing a speech sound in a phonetic context defined by one or more phonetic features, and associating with each phonetic context unit an initial acoustic model to represent its associated speech sound;

time aligning successive time frames of the training signals against the initial models of the phonetic context units of the words corresponding to those training signals, to associate each frame with the phonetic context unit whose sound it represents;

storing a set of classifications, each representing a possible set of one or more of the phonetic features which can be associated with one of said phonetic context units;

using an automatic classification routine to select a plurality of sub-sets of said classifications which divide the phonetic context units into phonetic context groups, such that the phonetic context units in each such phonetic context group tend to be time aligned against acoustically similar frames;

developing an acoustic model for each phonetic context group in said plurality of phonetic context groups based on the frames associated with the phonetic context units in that phonetic context group;

receiving additional signals representing additional sounds of spoken words;

time aligning successive time frames of the additional signals against the models of phonetic context groups corresponding to the phonetic context units of the words corresponding to those additional signals, to associate each such additional frame with a phonetic context group; and

automatically combining acoustic data from an additional frame time aligned against a given phonetic context group into the acoustic models of a different phonetic context group which shares a sub-sub-set of said classifications with, and whose associated frames have a given acoustic similarity to, said given phonetic context group.

10. A computerized method for automatically creating models of speech sounds to be used in speech recognition comprising the steps of:

receiving training signals representing the sound of spoken words;

storing a plurality of phonetic context units, each representing a speech sound in a phonetic context defined by one or more phonetic features, and associating with each phonetic context unit an initial acoustic model to represent its associated speech sound using p-parameters;

time aligning successive time frames of the training signals represented in said p parameters, against the initial models of the phonetic context units of the words corresponding to those training signals, to associate each frame with the phonetic context unit whose sound it represents;

storing a set of classifications, each representing a possible set of one or more the phonetic features which can be associated with one of said phonetic context units;

using an automatic classification routine to select a plurality of sub-sets of said classifications which divide the phonetic context units into phonetic context groups, such that the phonetic context units in each such phonetic context group tend to be time aligned against acoustically similar p-parameter frames;

using an automatic parameter selection routine to select a new set of q acoustic parameters derived from said p parameters, where q is less than p, which, for a given q, produces a relatively optimal separation between the sets of frames associated with each phonetic context group; and

using said q parameters to build a set of second acoustic models, for use in speech recognition, to represent said phonetic context units.

11. A computerized method as in claim 10 wherein said automatic parameter selection routine performs linear discriminant analysis to produce an L matrix which converts said p parameters into said q parameters.

12. A computerized method for automatically creating models of speech sounds to be used in speech recognition comprising the steps of:

receiving training signals representing the sound of spoken words;

storing a plurality of phonetic context units, each representing a speech sound in a phonetic context defined by one or more phonetic features, and associating with each phonetic context unit an initial acoustic model to represent its associated speech sound using p-parameters;

time aligning successive time frames of the training signals represented in said p parameters, against the initial models of the phonetic context units of the words corresponding to those training signals, to associate each frame with the phonetic context unit whose sound it represents;

storing a set of classifications, each representing a possible set of one or more the phonetic features which can be associated with one of said phonetic context units;

using an automatic classification routine to select a plurality of sub-sets of said classifications which divide the phonetic context units into phonetic context groups, such that the phonetic context units in each such phonetic context group tend to be time aligned against acoustically similar p-parameter frames;

using an automatic parameter selection routine to select a new set of q acoustic parameters derived from said p parameters, where q is less than p, which, for a given q, produces a relatively optimal separation between the sets of frames associated with each phonetic context group; and

using said q parameters to build a set of second acoustic models, for use in speech recognition, to represent said phonetic context units, wherein;

said automatic parameter selection routine performs linear discriminant analysis to produce an L matrix which converts said p parameters into said q parameters,

said classification routine builds a decision tree to select the plurality of sub-sets of classifications used to divide the phonetic context units into phonetic context groups.

13. A computerized method as in claim 12 wherein said set of classifications used by said decision tree classification routine represents a sub-set of all possible sets of phonetic features, which sub-set is selected by a human based on linguistic knowledge of which phonetic features are likely to be associated with acoustical similarity.

14. A computerized method as in claim 12 wherein said using of said q parameters to build a set of second acoustic models, for use in speech recognition, to represent said phonetic context units, includes:

using said L matrix to convert frames of training signals represented in said p parameters into frames represented by said q parameters;

using said automatic decision tree classification routine to build a new tree to select a new plurality of sub-sets of said classifications which divide the phoneme context units into groups, such that the phonetic context units in each such phonetic context group tend to be associated with acoustically similar q-parameter frames; and

developing a q-parameter acoustic model for each phonetic context group created by said decision tree classification based on the frames associated with the phonetic context units in that group.

15. A computerized method as in claim 12 wherein said phonetic context units each represent one of a sequence of nodes in a multi-node phoneme-in-context model representing a given phoneme in a given context defined by at least its preceding and/or following phonemes, where the sequence of phonetic context units in the multi-node model represent a sequence of speech sounds associated with the phoneme in said given context.

16. A computerized method as in claim 15 wherein said classification routine creates a separate decision tree for each node of a generalized phoneme-in-context model created for each phoneme representing that phoneme in multiple contexts represented by a plurality of its phoneme-in-context models.