State-dependent speaker clustering for speaker adaptation

• US 5,787,394 A
• Filed: 12/13/1995
• Issued: 07/28/1998
• Est. Priority Date: 12/13/1995
• Status: Expired due to Term

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: International Business Machines Corporation
• Inventors: Nahamoo, David, Gopalakrishnan, Ponani, Padmanabhan, Mukund, Bahl, Lalit Rai
• Primary Examiner(s): Hudspeth, David R.
• Assistant Examiner(s): Opsasnick, Michael N.
• Application Number:
  US08/572,223
• Time in Patent Office: 958 Days
• Field of Search: 395/2.45, 395/2.52, 395/2.54, 395/2.62, 395/2.63, 704/236, 704/238, 704/243, 704/245, 704/253, 704/254
• US Class Current: 704/238
• CPC Class Codes: G10L 15/07 to the speaker
• CPC Class Codes: G10L 2015/0631 Creating reference templates; Clustering

Abstract

A system and method for adaptation of a speaker independent speech recognition system for use by a particular user. The system and method gather acoustic characterization data from a test speaker and compare the data with acoustic characterization data generated for a plurality of training speakers. A match score is computed between the test speaker'"'"'s acoustic characterization for a particular acoustic subspace and each training speaker'"'"'s acoustic characterization for the same acoustic subspace. The training speakers are ranked for the subspace according to their scores and a new acoustic model is generated for the test speaker based upon the test speaker'"'"'s acoustic characterization data and the acoustic characterization data of the closest matching training speakers. The process is repeated for each acoustic subspace.

Claims

1. A method for adapting the parameters of a speech recognition system during a training process, to better recognize speech of a particular test speaker comprising the steps of:

calculating the acoustic characterization of a plurality of training speakers for all acoustic subspaces of an acoustic space, the acoustic characterizations being individually identifiable for each training speaker for each acoustic subspace;

calculating the acoustic characterization of a test speaker from adaptation data provided by said test speaker for acoustic subspaces of the acoustic space;

computing a match score between the test speaker'"'"'s characterization for each acoustic subspace, and each training speaker'"'"'s characterization for the same acoustic subspace;

ranking each of the training speakers in the acoustic subspace based upon the score; and

for each acoustic space, generating a re-estimated acoustic model for the particular acoustic subspace using individually identifiable data respectively derived from the one or more training speakers closest to the test speaker for that acoustic subspace, the re-estimated acoustic model for each acoustic subspace being used during a decoding process.

2. The method of claim 1, wherein the speakers are characterized by the duration of a phone class, and wherein the training speakers closest to the test speaker are determined to be those with the closest duration distribution.

3. The method of claim 1, wherein the step of generating an acoustic model comprises:

generating a shortlist of training speakers who are deemed to be close to the test speaker in that acoustic subspace; and

using the training data of the shortlisted speakers to construct a model for the test speaker in that acoustic subspace.

4. The method of claim 1, wherein the step of calculating the acoustic characterization of a plurality of training speakers comprises:

generating a hierarchical tree representing the contextual characterization of each phone for each training speaker, the tree terminating in a plurality of leaves, each leaf representing a phone in a given context, the tree further including one or more nodes each representing similar ones of the leaves, or of similar ones of the leaves and other nodes.

5. The method of claim 4, wherein each leaf represents a region in the acoustic subspace.

6. The method of claim 5, wherein the step of calculating the acoustic characterization of the test speaker comprises:

producing an alignment of the test speaker'"'"'s data and tagging each feature vector with a leaf; and

wherein when insufficient data exists for the construction of a model at a leaf, constructing a model at a node of the tree that includes that leaf.

7. The method of claim 6, wherein the step of computing a match comprises:

measuring the distance, in acoustic space, between corresponding nodes of the test speaker'"'"'s model and each training speaker'"'"'s model.

8. The method of claim 7, wherein the step of measuring the distance comprises:

determining the likelihood of the test speaker'"'"'s data at a particular node of the test speaker'"'"'s tree given the training speaker'"'"'s model at the corresponding node, made up by using the training speaker data at all leaves comprising that node.

9. The method of claim 8, wherein the likelihood is computed using the training speaker'"'"'s mixture of Gaussians for that node.

10. The method of claim 1, further comprising:

computing a transformation to map the acoustic subspace of the selected training speaker closer to the corresponding subspace of the test speaker.

11. The method of claim 1, further comprising:

computing a transformation to map the acoustic subspace of the selected training speaker closer to the corresponding subspace of the test speaker; and

using the transformed training data of the shortlisted speakers to construct a model for the test speaker in that acoustic subspace.

12. A system comprising:

means for adapting the parameters of a speech recognition system during a training process to better recognize speech of a particular test speaker;

means for calculating the acoustic characterization of a plurality of training speakers for all acoustic subspaces of an acoustic space, the acoustic characterizations being individually identifiable for each training speaker for each acoustic subspace;

means for calculating the acoustic characterization of a test speaker from adaptation data provided by said test speaker for acoustic subspaces of the acoustic space;

means for computing a match score between the test speaker'"'"'s characterization for each acoustic subspace, and each training speaker'"'"'s characterization for the same acoustic subspace;

means for ranking each of the training speakers in the acoustic subspace based upon the score; and

means for each acoustic space, for generating a re-estimated acoustic model for the particular acoustic subspace using individually identifiable data respectively derived from the one or more training speakers closest to the test speaker for that acoustic subspace, the re-estimated acoustic model for each acoustic subspace being used during a decoding process.

13. The system of claim 12, further comprising:

means for speaker characterization by the duration of a phone class, and wherein the training speakers closest to the test speaker are determined to be those with the closest duration distribution.

14. The system of claim 12, wherein the means for generating an acoustic model comprises:

means for generating a shortlist of training speakers who are deemed to be close to the test speaker in that acoustic subspace; and

means for using the training data of the shortlisted speakers to construct a model for the test speaker in that acoustic subspace.

15. The system of claim 12, wherein the means for calculating the acoustic characterization of a plurality of training speakers comprises:

means for generating a hierarchical tree representing the contextual characterization of each phone for each training speaker, the tree terminating in a plurality of leaves, each leaf representing a phone in a given context, the tree further including one or more nodes each representing similar ones of the leaves, or of similar ones of the leaves and other nodes.

16. The system of claim 15, wherein each leaf represents a region in the acoustic subspace.

17. The system of claim 16, wherein the means for calculating the acoustic characterization of the test speaker comprises:

means for producing an alignment of the test speaker'"'"'s data and tagging each featured vector with a leaf; and

where insufficient data exists for the construction of a model at a leaf, means for constructing a model at a node of the tree that includes the leaf.

18. The system of claim 17, wherein the means for computing a match comprises:

means for measuring the distance, in acoustic space, between corresponding nodes of the test speaker'"'"'s model and each training speaker'"'"'s model.

19. The system of claim 18, wherein the means for measuring the distance comprises:

means for determining the likelihood of the test speaker'"'"'s data at a particular node of the test speaker'"'"'s tree given the training speaker'"'"'s model at the corresponding node, made up by using the training speaker data at all leaves comprising that node.

20. The system of claim 19, wherein the likelihood is computed using the training speaker'"'"'s mixture of Gaussians for that node.

21. The system of claim 12, further comprising:

means for computing a transformation that maps the training speaker data in the acoustic subspace to bring the training speaker data closer with respect to a calculated distance to the test speakers subspace, the transformation mapping reducing the calculated distance to achieve acoustic similarity between the transformed data of the training speaker and the data of the test speaker.

22. The system of claim 12, further comprising:

means for computing a transformation to map the acoustic subspace of the selected training speaker closer to the corresponding subspace of the test speaker.

23. A training system, comprising:

a training speaker feature vector and training script store for storing a script and a plurality of test speakers utterances of the training script;

a viterbi alignment processor for aligning each of the plurality of training speakers'"'"' feature vectors with the training script;

a training speaker data model for modeling each of the training speakers'"'"' feature vectors in acoustic space, given the alignment with the training script;

a test speaker feature vector and test script store for storing feature vectors representing a test speaker'"'"'s utterances of the test script;

a test speaker viterbi alignment processor for aligning the test speaker'"'"'s feature vectors with the test script;

a training speaker select processor for selecting for each of a plurality of acoustic subspaces in the acoustic space, one or more individually identifiable training speakers whose feature vectors most closely match the test speaker'"'"'s feature vectors in that acoustic subspace;

a training data transformer for mapping the selected training speaker'"'"'s feature vectors to the test speakers'"'"' feature vectors;

a Gaussian parameter re-estimation processor for estimating the first and second moment and posterior probability of the mapped feature vectors; and

a decoder for decoding the estimated moments and probability.