Speech recognition system for recognizing continuous and isolated speech

• US 6,076,056 A
• Filed: 09/19/1997
• Issued: 06/13/2000
• Est. Priority Date: 09/19/1997
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Jiang, Li, Huang, Xuedong D., Hwang, Mei-Yuh, Alleva, Fileno A.
• Primary Examiner(s): Dorvil, Richemond
• Application Number:
  US08/934,622
• Time in Patent Office: 998 Days
• Field of Search: 704/256, 704/251, 704/252, 704/231, 704/240, 704/242, 704/254, 704/249, 704/200, 704/245, 704/238, 704/239
• US Class Current: 704/254
• CPC Class Codes: G10L 15/05 Word boundary detection
• CPC Class Codes: G10L 15/08 Speech classification or search

Abstract

Speech recognition is performed by receiving isolated speech training data indicative of a plurality of discretely spoken training words, and receiving continuous speech training data indicative of a plurality of continuously spoken training words. A plurality of speech unit models is trained based on the isolated speech training data and the continuous speech training data. Speech is recognized based on the speech unit models trained.

Claims

1. A method of implementing a speech recognition system, comprising:

receiving isolated speech training data indicative of a plurality of discretely spoken training words;

receiving continuous speech training data indicative of a plurality of continuously spoken training words;

providing a plurality of speech unit models including output probability distributions representative of speech units in the continuous and isolated speech training data, the models being trained based on both the isolated speech training and the continuous speech training data; and

providing a recognizer to recognize speech based on the speech unit models trained.

2. The method of claim 1 wherein receiving the isolated speech training data includes receiving a first plurality of acoustic signals, wherein receiving continuous speech training data includes receiving a second plurality of acoustic signals, and wherein providing a plurality of speech unit models comprises:

developing a plurality of acoustic models based on the first and second plurality of acoustic signals.

3. The method of claim 2 wherein developing a plurality of acoustic models comprises:

developing a plurality of output probability distributions representative of phonemes in the continuous and isolated speech training data based on the first and second plurality of acoustic signals.

4. The method of claim 3 wherein providing a plurality of speech unit models further comprises:

associating each of the output distributions with one of a predetermined number of states in a phoneme forming at least part of one of the training words.

5. The method of claim 4 and further comprising:

for each phoneme, grouping output distributions associated with a selected phoneme from all of the training words containing the selected phoneme to form an output distribution group; and

for each state in each phoneme, creating a senone tree for a selected state in the selected phoneme by separating the output distributions associated with the selected state in the output distribution group into senones based on linguistic context information associated with the selected phoneme.

6. The method of claim 5 wherein providing a recognizer for recognizing speech comprises configuring the speech recognizer to perform the steps of:

receiving an output distribution for each successive state of each successive target phoneme in a target word to be recognized;

for each target phoneme, identifying a number of likely phonemes which are most likely representative of the target phoneme;

comparing senones associated with the states of the likely phonemes with the output distributions associated with corresponding states of the target phoneme; and

identifying a most likely phoneme having senones that most closely match the output distributions of the target phoneme.

7. The method of claim 6 wherein comparing comprises:

traversing the senone tree associated with each state in each likely phoneme, based on linguistic context information of the target phoneme, to identify a senone for each state in the target phoneme; and

comparing the output distribution associated with the state in the target phoneme with the output distribution associated with the identified senone in the likely phoneme.

8. The method of claim 6 wherein identifying a number of likely phonemes comprises:

forming a plurality of monophone models, based on the isolated speech training data and the continuous speech training data, indicative of phonemes in the training words;

comparing the output distributions associated with the target phoneme to the monophone models; and

identifying a number of likely phonemes having monophone models which closely match the output distributions associated with the target phoneme.

9. The method of claim 1 wherein receiving isolated speech training data comprises:

receiving isolated speech data which includes silence context information associated with a plurality of discretely spoken training words.

10. The method of claim 9 wherein receiving isolated speech data comprises:

receiving the isolated speech data indicative of a user speaking the plurality of training words discretely, with pauses between each of the plurality of training words.

11. The method of claim 1 wherein receiving continuous speech training data comprises:

receiving continuous speech data indicative of a user speaking a plurality of training words fluently.

12. The method of claim 1 and further comprising:

weighting the continuous speech training data and the isolated speech training data, prior to training the speech unit models, based on expected speech to be recognized.

13. The method of claim 1 and further comprising:

receiving additional speech training data indicative of a user speaking a plurality of training words in different styles.

14. The method of claim 13 wherein receiving additional speech training data comprises:

receiving the additional speech training data indicative of the user speaking the plurality of training words at a first amplitude and at a second amplitude, the second amplitude being larger than the first amplitude.

15. The method of claim 13 wherein receiving additional speech training data comprises:

receiving the additional speech training data indicative of the user speaking the plurality of training words fluently, at a first pace, and at a second pace, the second pace being faster than the first pace.

16. The method of claim 1 and further comprising:

providing a plurality of word duration models based on the isolated speech training data and the continuous speech training data indicative of an approximate word duration of words contained in word phrases of varying length, the length being defined by a word count in the word phrases.

17. The method of claim 16 wherein providing a speech recognizer to recognize speech comprises configuring the recognizer to perform the steps of:

receiving a plurality of target words to be recognized;

detecting phrase boundaries indicative of target word phrases in the target words;

determining an approximate duration of the target word phrases;

obtaining a plurality of word phrase hypotheses indicative of likely word phrases represented by the target word phrases;

determining approximate word counts and durations for words in the word phrase hypotheses; and

comparing the word durations for the words in the word phrase hypotheses with word duration models having a word count equal to the number of words in the word phrase hypotheses to obtain a likely word phrase hypothesis based on how closely the word duration in the word phrase hypotheses match the word duration models.

18. The method of claim 16 wherein providing a plurality of word duration models comprises:

detecting training word phrases in the isolated speech training data and the continuous speech training data;

determining a number of words in the training word phrases;

determining an approximate word duration for the words in a plurality of the training word phrases detected; and

determining a plurality of word duration distributions parameterized on the number of words in the training word phrases and the duration of the words in the training word phrases.

19. A method of recognizing speech, comprising:

receiving input data indicative of the speech to be recognized;

detecting pauses in the speech, based on the input data, to identify a phrase duration;

generating a plurality of phrase hypotheses representative of likely word phrases represented by the input data between the pauses detected;

comparing a word duration associated with each word in each phrase hypothesis, based on a number of words in the phrase hypothesis and based on the phrase duration, with an expected word duration for a phrase having a number of words equal to the number of words in the phrase hypothesis; and

assigning a score to each phrase hypothesis based on the comparison of the word duration with the expected word duration to obtain a most likely phrase hypothesis represented by the input data.

20. The method of claim 19 and further comprising:

receiving training data indicative of words to be recognized;

detecting pauses in the training data to identify a plurality of training word phrases;

determining a number of words in each of the training word phrases;

generating a plurality of distributions of word durations corresponding to the training word phrases based on a number of words in each of the training word phrases.

21. The method of claim 20 wherein comparing a word duration in each phrase hypothesis with an expected word duration comprises:

for each hypothesis, determining a word duration for the words in the hypothesis based on the number of words in the hypothesis and the phrase duration;

choosing a selected one of the plurality of distributions associated with a number of words per phrase equal to the number of words in the hypothesis; and

comparing the word duration determined for the hypothesis to the selected distribution.

22. The method of claim 21 wherein assigning a score to each phrase hypothesis comprises:

assigning a score to each word hypothesis indicative of how closely the word duration determined for the hypothesis matches the selected distribution.

23. A method of performing speech recognition, comprising:

receiving isolated speech training data indicative of a plurality of discretely spoken training words, the isolated speech training data including a first plurality of output distributions, each output distribution being associated with one of a predetermined number of states in a phoneme forming at least part of one of the discretely spoken training words;

receiving continuous speech training data indicative of a plurality of continuously spoken training words, the continuous speech training data including a second plurality of output distributions, each of the second plurality of output distributions being associated with one of a predetermined number of states in a phoneme forming at least part of one of the continuously spoken training words;

grouping output distributions associated with a selected phoneme from all of the training words containing the selected phoneme to form an output distribution group; and

creating a senone tree for a selected state in the selected phoneme by separating the output distributions associated with the selected state in the output distribution group based on linguistic context information associated with the selected phoneme.