METHOD AND APPARATUS FOR RECOGNIZING SPEECH

• US 20090076817A1
• Filed: 03/13/2008
• Published: 03/19/2009
• Est. Priority Date: 09/19/2007
• Status: Abandoned Application

• Current Assignee: Electronics and Telecommunications Research Institute
• Original Assignee: Electronics and Telecommunications Research Institute
• Inventors: CHUNG, Hoon, HWANG, Kyu Woong, JEON, Hyung Bae, KIM, Seung Hi, PARK, Jun, LEE, Yun Keun
• Application Number:
  US12/047,634
• Publication Number:
  US 20090076817A1
• Time in Patent Office: Days
• US Class Current: 704/240
• CPC Class Codes: G10L 15/187 Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams
• CPC Class Codes: G10L 2015/025 Phonemes, fenemes or fenones being the recognition units

Abstract

Provided are an apparatus and method for recognizing speech, in which reliability with respect to phoneme-recognized phoneme sequences is calculated and performance of speech recognition is enhanced using the calculated results. The method of recognizing speech includes the steps of: determining a boundary between phonemes included in character sequences that are phonetically input to detect each phoneme interval; calculating reliability according to a probability that a phoneme indicated by the detected phoneme interval corresponds to a phoneme included in a predefined phoneme model; calculating a phoneme alignment cost with respect to the character sequences based on the calculated reliability and a pre-trained and stored phoneme recognition probability distribution; and performing phoneme alignment based on the calculated phoneme alignment cost to perform speech recognition on the input character sequences. As a result, reliability with respect to the phoneme-recognized phoneme sequences can be calculated, and the performance of speech recognition can be enhanced using the calculated results.

Claims

1. A method of recognizing speech comprising, the steps of:

determining a boundary between phonemes included in character sequences that are phonetically input to detect each phoneme interval;

calculating reliability according to a probability that a phoneme indicated by the detected phoneme interval corresponds to a phoneme included in a predefined phoneme model;

calculating a phoneme alignment cost with respect to the character sequences based on the calculated reliability and a pre-trained and stored phoneme recognition probability distribution; and

performing phoneme alignment based on the calculated phoneme alignment cost to perform speech recognition on the input character sequences.

2. The method of claim 1, wherein the step of calculating the reliability comprises the steps of comparing a pattern of each phoneme interval with a pattern of each phoneme included in the predefined phoneme model to calculate likelihood, and calculating the reliability based on the calculated likelihood.

3. The method of claim 2, wherein the reliability(feature[q][i]) is calculated by the following equation:

$\mathrm{feature}<br><br>\left[q\right]<br><br>\left[i\right]=\mathrm{prob}<br><br>\left[q\right]<br><br>\left[i\right]=\frac{\mathrm{likelihood}<br><br>\left[q\right]<br><br>\left[i\right]}{\underset{j=1}{\overset{N}{\sum <br><br>}}<br><br>\mathrm{likelihood}<br><br>\left[q\right]<br><br>\left[j\right]}$

wherein feature[q][i] denotes reliability according to a probability that a phoneme indicated by a q^th phoneme interval of the entire detected phoneme intervals corresponds to an i^th phoneme of N phonemes included in a phoneme model,

prob[q][i] denotes a probability that the phoneme indicated by the q^th phoneme interval of the entire detected phoneme intervals corresponds to the i^th phoneme of N phonemes included in the phoneme model,

likelihood[q][i] denotes a likelihood between the phoneme indicated by the q^th phoneme interval of the entire detected phoneme intervals and the i^th phoneme of N phonemes included in a phoneme model, and

$\underset{j=1}{\overset{N}{\sum <br><br>}}<br><br>\mathrm{likelihood}<br><br>\left[q\right]<br><br>\left[j\right]$

denotes a sum of the likelihood between the phoneme indicated by the q^th phoneme interval of the entire detected phoneme intervals and each phoneme of N phonemes included in the phoneme model.

4. The method of claim 2, wherein the reliability(feature[q][i]) is calculated by the following equation:

$\mathrm{feature}<br><br>\left[q\right]<br><br>\left[i\right]=\mathrm{prob}<br><br>\left[q\right]<br><br>\left[i\right]=\frac{{\mathrm{<br><br>}}^{\ln <br><br><br><br>\mathrm{likelihood}<br><br>\left[q\right]<br><br>\left[i\right]}}{\underset{j=1}{\overset{N}{\sum <br><br>}}<br><br>{\mathrm{<br><br>}}^{\ln <br><br><br><br>\mathrm{likelihood}<br><br>\left[q\right]<br><br>\left[j\right]}}$

wherein feature[q][i] denotes reliability according to a probability that a phoneme indicated by a q^th phoneme interval of the entire detected phoneme intervals corresponds to an i^th phoneme of N phonemes included in a phoneme model,

prob[q][i] denotes a probability that the phoneme indicated by the q^th phoneme interval of the entire detected phoneme intervals corresponds to the i^th phoneme of N phonemes included in the phoneme model,

e^{lnlikelihood[q][i]}=likelihood[q][i] denotes a likelihood between the phoneme indicated by the q^th phoneme interval of the entire detected phoneme intervals and the i^th phoneme of N phonemes included in the phoneme model, and

$\underset{j=1}{\overset{N}{\sum <br><br>}}<br><br>{\mathrm{<br><br>}}^{\ln <br><br><br><br>\mathrm{likelihood}<br><br>\left[q\right]<br><br>\left[j\right]}=\underset{j=1}{\overset{N}{\sum <br><br>}}<br><br>\mathrm{likelihood}<br><br>\left[q\right]<br><br>\left[j\right]$

denotes a sum of the likelihoods between the phoneme indicated by the q^th phoneme interval of the entire detected phoneme intervals and each phoneme of N phonemes included in the phoneme model.

5. The method of claim 3, wherein the phoneme alignment cost cost(feature[q]|W_P) is calculated by the following equation:

$\mathrm{cost}<br><br>\left(\mathrm{feature}<br><br>\left[q\right]<br><br>W_P\right)=-\ln <br><br>\left(\underset{j=1}{\overset{N}{\sum <br><br>}}<br><br>\left(W_P<br><br>\left[i\right]\times <br><br>\mathrm{feature}<br><br>\left[q\right]<br><br>\left[i\right]\right)\right)$

wherein feature[q] denotes a reliability vector having reliability elements according to probabilities that the phoneme indicated by the q^th phoneme interval of the entire detected phoneme intervals corresponds to each phoneme of N phonemes included in the phoneme model,

W_P denotes a phoneme recognition probability distribution that is pre-trained with respect to a phoneme p included in the phoneme model,

W_P[i] denotes an average probability value of the i^th phoneme of the phoneme recognition probability distribution that is pre-trained with respect to the phoneme p included in the phoneme model, and

feature[q][i] denotes reliability according to the probability that the phoneme indicated by the q^th phoneme interval of the entire detected phoneme intervals corresponds to the i^th phoneme of N phonemes included in the phoneme model.

6. The method of claim 5, wherein the reliability(feature[q][i]) is calculated by the following equation:

$\mathrm{feature}<br><br>\left[q\right]<br><br>\left[i\right]=\ln <br><br>\left(\mathrm{prob}<br><br>\left[q\right]<br><br>\left[i\right]\right)=\ln \left(\frac{\mathrm{likelihood}<br><br>\left[q\right]<br><br>\left[i\right]}{\underset{j=1}{\overset{N}{\sum <br><br>}}<br><br>\mathrm{likelihood}<br><br>\left[q\right]<br><br>\left[j\right]}\right)$

wherein feature[q][i] denotes reliability according to a probability that the phoneme indicated by the q^th phoneme interval of the entire detected phoneme intervals corresponds to the i^th phoneme of N phonemes included in the phoneme model,

prob[q][i] denotes a probability that the phoneme indicated by the q^th phoneme interval of the entire detected phoneme intervals corresponds to the i^th phoneme of N phonemes included in the phoneme model,

likelihood[q][i] denotes a likelihood between the phoneme indicated by the q^th phoneme interval of the entire detected phoneme intervals and the i^th phoneme of N phonemes included in the phoneme model, and

$\underset{j=1}{\overset{N}{\sum <br><br>}}<br><br>\mathrm{likelihood}<br><br>\left[q\right]<br><br>\left[j\right]$

denotes a sum of the likelihoods between the phoneme indicated by the q^th phoneme interval of the entire detected phoneme intervals and each phoneme of N phonemes included in the phoneme model.

7. The method of claim 5, wherein the reliability(feature[q][i]) is calculated by the following equation:

$\mathrm{feature}<br><br>\left[q\right]<br><br>\left[i\right]=\ln <br><br>\left(\mathrm{prob}<br><br>\left[q\right]<br><br>\left[i\right]\right)+\ln \left(\frac{{\mathrm{<br><br>}}^{\ln <br><br><br><br>\mathrm{likelihood}<br><br>\left[q\right]<br><br>\left[i\right]}}{\underset{j=1}{\overset{N}{\sum <br><br>}}<br><br>{\mathrm{<br><br>}}^{\ln <br><br><br><br>\mathrm{likelihood}<br><br>\left[q\right]<br><br>\left[j\right]}}\right)$

wherein feature[q][i] denotes reliability according to a probability that the phoneme indicated by the q^th phoneme interval of the entire detected phoneme intervals corresponds to the i^th phoneme of N phonemes included in the phoneme model,

prob[q][i] denotes a probability that the phoneme indicated by the q^th phoneme interval of the entire detected phoneme intervals corresponds to the i^th phoneme of N phonemes included in the phoneme model,

e^{lnlikelihood[q][i]}=likelihood[q][i] denotes a likelihood between the phoneme indicated by the q^th phoneme interval of the entire detected phoneme intervals and the i^th phoneme of N phonemes included in the phoneme model, and

$\underset{j=1}{\overset{N}{\sum <br><br>}}<br><br>{\mathrm{<br><br>}}^{\ln <br><br><br><br>\mathrm{likelihood}<br><br>\left[q\right]<br><br>\left[j\right]}=\underset{j=1}{\overset{N}{\sum <br><br>}}<br><br>\mathrm{likelihood}<br><br>\left[q\right]<br><br>\left[j\right]$

denotes a sum of the likelihoods between the phoneme indicated by the q^th phoneme interval of the entire detected phoneme intervals and each phoneme of N phonemes included in the phoneme model.

8. The method of claim 6, wherein the phoneme alignment cost(cost feature[q]|W_P)) is calculated by the following equation:

$\mathrm{cost}<br><br>\left(\mathrm{feature}<br><br>\left[q\right]<br><br>W_P\right)=-\ln <br><br>\left(\underset{i=1}{\overset{N}{\sum <br><br>}}<br><br>\left({\mathrm{<br><br>}}^{\mathrm{feature}<br><br>\left[q\right]<br><br>\left[i\right]}\times <br><br>{\mathrm{<br><br>}}^{W_P<br><br>\left[i\right]}\right)\right)$

wherein feature[q] denotes a reliability vector having reliability elements according to probabilities that the phoneme indicated by the q^th phoneme interval of the entire detected phoneme intervals corresponds to each phoneme of N phonemes included in the phoneme model

W_P denotes a phoneme recognition probability distribution that is pre-trained with respect to the phoneme p included in the phoneme model,

W_P[i] denotes an average probability value of an i^th phoneme of the phoneme recognition probability distribution that is pre-trained with respect to the phoneme p included in the phoneme model, and

feature[q][i] denotes reliability according to a probability that the phoneme indicated by the q^th phoneme interval of the entire detected phoneme intervals corresponds to the i^th phoneme of N phonemes included in the phoneme model.

9. The method of claim 1, further comprising the step of smoothing the phoneme alignment cost by taking into account at least one of accuracy and noise environment of the phoneme interval detection, and a difference between evaluation and training environments for calculating the phoneme recognition probability distribution.

10. The method of claim 5, wherein the phoneme alignment cost (cost(feature[q]|W_P)) is calculated by the following equation:

$\mathrm{cost}<br><br>\left(\mathrm{feature}<br><br>\left[q\right]<br><br>W_P\right)=-\ln <br><br>\left(\underset{i=1}{\overset{N}{\sum <br><br>}}<br><br>\left({\left(\mathrm{feature}<br><br>\left[q\right]<br><br>\left[i\right]\right)}^{\mathrm{\alpha <br><br>}}\times <br><br>{\left(W_P<br><br>\left[i\right]\right)}^{\mathrm{\beta <br><br>}}\right)\right)$

wherein feature[q] denotes a reliability vector having reliability elements according to probabilities that the phoneme indicated by the q^th phoneme interval of the entire detected phoneme intervals corresponds to each phoneme of N phonemes included in the phoneme model,

W_P denotes a phoneme recognition probability distribution that is pre-trained with respect to the phoneme p included in the phoneme model,

W_P[i] denotes an average probability value of the i^th phoneme of the phoneme recognition probability distribution that is pre-trained with respect to the phoneme p included in the phoneme model,

feature[q][i] denotes reliability according to a probability that a phoneme indicated by the q^th phoneme interval of the entire detected phoneme intervals corresponds to the i^th phoneme of N phonemes included in the phoneme model,

α

denotes a parameter reflecting noise environment and accuracy of the phoneme interval detection, and

β

denotes a parameter reflecting difference between evaluation and training environments for calculating the phoneme recognition probability distribution.

11. The method of claim 8, wherein the phoneme alignment cost cost(feature[q]|W_P) is calculated by the following equation:

$\mathrm{cost}<br><br>\left(\mathrm{feature}<br><br>\left[q\right]<br><br>W_P\right)=-\ln <br><br>\left(\underset{i=1}{\overset{N}{\sum <br><br>}}<br><br>\left({\left({\mathrm{<br><br>}}^{\mathrm{feature}<br><br>\left[q\right]<br><br>\left[i\right]}\right)}^{\mathrm{\alpha <br><br>}}\times <br><br>{\left({\mathrm{<br><br>}}^{W_P<br><br>\left[i\right]}\right)}^{\mathrm{\beta <br><br>}}\right)\right)$

wherein feature[q] denotes a reliability vector having reliability elements according to probabilities that the phoneme indicated by the q^th phoneme interval of the entire detected phoneme intervals corresponds to each phoneme included in the phoneme model comprising N phonemes,

W_P denotes a phoneme recognition probability distribution that is pre-trained with respect to the phoneme p included in the phoneme model,

W_P[i] denotes an average probability value of the i^th phoneme of the phoneme recognition probability distribution that is pre-trained with respect to the phoneme p included in the phoneme model,

feature[q][i] denotes reliability according to a probability that the phoneme indicated by the q^th phoneme interval of the entire detected phoneme intervals corresponds to the i^th phoneme of N phonemes included in a phoneme model,

α

denotes a parameter reflecting noise environment and accuracy of the phoneme interval detection, and

β

denotes a parameter reflecting a difference between evaluation and training environments for calculating the phoneme recognition probability distribution.

12. The method of claim 1, further comprising the step of calculating the phoneme recognition probability distribution by phonetically receiving phoneme sequences for calculating the phoneme recognition probability distribution and accumulating determination results that a phoneme included in the phonetically input phoneme sequences is recognized as a phoneme among a plurality of phonemes that are predefined.

13. The method of claim 12, wherein the step of determining that a phoneme included in the phonetically input phoneme sequences is recognized as a phoneme among a plurality of phonemes that are predefined comprises a step of calculating a cost for aligning the phonetically input phoneme sequences with respect to answer phoneme sequences, so that a phoneme that requires the lowest cost is recognized as the phoneme.

14. An apparatus for recognizing speech, comprising:

a phoneme interval detector for detecting each phoneme interval by determining a boundary between phonemes included in phonetically input character sequences;

a reliability determination unit for calculating reliability according to probabilities that a phoneme indicated by each detected phoneme interval corresponds to each phoneme included in a predefined phoneme model;

a reliability-based phoneme error model for storing a phoneme recognition probability distribution obtained by pre-training that a phonetically input phoneme is recognized as a phoneme; and

a word recognition unit for calculating a phoneme alignment cost with respect to the character sequences based on the calculated reliability and the phoneme recognition probability distribution, and performing phoneme alignment based on the calculated phoneme alignment cost to perform speech recognition with respect to the character sequences.

15. The apparatus of claim 14, wherein the reliability determination unit calculates a likelihood between the phoneme indicated by each phoneme interval and each phoneme included in the phoneme model, and calculates the reliability based on the calculated likelihood.

16. The apparatus of claim 15, wherein the word recognition unit calculates the reliability(feature[q][i]) by the following equation:

$\mathrm{feature}<br><br>\left[q\right]<br><br>\left[i\right]=\mathrm{prob}<br><br>\left[q\right]<br><br>\left[i\right]=\frac{{\mathrm{<br><br>}}^{\ln <br><br><br><br>\mathrm{likelihood}<br><br>\left[q\right]<br><br>\left[i\right]}}{\underset{j=1}{\overset{N}{\sum <br><br>}}<br><br>{\mathrm{<br><br>}}^{\ln <br><br><br><br>\mathrm{likelihood}<br><br>\left[q\right]<br><br>\left[j\right]}}$

wherein feature[q][i] denotes reliability according to a probability that the phoneme indicated by the q^th phoneme interval of the entire detected phoneme intervals corresponds to the i^th phoneme of N phonemes included in the phoneme model,

prob[q][i] denotes a probability that the phoneme indicated by the q^th phoneme interval of the entire detected phoneme intervals is the i^th phoneme of N phonemes included in the phoneme model,

e^{lnlikelihood[q][i]}=likelihood[q][i] denotes a likelihood between the phoneme indicated by the q^th phoneme interval of the entire detected phoneme intervals and the i^th phoneme of N phonemes included in the phoneme model, and

$\underset{j=1}{\overset{N}{\sum <br><br>}}<br><br>{\mathrm{<br><br>}}^{\ln <br><br><br><br>\mathrm{likelihood}<br><br>\left[q\right]<br><br>\left[j\right]}=\underset{j=1}{\overset{N}{\sum <br><br>}}<br><br>\mathrm{likelihood}<br><br>\left[q\right]<br><br>\left[j\right]$

denotes a sum of the likelihoods between the phoneme indicated by the q^th phoneme interval of the entire detected phoneme intervals and each phoneme of N phonemes included in the phoneme model.

17. The apparatus of claim 14, wherein the reliability determination unit calculates the reliability(feature[q][i]) by the following equation:

$\mathrm{feature}<br><br>\left[q\right]<br><br>\left[i\right]=\ln <br><br>\left(\mathrm{prob}<br><br>\left[q\right]<br><br>\left[i\right]\right)=\ln \left(\frac{{\mathrm{<br><br>}}^{\ln <br><br><br><br>\mathrm{likelihood}<br><br>\left[q\right]<br><br>\left[i\right]}}{\underset{j=1}{\overset{N}{\sum <br><br>}}<br><br>{\mathrm{<br><br>}}^{\ln <br><br><br><br>\mathrm{likelihood}<br><br>\left[q\right]<br><br>\left[j\right]}}\right)$

wherein feature[q][i] denotes reliability according to a probability that the phoneme indicated by the q^th phoneme interval of the entire detected phoneme intervals corresponds to the i^th phoneme of N phonemes included in the phoneme model,

prob[q][i] denotes a probability that the phoneme indicated by the q^th phoneme interval of the entire detected phoneme intervals corresponds to the i^th phoneme of N phonemes included in the phoneme model,

e^{lnlikelihood[q][i]}=likelihood[q][i] denotes a likelihood between a phoneme that a q^th phoneme interval of the entire detected phoneme intervals indicates and an i^th phoneme of N phonemes included in the phoneme model, and

$\underset{j=1}{\overset{N}{\sum <br><br>}}<br><br>{\mathrm{<br><br>}}^{\ln <br><br><br><br>\mathrm{likelihood}<br><br>\left[q\right]<br><br>\left[j\right]}=\underset{j=1}{\overset{N}{\sum <br><br>}}<br><br>\mathrm{likelihood}<br><br>\left[q\right]<br><br>\left[j\right]$

denotes a sum of the likelihoods between the phoneme indicated by the q^th phoneme interval of the entire detected phoneme intervals and each phoneme of N phonemes included in the phoneme model.

18. The apparatus of claim 17, wherein the word recognition unit calculates the phoneme alignment cost(cost(feature[q]|W_P)) by the following equation:

$\mathrm{cost}<br><br>\left(\mathrm{feature}<br><br>\left[q\right]<br><br>W_P\right)=-\ln <br><br>\left(\underset{i=1}{\overset{N}{\sum <br><br>}}<br><br>\left({\mathrm{<br><br>}}^{\mathrm{feature}<br><br>\left[q\right]<br><br>\left[i\right]}\times <br><br>{\mathrm{<br><br>}}^{W_P<br><br>\left[i\right]}\right)\right)$

wherein feature[q] denotes a reliability vector having reliability elements according to probabilities that the phoneme indicated by the q^th phoneme interval of the entire detected phoneme intervals corresponds to each phoneme of N phonemes included in the phoneme model,

W_P denotes a phoneme recognition probability distribution that is pre-trained with respect to the phoneme p included in the phoneme model,

W_P[i] denotes an average probability value of the i^th phoneme of the phoneme recognition probability distribution that is pre-trained with respect to the phoneme p included in the phoneme model, and

feature[q][i] denotes reliability according to a probability that the phoneme indicated by the q^th phoneme interval of the entire detected phoneme intervals corresponds to the i^th phoneme of N phonemes included in the phoneme model.

19. The apparatus of claim 14, wherein the word recognition unit performs smoothing on the phoneme alignment cost by taking into account at least one of performance of the phoneme interval detector, noise environment and a difference between the evaluation environment and training environment of the reliability-based phoneme error model.

20. The apparatus of claim 18, wherein the word recognition unit calculates the phoneme alignment cost(cost(feature[q]|W_P)) by the following equation:

$\mathrm{cost}<br><br>\left(\mathrm{feature}<br><br>\left[q\right]<br><br>W_P\right)=-\ln <br><br>\left(\underset{i=1}{\overset{N}{\sum <br><br>}}<br><br>\left({\left({\mathrm{<br><br>}}^{\mathrm{feature}<br><br>\left[q\right]<br><br>\left[i\right]}\right)}^{\mathrm{\alpha <br><br>}}\times <br><br>{\left({\mathrm{<br><br>}}^{W_P<br><br>\left[i\right]}\right)}^{\mathrm{\beta <br><br>}}\right)\right)$

wherein feature[q] denotes a reliability vector having reliability elements according to probabilities that the phoneme indicated by the q^th phoneme interval of the entire detected phoneme intervals corresponds to each phoneme of N phonemes included in the phoneme model,

W_P denotes a phoneme recognition probability distribution that is pre-trained with respect to the phoneme p included in the phoneme model,

W_P[i] denotes an average probability value of the i^th phoneme of the phoneme recognition probability distribution that is pre-trained with respect to the phoneme p included in the phoneme model,

feature[q][i] denotes reliability according to a probability that the phoneme indicated by the q^th phoneme interval of the entire detected phoneme intervals corresponds to the i^th phoneme of N phonemes included in the phoneme model,

α

denotes a parameter reflecting noise environment and performance of the phoneme interval detector, and

β

denotes a parameter reflecting a difference between the evaluation and training environments for calculating a phoneme recognition probability distribution.