Automatic speech segmentation and verification method and system

• US 20050060151A1
• Filed: 02/23/2004
• Published: 03/17/2005
• Est. Priority Date: 09/12/2003
• Status: Active Grant

• Current Assignee: Industrial Technology Research Institute
• Original Assignee: Industrial Technology Research Institute
• Inventors: Kuo, Chi-Shiang, Kuo, Chih-Chung, Chen, Jau-Hung
• Granted Patent: US 7,472,066 B2
• Time in Patent Office: Days
• US Class Current: 704/240
• CPC Class Codes: G10L 13/06 Elementary speech units used in speech synthesisers; Concatenation rules
• CPC Class Codes: G10L 15/04 Segmentation; Word boundary detection

Abstract

An automatic speech segmentation and verification system and method is disclosed, which has a known text script and a recorded speech corpus corresponding to the known text script. A speech unit segmentor segments the recorded speech corpus into N test speech unit segments referring to the phonetic information of the known text script. Then, a segmental verifier is applied to obtain a confidence measure of syllable segmentation for verifying the correctness of the cutting points of test speech unit segments. A phonetic verifier obtains a confidence measure of syllable verification by using verification models for verifying whether the recorded speech corpus is correctly recorded. Finally, a speech unit inspector integrates the confidence measure of syllable segmentation and the confidence measure of syllable verification to determine whether the test speech unit segment is accepted or not.

Claims

1. An automatic speech segmentation and verification method comprising:

a retrieving step, for retrieving a recorded speech corpus, the recorded speech corpus corresponding to a known text script, the known text script defining phonetic information with N phonetic units;

a segmenting step, for segmenting the recorded speech corpus into N test speech unit segments referring to the phonetic information of the N phonetic units in the known text script;

a segment-confidence-measure verifying step, for verifying segment confidence measures of N cutting points of the test speech unit segments to determine if the N cutting points of the test speech unit segments are correct;

a phonetic-confidence-measure verifying step, for verifying phonetic confidence measures of the test speech unit segments to determine if the test speech unit segments correspond to the known text script; and

a determining step, for determining acceptance of the phonetic unit by comparing a combination of segment reliability and the phonetic confidence measures of the test speech unit segments to a predetermined threshold value;

wherein if the combined confidence measure is greater than the predetermined threshold value, the phonetic is accepted.

2. The method as claimed in claim 1, wherein the segmenting step further comprises:

using a hidden Markov model (HMM) to cut the recorded speech corpus into N test speech unit segments referring to the phonetic information of the N phonetic units in the known text script, wherein each test speech unit segment is defined as correspondingly having an initial cutting point;

performing a fine adjustment on the initial cutting point of the test speech unit segment according to at least one feature factor corresponding to each test speech unit segment and calculating at least one cutting point fine adjustment value corresponding to each test speech unit segment; and

integrating the initial cutting point and the cutting point fine adjustment value of the test speech unit segment to obtain a cutting point of the test speech unit segment.

3. The method as claimed in claim 2, wherein the feature factor of the test speech unit segment is a neighboring cutting point of the initial cutting point.

4. The method as claimed in claim 2, wherein the feature factor of the test speech unit segment is a zero crossing rate (ZCR) of the test speech unit segment.

5. The method as claimed in claim 2, wherein the feature factor of the test speech unit segment is an energy value of the test speech unit segment.

6. The method as claimed in claim 5, wherein the energy value is an energy value of a band pass signal and a high pass signal retrieved from a speaker-dependent band.

7. The method as claimed in claim 2, wherein each cutting point fine adjustment value has a weighted value, and the cutting point of the test speech unit segment is a weighted average of the initial cutting point and the cutting point fine adjustment value.

8. The method as claimed in claim 1, wherein in the segment-confidence-measure step, each segment confidence measure of the test speech unit segment is:

$\mathrm{CMS}=\max <br><br>\left(1-h<br><br>\left(D\right)-\underset{s,f}{\overset{\text{\hspace{1em}<br><br>}}{\sum <br><br>}}<br><br>g<br><br>\left(c<br><br>\left(s\right),f<br><br>\left(s\right)\right),0\right),$

where

$h<br><br>\left(D\right)=K<br><br>\left(\underset{i}{\overset{\text{\hspace{1em}<br><br>}}{\sum <br><br>}}<br><br>w_i<br><br><br><br>d_i-\stackrel{\_}{d}<br><br>\right),$

D is a vector of multiple expert decisions of the cutting point, d_i is the cutting point, {overscore (d)}=p(D) is a final decision of the cutting point, K(x) is a monotonically increasing function that maps a non-negative variable x into a value between 0 and 1, g(c(s), f(s)) is a cost function value between a cost function ranging from 0 to 1, s is a segment, c(s) is a type category of the segment s and, f(s) are acoustic features of the segment.

9. The method as claimed in claim 1, wherein in the phonetic-confidence-measure step, each phonetic confidence measure of the test speech unit segments is:

CMV=min{LLR₁,LLR_F,0}, $\mathrm{where},\{\begin{array}{c}{\mathrm{LLR}}_I=\log <br><br>\text{\hspace{1em}<br><br>}<br><br>P<br><br>\left(X_I|H_0\right)-\log <br><br>\text{\hspace{1em}<br><br>}<br><br>P<br><br>\left(X_I|H_1\right)\\ {\mathrm{LLR}}_F=\log <br><br>\text{\hspace{1em}<br><br>}<br><br>P<br><br>\left(X_F|H_0\right)-\log <br><br>\text{\hspace{1em}<br><br>}<br><br>P<br><br>\left(X_F|H_1\right)\end{array},$

X₁ is an initial segment of the test speech unit segment, X_F is a final segment of the test speech unit segment, H₀ is a null hypothesis of the test speech unit segment recorded correctly, H₁ is an alternative hypothesis of the test speech unit segment recorded incorrectly, and LLR is a log likelihood ratio.

10. An automatic speech segmentation and verification system comprising:

a database for storing a known text script and a recorded speech corpus corresponding to the known text script, and the known text script has phonetic information with N speech unit segment wherein N is a positive integer;

a speech unit segmentor for segmenting the recorded speech corpus into N test speech unit segments referring to the phonetic information of the known text script;

a segmental verifier for verifying the correctness of the cutting points of test speech unit segments by obtaining a segmental confidence measure;

a phonetic verifier for obtaining a confidence measure of syllable verification by using verification models for verifying whether the recorded speech corpus is correctly recorded; and

a speech unit inspector for integrating the confidence measure of syllable segmentation and the confidence measure of syllable verification to determine whether the test speech unit segment is accepted.

11. The system as claimed in claim 10, wherein the segmental verifier performs the following steps:

using a hidden Markov model (HMM) to cut the recorded speech corpus into N test speech unit segments referring to the phonetic information of the N phonetic units in the known text script, wherein each test speech unit segment is defined as correspondingly having an initial cutting point;

performing a fine adjustment on the initial cutting point of the test speech unit segment according to at least one feature factor corresponding to each test speech unit segment and calculating at least one cutting point fine adjustment value corresponding to each test speech unit segment; and

integrating the initial cutting point and the cutting point fine adjustment value of the test speech unit segment to obtain a cutting point of the test speech unit segment.

12. The system as claimed in claim 11, wherein the feature factor of the test speech unit segment is a neighboring cutting point of the initial cutting point.

13. The system as claimed in claim 11, wherein the feature factor of the test speech unit segment is a zero crossing rate (ZCR) of the test speech unit segment.

14. The system as claimed in claim 11, wherein the feature factor of the test speech unit segment is an energy value of the test speech unit segment.

15. The system as claimed in claim 14, wherein the energy value is an energy value of a band pass signal and a high pass signal retrieved from a speaker-dependent band.

16. The system as claimed in claim 11, wherein each cutting point fine adjustment value has a weighted value, and the cutting point of the test speech unit segment is a weighted average of the initial cutting point and the cutting point fine adjustment value.

17. The system as claimed in claim 10, wherein each segment confidence measure of the test speech unit segment is determined by:

$\mathrm{CMS}=\max <br><br>\left(1-h<br><br>\left(D\right)-\underset{s,f}{\overset{\text{\hspace{1em}<br><br>}}{\sum <br><br>}}<br><br>g<br><br>\left(c<br><br>\left(s\right),f<br><br>\left(s\right)\right),0\right),$

where

$h<br><br>\left(D\right)=K<br><br>\left(\underset{i}{\overset{\text{\hspace{1em}<br><br>}}{\sum <br><br>}}<br><br>w_i<br><br><br><br>d_i-\stackrel{\_}{d}<br><br>\right),$

D is the vector of multiple expert decisions of the cutting point, d_i is the cutting point, {overscore (d)}=p(D) is a final decision of the cutting point, K(x) is a monotonically increasing function that maps a non-negative variable x into a value between 0 and 1, g(c(s), f(s)) is a cost function value between a cost function ranging from 0 to 1, s is a segment, c(s) is the type category of the segment s and, f(s) is the acoustic feature of the segment.

18. The method as claimed in claim 10, wherein each phonetic confidence measure of the test speech unit segments is determined by:

CMV=min{LLR₁,LLR_F,0}, $\mathrm{where},\{\begin{array}{c}{\mathrm{LLR}}_I=\log <br><br>\text{\hspace{1em}<br><br>}<br><br>P<br><br>\left(X_I|H_0\right)-\log <br><br>\text{\hspace{1em}<br><br>}<br><br>P<br><br>\left(X_I|H_1\right)\\ {\mathrm{LLR}}_F=\log <br><br>\text{\hspace{1em}<br><br>}<br><br>P<br><br>\left(X_F|H_0\right)-\log <br><br>\text{\hspace{1em}<br><br>}<br><br>P<br><br>\left(X_F|H_1\right)\end{array},$

X₁ is initial segment of the test speech unit segment, X_F is final segment of the test speech unit segment, H₀ is a null hypothesis of the test speech unit segment recorded correctly, H₁ is an alternative hypothesis of the test speech unit segment recorded incorrectly, and LLR is a log likelihood ratio.