Method for assessing learner's pronunciation through voice and image

• US 20080004879A1
• Filed: 06/29/2006
• Published: 01/03/2008
• Est. Priority Date: 06/29/2006
• Status: Abandoned Application

• Current Assignee: National Kaohsiung First University of Science and Technology
• Original Assignee: National Kaohsiung First University of Science and Technology
• Inventors: Huang, Wen-Chen
• Application Number:
  US11/476,716
• Publication Number:
  US 20080004879A1
• Time in Patent Office: Days
• US Class Current: 704/270
• CPC Class Codes: G09B 19/06 Foreign languages with audible presentation of material to be studied G09B5/04
• CPC Class Codes: G10L 15/142 Hidden Markov Models [HMMs]
• CPC Class Codes: G10L 2015/025 Phonemes, fenemes or fenones being the recognition units

Abstract

The present invention relates to a method for assessing a learner'"'"'s pronunciation through voices and images, in which variation of the teacher'"'"'s and the learner'"'"'s lips are compared in a visual manner. Accordingly, incorrect pronunciation can be pointed out and recorded for assessment and rectification.

Claims

1. A method for assessing a learner'"'"'s pronunciation through voices and images, comprising steps of:

(1) adding a new word or sentence by a teacher via an interface, capturing the teacher'"'"'s lip with a WebCam, and storing lip images and corresponding acoustic signals in a database;

(2) selecting and speaking a word or a sentence from the database by the learner;

(3) capturing the learner'"'"'s lip images with a WebCam;

(4) automatically finding a lip zone by distinguishing colors of the learner'"'"'s face with color space conversion and then dividing the face into several pixels, eliminating spots, computing the image with an algorithm, and defining a range of the learner'"'"'s face so as to searching a boundary of the lip;

(5) assessing the learner'"'"'s pronunciation by comparing the captured lip images and voices with those built in the database.

2. The method of claim 1, wherein the color space conversion for defining the learner'"'"'s face in the step (4) is a YCbCr system.

3. The method of claim 1, wherein the colors on the learner'"'"'s face are distinguished to determine top, bottom, right and left boundaries, and then the right and left boundaries are moved inward about one eighth respectively as critical lines of the lip zone.

4. The method of claim 3, wherein the learner'"'"'s lip is divided with a RGB color system, in which a ratio R/G is limited as follows:

5. The method of claim 3, wherein the learner'"'"'s lip is divided with a HSV color system, in which thresholds of H (hue), S (saturation) and V (value) are preset for converting pixels into 1 (within the thresholds) or 0 (beyond the thresholds) and thus forming a binary image;

pixels of the binary image is then numbered with “

connected component analysis”

, wherein the pixels of the most numbers is determined as the lip zone, and then undesired spots on the binary image are eliminated with morphology operation and median filter.

6. The method of claim 3, wherein the learner'"'"'s lip is divided with a RGB color system and a HSV color system, wherein:

in the RGB system, a ratio R/G is limited as follows;

7. The method of claim 7, 10 or 13, wherein the binary image is processed with morphology operation and median filter to eliminate undesired spots.

8. The method of claim 1, wherein the step (4) utilizes “

connected component analysis”

for computation to distinguish a pixel and its neighboring pixels, and particularly give different numbers to pixels of different features.

9. The method of claim 1, wherein the step (5) utilizes “

dynamic time warping (DTW)” and

“

pattern matching”

to compare a teacher'"'"'s and a learner'"'"'s lip images by deleting useless images and remaining useful image for assessment.

10. The method of claim 1, wherein the step (5) utilizes proportional contours for assessment, in which differences of assessed images and standard images are summarized:

11. The method of claim 1, wherein the step (5) is to process acoustic signals by converting analog acoustic signals into digital signals through an input device, and then extracting features of the signals for assessment.

12. The method of claim 11, wherein the step (5) for processing acoustic signals includes sub-steps of:

(1) sampling analog signals—

converting the analog signals into digital signals;

(2) detecting endpoints—

deleting silence on two ends;

(3) extracting features—

combining proper features into feature vectors as basises of assessment;

(4) pattern matching assessment—

comparing assessed speech and standard speech phoneme by phoneme to find their difference for assessment.

13. The method of claim 12, wherein the feature is extracted with linear prediction coding (LPC) and cepstrum coefficient.

14. The method of claim 12, wherein the endpoints are detected with short-term energy and zero crossing rate, and the silence is judged with a threshold.

15. The method of claim 12, wherein pattern matching uses volume strength curve, pitch contour, Mel-frequency cepstral coefficients to adjust parameters, and then a minimum average deviation of dynamic time warping (DTW) is calculated for assessment.

16. The method of claim 12, further comprising speech recognition by utilizing an acoustic model derived from hidden Markov model (HMM) which includes a hidden random process and an observation sequence for describing a probability distribution of all features with “

state observation probability”

.