Language learning system, language learning method, and computer program product thereof

• US 8,870,575 B2
• Filed: 10/08/2010
• Issued: 10/28/2014
• Est. Priority Date: 08/03/2010
• Status: Active Grant

• Current Assignee: Industrial Technology Research Institute
• Original Assignee: Industrial Technology Research Institute
• Inventors: Chang, Sen-Chia, Liao, Hsien-Cheng, Chen, Jiang-Chun
• Primary Examiner(s): Utama, Robert J
• Application Number:
  US12/900,482
• Publication Number:
  US 20120034581A1
• Time in Patent Office: 1,481 Days
• Field of Search: 434/156, 434/157, 434/185
• US Class Current: 434/157
• CPC Class Codes: G09B 19/06 Foreign languages with audible presentation of material to be studied G09B5/04

Abstract

A language learning system including a storage module, a feature extraction module, and an assessment and diagnosis module is provided. The storage module stores training data and an assessment decision tree generated according to the training data. The feature extraction module extracts pronunciation features of a pronunciation given by a language learner. The assessment and diagnosis module identifies a diagnosis path corresponding to the pronunciation of the language learner in the assessment decision tree and outputs feedback information corresponding to the diagnosis path. Thereby, the language learning system can assess and provide feedback information regarding words, phrases or sentences pronounced by the language learner.

Claims

1. A language learning system, for assessing a pronunciation in a learning sentence, the language learning system comprising:

a storage module, configured for storing a plurality of training data and at least one assessment decision tree generated according to the training data, wherein the assessment decision tree has a plurality of decision paths, each of the decision paths comprises a plurality of decision nodes and is corresponding to at least one feedback information, and the decision paths and decision nodes on the assessment decision tree represent a specific type of pronunciations having a plurality of predefined types of tones;

a feature extraction module, configured for extracting at least one pronunciation feature of the pronunciation, wherein the feature extraction module is further configured for extracting at least one pronunciation feature of each of the training data, wherein the training data has at least one grade mark,

a decision tree generation module, configured for generating the assessment decision tree according to the pronunciation features and the grade marks of the training data,

a feedback information generation module, configured for analyzing each of the decision paths of the assessment decision tree according to the pronunciation features corresponding to the decision nodes on the decision path to identify incorrect pronunciation types represented by the decision nodes and setting the feedback information corresponding to the decision path according to the incorrect pronunciation types represented by the decision nodes; and

an assessment and diagnosis module, is configured for determining a diagnosis path corresponding to the pronunciation among the decision paths of the assessment decision tree according to the pronunciation feature of the pronunciation and outputting the feedback information corresponding to the diagnosis path to correct at least one incorrect pronunciation in the learning sentence.

2. The language learning system according to claim 1, wherein the feature extraction module performs a phonetic segmentation operation on a plurality of training sentences to obtain a plurality of pronunciation units of the training sentences, and the feature extraction module obtains the training data from the pronunciation units of the training sentences,

wherein the feature extraction module performs the phonetic segmentation operation on the learning sentence to obtain one or more pronunciation units of the learning sentence.

3. The language learning system according to claim 2, wherein the feature extraction module extracts a fundamental frequency contour of the pronunciation, divides the fundamental frequency contour of the pronunciation into a plurality of sub fundamental frequency contours of the pronunciation, calculates a mean value of each of the sub fundamental frequency contours of the pronunciation, calculates a slope of each of the sub fundamental frequency contours of the pronunciation, calculates a plurality of difference values between the sub fundamental frequency contours of the pronunciation, and serves at least one of the mean values, the slopes, and the difference values of the sub fundamental frequency contours of the pronunciation as the pronunciation feature of the pronunciation,

wherein the feature extraction module extracts a fundamental frequency contour of each of the training data, divides the fundamental frequency contour of the training data into a plurality of sub fundamental frequency contours, calculates a mean value of each of the sub fundamental frequency contours of the training data, calculates a slope of each of the sub fundamental frequency contours of the training data, calculates a plurality of difference values between the sub fundamental frequency contours of the training data, and serves at least one of the mean values, the slopes, and the difference values of the sub fundamental frequency contours of the training data as the pronunciation feature of the training data.

4. The language learning system according to claim 2, wherein the feature extraction module extracts an energy contour of the pronunciation, divides the energy contour of the pronunciation into a plurality of sub energy contours of the pronunciation, calculates a mean value of each of the sub energy contours of the pronunciation, calculates a slope of each of the sub energy contours of the pronunciation, calculates a plurality of difference values between the sub energy contours of the pronunciation, and serves at least one of the mean values, the slopes, and the difference values of the sub energy contours of the pronunciation as the pronunciation feature of the pronunciation,

wherein the feature extraction module extracts an energy contour of each of the training data, divides the energy contour of the training data into a plurality of sub energy contours, calculates a mean value of each of the sub energy contours of the training data, calculates a slope of each of the sub energy contours of the training data, calculates a plurality of difference values between the sub energy contours of the training data, and serves at least one of the mean values, the slopes, and the difference values of the sub energy contours of the training data as the pronunciation feature of the training data.

5. The language learning system according to claim 2, wherein the feature extraction module obtains a plurality of phonemes of the pronunciation to one or more mel-scale frequency cepstral coefficients (MFCCs) corresponding to the phonemes of the pronunciation, calculates a MFCC mean value of each of the phonemes, calculates a plurality of MFCC difference values between the MFCC mean values of the pronunciation, and serves at least one of the MFCC mean values and the MFCC difference values of the pronunciation as the pronunciation feature of the pronunciation,

wherein the feature extraction module extracts a plurality of MFCCs corresponding to the phonemes, calculates a MFCC mean value of each of the phonemes of the training data, calculates a plurality of MFCC difference values between the MFCC mean values of the training data, and serves at least one of the MFCC mean values and the MFCC difference values of the training data as the pronunciation feature of the training data.

6. The language learning system according to claim 2, wherein the feature extraction module extracts a plurality of durations corresponding to the phonemes of the pronunciation, calculates a mean duration of each of the phonemes, calculates a plurality of duration differences between the mean durations in the pronunciation, and serves at least one of the mean durations and the duration differences of the pronunciation as the pronunciation feature of the pronunciation,

wherein the feature extraction module extracts a plurality of phonemes of each of the training data and a plurality of durations corresponding to the phonemes, calculates a mean duration of each of the phonemes of the training data, calculates a plurality of duration differences between the mean durations in the training data, and serves at least one of the mean durations and the duration differences of the training data as the pronunciation feature of the training data.

7. The language learning system according to claim 2, wherein the feature extraction module extracts at least one fundamental frequency contour, at least one energy contour, at least one MFCC, and at least one duration corresponding to the pronunciation and serves at least one of the fundamental frequency contour, the energy contour, the MFCC, and the duration as the pronunciation feature of the pronunciation,

wherein the feature extraction module extracts at least one fundamental frequency contour, at least one energy contour, at least one MFCC, and at least one duration corresponding to each of the training data and serves at least one of the fundamental frequency contour, the energy contour, the MFCC, and the duration as the pronunciation feature of the training data.

8. The language learning system according to claim 2, wherein the feature extraction module groups the training data into a plurality of training data groups according to combinations of each pronunciation unit and the next pronunciation unit in the training sentences,

wherein the decision tree generation module generates the assessment decision trees according to the training data groups, and each of the assessment decision trees is corresponding to one of the training data groups.

9. The language learning system according to claim 2, wherein the feature extraction module groups the training data into a plurality of training data groups according to combinations of each pronunciation unit and the previous pronunciation unit in the training sentences,

wherein the decision tree generation module generates the assessment decision trees according to the training data groups, and each of the assessment decision trees is corresponding to one of the training data groups.

10. The language learning system according to claim 2, wherein the feature extraction module groups the training data into a plurality of training data groups according to combinations of each pronunciation unit, the next pronunciation unit and the previous pronunciation unit in the training sentences,

wherein the decision tree generation module generates the assessment decision trees according to the training data groups, and each of the assessment decision trees is corresponding to one of the training data groups.

11. A language learning method, for assessing a pronunciation in a learning sentence, the language learning method comprising:

collecting a plurality of training data;

extracting at least one pronunciation feature of each of the training data, wherein the training data has at least one grade mark;

generating at least one assessment decision tree according to the pronunciation features and the grade marks of the training data, wherein the assessment decision tree has a plurality of decision paths, each of the decision paths comprises a plurality of decision nodes, and the decision paths and the decision nodes on the assessment decision tree represent a specific type of pronunciations having a plurality of predefined types of tones;

analyzing each of the decision paths of the assessment decision tree according to the pronunciation features corresponding to the decision nodes on each of the decision path to identify incorrect pronunciation types represented by the decision nodes and setting feedback information corresponding to the decision path according to the incorrect pronunciation types represented by the decision nodes;

extracting at least one pronunciation feature of the pronunciation;

determining a diagnosis path corresponding to the pronunciation among the decision paths of the assessment decision tree according to the pronunciation feature of the pronunciation; and

outputting the feedback information corresponding to the diagnosis path to correct at least one incorrect pronunciation in the learning sentence.

12. The language learning method according to claim 11, wherein the step of extracting the pronunciation feature of each of the training data comprises:

collecting a plurality of training sentences;

performing a phonetic segmentation operation on the training sentences to obtain a plurality of pronunciation units of the training sentences; and

obtaining the training data from the pronunciation units.

13. The language learning method according to claim 12 further comprising:

receiving the learning sentence; and

performing the phonetic segmentation operation on the learning sentence to obtain one or more pronunciation units of the learning sentence,

wherein the pronunciation is one of the pronunciation units of the learning sentence.

14. The language learning method according to claim 12,

wherein the step of extracting the pronunciation feature of the pronunciation comprises:

getting a fundamental frequency contour of the pronunciation;

dividing the fundamental frequency contour of the pronunciation into a plurality of sub fundamental frequency contours;

calculating a mean value of each of the sub fundamental frequency contours of the pronunciation;

calculating a slope of each of the sub fundamental frequency contours of the pronunciation;

calculating a plurality of difference values between the sub fundamental frequency contours of the pronunciation;

serving at least one of the mean values, the slopes, and the difference values of the sub fundamental frequency contours of the pronunciation as the pronunciation feature of the pronunciation,

wherein the step of extracting the pronunciation feature of each of the training data comprises;

getting a fundamental frequency contour of each of the training data;

dividing the fundamental frequency contour of each of the training data into a plurality of sub fundamental frequency contours;

calculating a mean value of each of the sub fundamental frequency contours of each of the training data;

calculating a slope of each of the sub fundamental frequency contours of each of the training data;

calculating a plurality of difference values between the sub fundamental frequency contours of each of the training data; and

serving at least one of the mean values, the slopes, and the difference values of the sub fundamental frequency contours of each of the training data as the pronunciation feature of the training data.

15. The language learning method according to claim 12,

wherein the step of extracting the pronunciation feature of the pronunciation comprises:

getting an energy contour of the pronunciation;

dividing the energy contour of the pronunciation into a plurality of sub energy contours;

calculating a mean value of each of the sub energy contours of the pronunciation;

calculating a slope of each of the sub energy contours of the pronunciation;

calculating a plurality of difference values between the sub energy contours of the pronunciation;

serving at least one of the mean values, the slopes, and the difference values of the sub energy contours of the pronunciation as the pronunciation feature of the pronunciation,

wherein the step of extracting the pronunciation feature of each of the training data comprises;

getting an energy contour of each of the training data;

dividing the energy contour of each of the training data into a plurality of sub energy contours;

calculating a mean value of each of the sub energy contours of each of the training data;

calculating a slope of each of the sub energy contours of each of the training data;

calculating a plurality of difference values between the sub energy contours of each of the training data; and

serving at least one of the mean values, the slopes, and the difference values of the sub energy contours of each of the training data as the pronunciation feature of the training data.

16. The language learning method according to claim 12,

wherein the step of extracting the pronunciation feature of the pronunciation comprises:

extracting a plurality of phonemes of the pronunciation and a plurality of MFCCs corresponding to the phonemes of the pronunciation;

calculating a MFCC mean value of each of the phonemes;

calculating a plurality of MFCC difference values between the MFCC mean values of the pronunciation; and

serving at least one of the MFCC mean values and the MFCC difference values of the pronunciation as the pronunciation feature of the pronunciation,

wherein the step of extracting the pronunciation features of the training data comprises;

extracting a plurality of phonemes of each of the training data and a plurality of MFCCs corresponding to the phonemes; and

calculating a MFCC mean value of each of the phonemes of each of the training data;

calculating a plurality of MFCC difference values between the MFCC mean values of each of the training data; and

serving at least one of the MFCC mean values and the MFCC difference values of each of the training data as the pronunciation feature of the training data.

17. The language learning method according to claim 12,

wherein the step of extracting the pronunciation feature of the pronunciation comprises:

extracting a plurality of phonemes of the pronunciation and a plurality of durations corresponding to the phonemes of the pronunciation;

calculating a mean duration of each of the phonemes;

calculating a plurality of duration differences between the mean durations in the pronunciation; and

serving at least one of the mean durations and the duration differences of the pronunciation as the pronunciation feature of the pronunciation,

wherein the step of extracting the pronunciation feature of each of the training data comprises;

extracting a plurality of phonemes of each of the training data and a plurality of durations corresponding to the phonemes;

calculating a mean duration of each of the phonemes of each of the training data;

calculating a plurality of duration differences between the mean durations in each of the training data; and

serving at least one of the mean durations and the duration differences of each of the training data as the pronunciation feature of the training data.

18. The language learning method according to claim 12,

wherein the step of extracting the pronunciation feature of the pronunciation comprises:

extracting at least one fundamental frequency contour, at least one energy contour, at least one MFCC, and at least one duration corresponding to the pronunciation; and

serving one of the fundamental frequency contour, the energy contour, the MFCC, and the duration as the pronunciation feature of the pronunciation,

wherein the step of extracting the pronunciation feature of each of the training data comprises;

extracting at least one fundamental frequency contour, at least one energy contour, at least one MFCC, and at least one duration corresponding to each of the training data; and

serving at least one of the fundamental frequency contour, the energy contour, the MFCC, and the duration of each of the training data as the pronunciation feature of the training data.

19. The language learning method according to claim 12 further comprising:

grouping the training data into a plurality of training data groups according to combinations of each pronunciation unit and the next pronunciation unit in the training sentences; and

generating the assessment decision trees according to the training data groups, wherein each of the assessment decision trees is corresponding to one of the training data groups.

20. The language learning method according to claim 12 further comprising:

grouping the training data into a plurality of training data groups according to combinations of each pronunciation unit and the previous pronunciation unit in the training sentences; and

generating the assessment decision trees according to the training data groups, wherein each of the assessment decision trees is corresponding to one of the training data groups.

21. The language learning method according to claim 12 further comprising:

grouping the training data into a plurality of training data groups according to combinations of each pronunciation unit, the next pronunciation unit and the previous pronunciation unit in the training sentences; and

generating the assessment decision trees according to the training data groups, wherein each of the assessment decision trees is corresponding to one of the training data groups.

22. A non-transitory program product, comprising at least one program instruction, wherein the at least one program instruction when executed by a computer system performing a language learning method comprising:

collecting a plurality of training data;

extracting at least one pronunciation feature of each of the training data, wherein the training data has at least one grade mark;

generating at least one assessment decision tree according to the pronunciation features and the grade marks of the training data, wherein the assessment decision tree has a plurality of decision paths, each of the decision paths comprises a plurality of decision nodes, and the decision paths and the decision nodes on the assessment decision tree represent a specific type of pronunciations having a plurality of predefined types of tones;

analyzing each of the decision paths of the assessment decision tree according to the pronunciation features corresponding to the decision nodes on each of the decision path to identify incorrect pronunciation types represented by the decision nodes and setting feedback information corresponding to the decision path according to the incorrect pronunciation types represented by the decision nodes;

storing at least one assessment decision tree;

extracting at least one pronunciation feature of the pronunciation;

determining a diagnosis path corresponding to the pronunciation among the decision paths of the assessment decision tree according to the pronunciation feature of the pronunciation; and

outputting the feedback information corresponding to the diagnosis path to correct at least one incorrect pronunciation in the learning sentence.