Systems and Methods for Assessment of Non-Native Spontaneous Speech

• US 20100145698A1
• Filed: 12/01/2009
• Published: 06/10/2010
• Est. Priority Date: 12/01/2008
• Status: Active Grant

• Current Assignee: Educational Testing Service
• Original Assignee: Educational Testing Service
• Inventors: Xi, Xiaoming, Chen, Lei, Zechner, Klaus
• Granted Patent: US 8,392,190 B2
• Time in Patent Office: Days
• US Class Current: 704/256.100
• CPC Class Codes: G09B 19/06 Foreign languages with audible presentation of material to be studied G09B5/04
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student
• CPC Class Codes: G10L 15/08 Speech classification or search
• CPC Class Codes: G10L 15/14 using statistical models, e.g. Hidden Markov Models [HMMs] G10L15/18 takes precedence
• CPC Class Codes: G10L 15/187 Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices

Abstract

Computer-implemented systems and methods are provided for assessing non-native spontaneous speech pronunciation. Speech recognition on digitized speech is performed using a non-native acoustic model trained with non-native speech to generate word hypotheses for the digitized speech. Time alignment is performed between the digitized speech and the word hypotheses using a reference acoustic model trained with native-quality speech. Statistics are calculated regarding individual words and phonemes in the word hypotheses based on the alignment. A plurality of features for use in assessing pronunciation of the speech are calculated based on the statistics, an assessment score is calculated based on one or more of the calculated features, and the assessment score is stored in a computer-readable memory.

Claims

1. A computer-implemented method of assessing spontaneous speech pronunciation, comprising:

performing speech recognition on digitized speech using a non-native acoustic model trained with non-native speech using a processing system to generate word hypotheses for the digitized speech;

performing time alignment between the digitized speech and the word hypotheses utilizing a reference acoustic model trained with native-quality speech to associate word hypotheses with corresponding sounds of the digitized speech;

calculating statistics regarding individual words and phonemes of the word hypotheses using the processing system based on said alignment;

calculating a plurality of features for use in assessing pronunciation of the digitized speech based on the statistics using the processing system;

calculating an assessment score based on one or more of the calculated features; and

storing the assessment score in a computer-readable memory.

2. The method of claim 1 further comprising excluding words not reliably recognized in generating the word hypotheses from contributing to the assessment score.

3. The method of claim 2, wherein a confidence level is associated with each word in the word hypotheses identifying a likelihood that a word in a word hypothesis was correctly recognized in generating the word hypotheses.

4. The method of claim 3, wherein words having corresponding word hypotheses that do not meet a confidence threshold are not considered in calculating the assessment score.

5. The method of claim 1 wherein the features are based on at least one of:

Hidden Markov Model probabilities, average phoneme duration, average word duration, phoneme duration distribution, word duration distribution, energy measurements, energy distributions, pitch measurements, pitch distributions, and pitch contours.

6. The method of claim 1 wherein speech samples are scored by a human, and a statistical model is built using the features and human scores;

wherein the assessment score is based on the scoring model and one or more of the calculated features.

7. The method of claim 6, wherein the statistical model is built using multiple regression.

8. The method of claim 1, wherein the reference acoustic model is trained with only native speech.

9. The method of claim 1, wherein generating word hypotheses utilizes the non-native acoustic model, a dictionary that maps pronunciations to words, and a language model that identifies a likelihood that a word will follow a sequence of already hypothesized words in a speech recording.

10. The method of claim 9, wherein the language model is an n-gram language model

11. The method of claim 1, wherein the assessment score is based on one or more features selected from the group consisting of:

an average likelihood across all letters;

L₁/m, where L₁ is a summation of likelihoods of all individual words;

12. The method of claim 1, wherein one or more of the features are utilized with related stress, intonation, vocabulary, or grammar to generate the assessment score indicating communicative competence or other construct of speaking proficiency that includes pronunciation proficiency.

13. The method of claim 1, wherein second statistics derived from native-quality speech are utilized in calculating the plurality of features.

24. The system of claim 1, wherein the assessment score is based on one or more features selected from the group consisting of:

an average likelihood across all letters;

L₁/m, where L₁ is a summation of likelihoods of all individual words;

28. The method of claim 1, wherein the digitized speech is spontaneous, non-native speech of a non-native language speaker.

14. A computer-implemented system for assessing spontaneous speech pronunciation, comprising:

a processing system;

a computer-readable memory programmed with instructions for causing the processing system to perform steps including;

performing speech recognition on digitized speech using a non-native acoustic model trained with non-native speech using a processing system to generate word hypotheses for the digitized speech;

performing time alignment between the digitized speech and the word hypotheses utilizing a reference acoustic model trained with native-quality speech to associate word hypotheses with corresponding sounds of the digitized speech;

calculating statistics regarding individual words and phonemes of the word hypotheses using the processing system based on said alignment;

calculating a plurality of features for use in assessing pronunciation of the speech based on the statistics using the processing system;

calculating an assessment score based on one or more of the calculated features; and

storing the assessment score in a computer-readable memory.

15. The system of claim 14 wherein the steps further comprise excluding words not reliably recognized in generating the word hypotheses from contributing to the assessment score.

16. The system of claim 15, wherein a confidence level is associated with each word in the word hypotheses identifying a likelihood that a word in a word hypothesis was correctly recognized in generating the word hypotheses.

17. The system of claim 16, wherein words having corresponding hypotheses that do not meet a confidence threshold are not considered in calculating the assessment score.

18. The system of claim 14 wherein the features are based on at least one of:

Hidden Markov Model probabilities, average phoneme duration, average word duration, phoneme duration distribution, word duration distribution, energy measurements, energy distribution, pitch measurements, pitch distribution, and pitch contours.

19. The system of claim 14 wherein speech samples are scored by a human, and a statistical model is built using the features and human scores;

wherein the assessment score is based on the scoring model and one or more of the calculated features.

20. The system of claim 19, wherein the statistical model is built using multiple regression.

21. The system of claim 14, wherein the reference acoustic model is trained with only native speech.

22. The system of claim 14, wherein generating word hypotheses utilizes the non-native acoustic model, a dictionary that maps pronunciations to words, and a language model that identifies a likelihood that a word will follow a sequence of already hypothesized words in a speech recording.

23. The system of claim 22, wherein the language model is an n-gram language model

25. The system of claim 14, wherein one or more of the features are utilized with related stress, intonation, vocabulary, or grammar to generate the assessment score indicating communicative competence or other construct of speaking proficiency that includes pronunciation proficiency.

26. The system of claim 14, wherein second statistics derived from native-quality speech are utilized in calculating the plurality of features.

29. The system of claim 14, wherein the digitized speech is spontaneous, non-native speech of a non-native language speaker.

27. A computer-readable memory comprising computer-readable instructions, which when executed cause a processing system to perform steps comprising:

performing speech recognition on digitized speech using a non-native acoustic model trained with non-native speech using a processing system to generate word hypotheses for the digitized speech;

performing time alignment between the digitized speech and the word hypotheses utilizing a reference acoustic model trained with native-quality speech;

calculating statistics regarding individual words and phonemes of the word hypotheses using the processing system based on said alignment;

calculating a plurality of features for use in assessing pronunciation of the speech based on the statistics using the processing system;

calculating an assessment score based on one or more of the calculated features; and

storing the assessment score in a computer-readable memory.