Method and system for using automatic generation of speech features to provide diagnostic feedback

• US 7,840,404 B2
• Filed: 02/23/2007
• Issued: 11/23/2010
• Est. Priority Date: 09/20/2004
• Status: Active Grant

• Current Assignee: Educational Testing Service
• Original Assignee: Educational Testing Service
• Inventors: Xi, Xiaoming, Bejar, Isaac, Zechner, Klaus
• Primary Examiner(s): MCFADDEN, SUSAN IRIS
• Application Number:
  US11/678,508
• Publication Number:
  US 20070213982A1
• Time in Patent Office: 1,369 Days
• Field of Search: 704/243
• US Class Current: 704/243
• CPC Class Codes: G09B 19/04 Speaking with audible presentation of the material to be studied G09B5/04
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

A method and system for providing immediate diagnostic feedback on speech samples of non-native speakers are disclosed. A scoring model is generated based on speech features extracted from one or more training speech samples. An evaluation speech sample is received and speech features of the evaluation speech sample are determined. Based on the scoring model and the speech features, diagnostic feedback is provided to the speaker. In an alternate embodiment, speech features are extracted from an evaluation speech sample. The speech features are then compared with optimal values, ranges of values, or norms for those features. Based on the result of the comparison, diagnostic feedback is provided to the speaker.

Claims

1. A processor-implemented method of using automated speech analysis to provide feedback on an evaluation speech sample comprising:

receiving a training speech sample in response to a first prompt;

determining a plurality of training speech features for the training speech sample using a processor;

generating a scoring model based on the plurality of training speech features, wherein the scoring model is effective for scoring high entropy evaluation speech responses;

receiving the evaluation speech sample in response to a second prompt using a processor;

determining a plurality of evaluation speech features for the evaluation speech sample using a processor; and

providing diagnostic feedback to the speaker based upon the scoring model and the evaluation speech features using a processor.

2. The method of claim 1 wherein the training speech sample is a high entropy speech sample.

3. The method of claim 1 wherein the plurality of training speech features and the plurality of evaluation speech features comprise at least two speech features selected from the following:

a lexical count;

a fluency measure;

a rate of speech measure;

a lexical similarity measure; and

a speech sample utterance duration.

4. The method of claim 3 wherein the lexical count comprises one or more of the following:

a number of utterances in response to the first prompt for the training speech sample;

a number of words used in the training speech sample;

a number of disfluencies used in the training speech sample;

a number of tokens used in the training speech sample, wherein the number of tokens comprises a sum of the number of words and the number of disfluencies;

a number of unique word forms used in the training speech sample; and

a ratio equal to the number of unique word forms used in the training speech sample divided by the number of tokens used in the training speech sample.

5. The method of claim 3 wherein the fluency measure comprises one or more of the following:

a number of silent periods during the training speech sample, wherein each silent period comprises a duration;

a ratio equal to the number of silent periods during the training speech sample divided by a number of words used in the training speech sample;

a mean of the durations for all silent periods during the training speech sample; and

a standard deviation of the durations of all silent periods during the training speech sample.

6. The method of claim 3 wherein the rate of speech measure comprises one or more of the following:

a number of words per time unit in the training speech sample;

a number of disfluencies per time unit in the training speech sample;

a number of unique word forms per time unit in the training speech sample; and

a number of silent periods per time unit in the training speech sample.

7. The method of claim 3 wherein the lexical similarity measure comprises one or more of the following:

an inner product of word frequencies for the training speech sample and a content vector, wherein the content vector comprises raw frequencies of word forms based on a corpus related to the first prompt; and

a ratio equal to the inner product divided by a number of words in the training speech sample.

8. The method of claim 1 wherein generating a scoring model comprises:

generating a classification tree based on a regression, for each training speech sample, between a score assigned to the training speech sample and the one or more speech features determined for the training speech sample.

9. The method of claim 1 wherein generating a scoring model comprises:

generating a scoring model based on weighting factors for the one or more speech features assigned by one or more expert judges.

10. The method of claim 1 wherein the evaluation speech sample is a high entropy speech sample.

11. A method of using automated speech analysis to provide, to a speaker, feedback on a high entropy speech sample comprising:

retrieving a scoring model based on a plurality of speech features;

receiving a high entropy speech sample in response to a prompt;

determining the plurality of speech features for the evaluation speech sample using a processor; and

providing diagnostic feedback to the speaker based upon the scoring model and the plurality of speech features using a processor.

12. The method of claim 11 wherein the plurality of speech features comprise at least two speech features selected from the following for each high entropy speech sample:

a lexical count;

a fluency measure;

a rate of speech measure;

a lexical similarity measure; and

a speech sample utterance duration.

13. The method of claim 12 wherein the lexical count comprises one or more of the following:

a number of utterances in response to a prompt for the high entropy speech sample;

a number of words used in the high entropy speech sample;

a number of disfluencies used in the high entropy speech sample;

a number of tokens used in the high entropy speech sample, wherein the number of tokens comprises a sum of the number of words and the number of disfluencies;

a number of words used in the high entropy speech sample; and

a ratio equal to the number of unique word forms used in the high entropy speech sample divided by the number of tokens used in the high entropy speech sample.

14. The method of claim 12 wherein the fluency measure comprises one or more of the following:

a number of silent periods during the high entropy speech sample, wherein each silent period comprises a duration;

a ratio equal to the number of silent periods during the high entropy speech sample divided by a number of words used in the high entropy speech sample;

a mean of the durations for all silent periods during the high entropy speech sample; and

a standard deviation of the durations of all silent periods during the high entropy speech sample.

15. The method of claim 12 wherein the rate of speech measure comprises one or more of the following:

a number of words per time unit in the high entropy speech sample;

a number of disfluencies per time unit in the high entropy speech sample;

a number of unique word forms per time unit in the high entropy speech sample; and

a number of silent periods per time unit in the high entropy speech sample.

16. The method of claim 12 wherein the lexical similarity measure comprises one or more of the following:

an inner product of word frequencies for the high entropy speech sample and a content vector, wherein the content vector comprises raw frequencies of word forms based on a corpus related to the prompt; and

a ratio equal to the inner product divided by a number of words in the high entropy speech sample.

17. A system for using automated speech analysis to provide, to a speaker, feedback on a high entropy speech sample, the system comprising:

a processor; and

a processor-readable storage medium,

wherein the processor-readable storage medium contains one or more programming instructions for performing a method for scoring a high entropy speech sample, the method comprising;

retrieving a scoring model based on a plurality of speech features;

receiving the high entropy speech sample in response to a prompt;

determining the plurality of speech features for the evaluation speech sample;

comparing one of the plurality of speech features to an optimal value or range of values for said speech feature; and

providing diagnostic feedback to the speaker if the speech feature diverges from the optimal value or range of values.

18. A method of using automated speech analysis to provide, to a speaker, feedback on a high entropy speech sample comprising:

receiving the high entropy speech sample;

determining a plurality of speech features for the evaluation speech sample;

comparing at least one of the plurality of speech features to an optimal value or range of values for the at least one of the plurality of speech features using a processor; and

providing diagnostic feedback to the speaker based on a result of the comparison using a processor.

19. The method of claim 18, wherein the optimal value or range of values is predetermined.

20. The method of claim 18, wherein the optimal value or range of values is calculated as a function of at least one of the plurality of speech features.

21. The method of claim 18 wherein the plurality of speech features comprise at least two speech features selected from the following for each high entropy speech sample:

a lexical count;

a fluency measure;

a rate of speech measure;

a lexical similarity measure; and

a speech sample utterance duration.

22. The method of claim 18 wherein the lexical count comprises one or more of the following:

a number of utterances in response to a prompt for the high entropy speech sample;

a number of words used in the high entropy speech sample;

a number of disfluencies used in the high entropy speech sample;

a number of tokens used in the high entropy speech sample, wherein the number of tokens comprises a sum of the number of words and the number of disfluencies;

a number of words used in the high entropy speech sample; and

a ratio equal to the number of unique word forms used in the high entropy speech sample divided by the number of tokens used in the high entropy speech sample.

23. The method of claim 18 wherein the fluency measure comprises one or more of the following:

a number of silent periods during the high entropy speech sample, wherein each silent period comprises a duration;

a ratio equal to the number of silent periods during the high entropy speech sample divided by a number of words used in the high entropy speech sample;

a mean of the durations for all silent periods during the high entropy speech sample; and

a standard deviation of the durations of all silent periods during the high entropy speech sample.

24. The method of claim 18 wherein the rate of speech measure comprises one or more of the following:

a number of words per time unit in the high entropy speech sample;

a number of disfluencies per time unit in the high entropy speech sample;

a number of unique word forms per time unit in the high entropy speech sample; and

a number of silent periods per time unit in the high entropy speech sample.

25. The method of claim 18 wherein the lexical similarity measure comprises one or more of the following:

an inner product of word frequencies for the high entropy speech sample and a content vector, wherein the content vector comprises raw frequencies of word forms based on a corpus related to the prompt; and

a ratio equal to the inner product divided by a number of words in the high entropy speech sample.