Method and system for the automatic generation of speech features for scoring high entropy speech

• US 8,209,173 B2
• Filed: 06/16/2008
• Issued: 06/26/2012
• Est. Priority Date: 09/20/2004
• Status: Active Grant

• Current Assignee: Educational Testing Service
• Original Assignee: Educational Testing Service
• Inventors: Bejar, Isaac, Zechner, Klaus
• Primary Examiner(s): JACKSON, JAKIEDA R
• Application Number:
  US12/140,209
• Publication Number:
  US 20080249773A1
• Time in Patent Office: 1,471 Days
• Field of Search: 704/243
• US Class Current: 704/243
• CPC Class Codes: G10L 15/063 Training

Abstract

A method and system for automatically generating a scoring model for scoring a speech sample are disclosed. One or more training speech samples are received in response to a prompt. One or more speech features are determined for each of the training speech samples. A scoring model is then generated based on the speech features. At least one of the training speech samples may be a high entropy speech sample. An evaluation speech sample is received and a score is assigned to the evaluation speech sample using the scoring model. The evaluation speech sample may be a high entropy speech sample.

Claims

1. A computer-implemented method of automatically generating a scoring model for scoring a speech sample to characterize speaking proficiency, the method comprising:

receiving one or more training speech samples in response to a prompt;

determining one or more speech features for each of the training speech samples, wherein the speech features include a previously assigned score, wherein the previously assigned score is assigned by a human rater based on a training speech sample; and

generating a scoring model using a computer based on the speech features, wherein the scoring model is effective for scoring spontaneous speech responses;

wherein the scoring model is independent of speech recognition accuracy associated with the training speech samples;

wherein the scoring model is used to generate a numeric speaking proficiency score for a received spontaneous speech response.

2. The method of claim 1 wherein at least one of the training speech samples comprises a spontaneous speech sample.

3. The method of claim 1 wherein the one or more speech features comprise one or more of the following for each training speech sample:

a lexical count;

a fluency measure;

a rate of speech measure; and

a speech sample utterance duration.

4. The method of claim 3 wherein the lexical count comprises one or more of the following:

a number of utterances in response to a prompt for the training speech sample;

a number of words used in the training speech sample;

a number of disfluencies used in the training speech sample;

a number of tokens used in the training speech sample, wherein the number of tokens comprises a sum of the number of words and the number of disfluencies;

a number of unique word forms used in the training speech sample; and

a ratio equal to the number of unique word forms used in the training speech sample divided by the number of tokens used in the training speech sample.

5. The method of claim 3 wherein the fluency measure comprises one or more of the following:

a ratio equal to the number of silent periods during the training speech sample divided by a number of words used in the training speech sample;

a mean of the durations for all silent periods during the training speech sample; and

a standard deviation of the durations of all silent periods during the training speech sample.

6. The method of claim 3 wherein the rate of speech measure comprises one or more of the following:

a number of words per time unit in the training speech sample;

a number of disfluencies per time unit in the training speech sample;

a number of unique word forms per time unit in the training speech sample; and

a number of silent periods per time unit in the training speech sample.

7. The method of claim 1 wherein generating a scoring model comprises:

generating a classification tree based on a regression, for each training speech sample, between a score assigned to the training speech sample and the one or more speech features determined for the training speech sample.

8. The method of claim 1 wherein generating a scoring model comprises:

generating a scoring model based on weighting factors for the one or more speech features assigned by one or more expert judges.

9. The method of claim 1, wherein the content of the received spontaneous speech response is unknown.

10. The method of claim 1, wherein the scoring model uses an automatic speech recognition system to recognize the content of the received spontaneous speech response prior to generating a numeric speaking proficiency score for the received spontaneous speech response.

11. The method of claim 1, wherein the scoring model uses at least one of intonation and stress of the speech sample in assigning the numeric speaking proficiency score.

12. The method of claim 11, wherein the intonation of the speech sample is quantified based on a falling, rising, or steady state of pitch of the spontaneous speech response.

13. A computer-implemented method for scoring a spontaneous speech sample to characterize speaking proficiency, the method comprising:

retrieving a scoring model, wherein the scoring model has been trained using speech features that include a previously assigned score for a training speech sample, wherein the previously assigned score is assigned by a human rater based on the training speech sample;

wherein the scoring model is independent of speech recognition accuracy associated with the spontaneous speech sample; and

assigning a numeric speaking proficiency score to a received spontaneous speech sample using the scoring model using a computer.

14. The method of claim 13 wherein the one or more speech features comprise one or more of the following for each spontaneous speech sample:

a lexical count;

a fluency measure;

a rate of speech measure; and

a speech sample utterance duration.

15. The method of claim 14 wherein the lexical count comprises one or more of the following:

a number of utterances in response to a prompt for the spontaneous speech sample;

a number of words used in the spontaneous speech sample;

a number of disfluencies used in the spontaneous speech sample;

a number of tokens used in the spontaneous speech sample,

wherein the number of tokens comprises a sum of the number of words and the number of disfluencies;

a number of words used in the spontaneous speech sample; and

a ratio equal to the number of unique word forms used in the spontaneous speech sample divided by the number of tokens used in the spontaneous speech sample.

16. The method of claim 14 wherein the fluency measure comprises one or more of the following:

a ratio equal to the number of silent periods during the spontaneous speech sample divided by a number of words used in the spontaneous speech sample;

a mean of the durations for all silent periods during the spontaneous speech sample; and

a standard deviation of the durations of all silent periods during the spontaneous speech sample.

17. The method of claim 14 wherein the rate of speech measure comprises one or more of the following:

a number of words per time unit in the spontaneous speech sample;

a number of disfluencies per time unit in the spontaneous speech sample;

a number of unique word forms per time unit in the spontaneous speech sample; and

a number of silent periods per time unit in the spontaneous speech sample.

18. A system for automatically generating a scoring model for scoring a spontaneous speech sample to characterize speaking proficiency, the system comprising:

a processor; and

a processor-readable storage medium, wherein the processor-readable storage medium contains one or more programming instructions for automatically generating a scoring model for scoring a spontaneous speech sample, the programming instructions when executed causing the processor to carry out steps comprising;

receiving one or more spontaneous training speech samples in response to a prompt,

determining one or more speech features for each of the training speech samples, wherein the speech features include a previously assigned score, wherein the previously assigned score is assigned by a human rater based on a training speech sample, and

generating a scoring model based on the speech features;

wherein the scoring model is independent of speech recognition accuracy associated with the training speech samples;

wherein the scoring model is used to generate a numeric speaking proficiency score for a received spontaneous speech response.

19. A system for scoring a spontaneous speech sample to characterize speaking proficiency, the system comprising:

a processor; and

a processor-readable storage medium, wherein the processor-readable storage medium contains one or more programming instructions for scoring a spontaneous speech sample, the programming instructions when executed causing the processor to carry out steps comprising;

retrieving a scoring model, wherein the scoring model has been trained using speech features that include a previously assigned score for a training speech sample, wherein the previously assigned score is assigned by a human rater based on the training speech sample,

wherein the scoring model is independent of speech recognition accuracy associated with the spontaneous speech sample, and

assigning a numeric speaking proficiency score to a spontaneous speech sample using the scoring model.

20. An article of manufacture comprising a non-transitory computer-readable medium for automatically generating a scoring model for scoring a spontaneous speech sample to characterize speaking proficiency comprising programming instructions, which when executed cause a processing system to perform steps comprising:

receiving one or more spontaneous training speech samples in response to a prompt,

determining one or more speech features for each of the training speech samples, wherein the speech features include a previously assigned score, wherein the previously assigned score is assigned by a human rater based on a training speech sample, and

generating a scoring model based on the speech features;

wherein the scoring model is independent of speech recognition accuracy associated with the training speech samples;

wherein the scoring model is used to generate a numeric speaking proficiency score for a received spontaneous speech response.

21. An article of manufacture comprising a non-transitory computer-readable medium for scoring a spontaneous speech sample to characterize speaking proficiency comprising programming instructions, which when executed cause a processing system to perform steps comprising:

retrieving a scoring model, wherein the scoring model has been trained using speech features that include a previously assigned score for a training speech sample, wherein the previously assigned score is assigned by a human rater based on the training speech sample,

wherein the scoring model is independent of speech recognition accuracy associated with the spontaneous speech sample, and

assigning a numeric speaking proficiency score to a spontaneous speech sample using the scoring model.