Systems and methods for assessment of non-native speech using vowel space characteristics

• US 9,262,941 B2
• Filed: 07/14/2011
• Issued: 02/16/2016
• Est. Priority Date: 07/14/2010
• Status: Active Grant

• Current Assignee: Educational Testing Service
• Original Assignee: Educational Testing Service
• Inventors: Chen, Lei, Evanini, Keelan, Sun, Xie
• Primary Examiner(s): Nguyen, Timothy
• Application Number:
  US13/182,770
• Publication Number:
  US 20120016672A1
• Time in Patent Office: 1,678 Days
• Field of Search: 704/236
• US Class Current: 1/1
• CPC Class Codes: G09B 19/04 Speaking with audible presentation of the material to be studied G09B5/04
• CPC Class Codes: G09B 19/06 Foreign languages with audible presentation of material to be studied G09B5/04
• CPC Class Codes: G10L 15/04 Segmentation; Word boundary detection
• CPC Class Codes: G10L 25/15 the extracted parameters being formant information
• CPC Class Codes: G10L 25/27 characterised by the analysis technique
• CPC Class Codes: G10L 25/60 for measuring the quality of voice signals

Abstract

Computer-implemented systems and methods are provided for assessing non-native speech proficiency. A non-native speech sample is processed to identify a plurality of vowel sound boundaries in the non-native speech sample. Portions of the non-native speech sample are analyzed within the vowel sound boundaries to extract vowel characteristics. The vowel characteristics are used to identify a plurality of vowel space metrics for the non-native speech sample, and the vowel space metrics are used to determine a non-native speech proficiency score for the non-native speech sample.

Claims

1. A computer-implemented method of assessing non-native speech proficiency, comprising:

processing, using a processing system, a non-native speech sample to identify vowel sounds in the non-native speech sample and vowel sound boundaries corresponding to the vowel sounds in the non-native speech sample;

analyzing, using the processing system, the vowel sounds within the vowel sound boundaries to extract vowel characteristics;

processing, using the processing system, the vowel sounds and the vowel characteristics to identify a plurality of vowel space metrics for the non-native speech sample, wherein the vowel space metrics indicate an extent of proficient utilization of AA, IY, and OW vowel sounds in the non-native speech sample; and

generating using the processing system, a non-native speech proficiency score based upon the plurality of vowel space metrics for the non-native speech sample, wherein the non-native speech proficiency score provides a measure of a pronunciation proficiency of the non-native speech sample based upon the plurality of vowel space metrics, wherein the non-native speech proficiency score is generated by processing the plurality of vowel space metrics with a statistical computer model, the statistical computer model comprising multiple weighted features determined by training the statistical computer model relative to a plurality of training items, the training utilizing calculated vowel space metrics of the training items and human-scored non-native speech proficiency scores of the training items to determine values for the weighted features.

2. The method of claim 1, wherein the vowel characteristics comprise vowel formant measurements.

3. The method of claim 2, wherein a vowel formant measurement comprises a measurement of an amplitude peak in a vowel spectrum that indicates a resonant frequency of a vowel.

4. The method of claim 1, wherein the vowel space metrics comprise a vowel space range, a vowel space area, a vowel space dispersion, a within category vowel dispersion, or a F2-F1 distance.

5. The method of claim 4, wherein the vowel space metrics comprise a vowel space range, wherein the vowel space range, F1 Range, is calculated according to

F1Range=Max_F1(AA)−

Min_F1(IY),

where Max_F1(AA) is the maximum value of the first vowel formant measurements for all AA vowel sounds produced by a speaker, and Min_F1(IY) is the minimum value of the first vowel formant measurements for all IY vowel sounds produced by a speaker.

6. The method of claim 5, wherein the vowel space metrics comprise a second vowel space range, wherein the second vowel space range, F2Range, is calculated according to

F2Range=Max_F2(IY)−

Min_F2(OW),

where Max_F2(IY) is the maximum value of the second vowel formant measurements for all IY vowel sounds produced by a speaker, and Min_F2(OW) is the minimum value of the second vowel formant measurements for all OW vowel sounds produced by the speaker,

wherein the non-native speech proficiency score is determined based on the F1Range and F2Range vowel space metrics.

7. The method of claim 1, wherein one or more of the vowel space metrics are calculated based on a first vowel characteristic, formant F1, and a second vowel characteristic, formant F2;

wherein the vowel space metrics comprise an area of a vowel triangle defined by a mean F1 and F2 values of the AA, IY, and OW vowel sounds.

8. The method of claim 7, wherein the area is calculated according to:

area=√

{square root over (s(s−

D_{IY, AA} )(s−

D_{AA, OW} )(s−

D_{OW, IY}))},

where s=0.5*(D_{IY, AA}+D_{AA, OW}+D_{OW, IY}), V represents mean F1 and F2 values for vowel V, and D_x,y represents a Euclidean distance between two values in an F1-F2 plane;

D_x,y=√

{square root over ((F1_x−

F1_y)²+(F2_x−

F2_y)²)}{square root over ((F1_x−

F1_y)²+(F2_x−

F2_y)²)}.

9. The method of claim 1, wherein one or more of the vowel space metrics are calculated based on a first vowel space characteristic, formant F1, and a second vowel space characteristic, formant F2;

wherein the vowel space metrics comprise a vowel space dispersion based on an average distance from individual AA, IY, and OW vowel tokens to a center of a vowel space.

10. The method of claim 9, wherein the vowel space center, V, is a mean of the F1 and F2 vowel space characteristics, wherein the vowel space dispersion is calculated according to:

11. The method of claim 1, wherein one or more of the vowel space metrics are calculated based on a first vowel space characteristic, formant F1, and a second vowel space characteristic, formant F2;

wherein the vowel space metrics comprise a within category vowel space dispersion.

12. The method of claim 11, wherein the within category vowel space dispersion is calculated according to:

13. The method of claim 1, wherein one or more of the vowel space metrics are calculated based on a first vowel space characteristic, F1, and a second vowel space characteristic, F2;

wherein the vowel space metrics comprise an F2-F1 distance.

14. The method of claim 1, wherein the non-native speech proficiency score is automatically provided via a graphical user interface to provide feedback on non-native speech proficiency.

15. The method of claim 14, wherein the feedback offers vowel pronunciation suggestions for improving communicative competence through better vowel pronunciation.

16. The method of claim 1, further comprising performing automatic speech recognition on the non-native speech sample to generate a transcript, wherein the transcript is used to identify the vowel sound boundaries.

17. The method of claim 1, wherein a forced alignment procedure is used to identify the plurality of vowel sound boundaries.

18. The method of claim 1, wherein the non-native speech proficiency score is determined based on one or more non-vowel-space-based metrics.

19. The method of claim 18, wherein the non-native speech proficiency score is further based on a stress metric, an intonation metric, a vocabulary metric, or a grammar metric.

32. The computer-implemented method of claim 1 comprising:

normalizing one or more vowel tokens included in the non-native speech sample to reduce effects of speaker-specific physiological characteristics.

20. A computer-implemented system for assessing non-native speech proficiency, comprising:

one or more data processors; and

one or more memories, wherein the one or more data processors are configured to execute steps comprising;

processing a non-native speech sample to identify vowel sounds in the non-native speech sample and vowel sound boundaries corresponding to the vowel sounds in the non-native speech sample;

analyzing the vowel sounds within the vowel sound boundaries to extract vowel characteristics;

processing the vowel sounds and the vowel characteristics to identify a plurality of vowel space metrics for the non-native speech sample, wherein the vowel space metrics indicate an extent of proficient utilization of AA, IY, and OW vowel sounds in the non-native speech sample; and

generating a non-native speech proficiency score based upon the plurality of vowel space metrics for the non-native speech sample, wherein the non-native speech proficiency score provides a measure of a pronunciation proficiency of the non-native speech sample based upon the plurality of vowel space metrics, wherein the non-native speech proficiency score is generated by processing the plurality of vowel space metrics with a statistical computer model, the statistical computer model comprising multiple weighted features determined by training the statistical computer model relative to a plurality of training items, the training utilizing calculated vowel space metrics of the training items and human-scored non-native speech proficiency scores of the training items to determine values for the weighted features.

22. The computer-implemented system of claim 20, wherein the vowel space metrics comprise a vowel space range, a vowel space area, a vowel space dispersion, a within category vowel dispersion, or a F2-F1 distance.

23. The computer-implemented system of claim 22, wherein the vowel space metrics comprise a vowel space range, wherein the vowel space range, F1 Range, is calculated according to

F1Range=Max_F1(AA)−

Min_F1(IY),

where Max_F1(AA) is the maximum value of the first vowel formant measurements for all AA vowel sounds produced by a speaker, and Min_F1(IY) is the minimum value of the first vowel formant measurements for all IY vowel sounds produced by a speaker.

24. The computer-implemented system of claim 23, wherein the vowel space metrics comprise a second vowel space range, wherein the second vowel space range, F2Range, is calculated according to

F2Range=Max_F2(IY)−

Min_F2(OW),

where Max_F2(IY) is the maximum value of the second vowel formant measurements for all IY vowel sounds produced by a speaker, and Min_F2(OW) is the minimum value of the second vowel formant measurements for all OW vowel sounds produced by the speaker,

wherein the non-native speech proficiency score is determined based on the F1Range and F2Range vowel space metrics.

25. The computer-implemented system of claim 20, wherein the steps include performing automatic speech recognition on the non-native speech sample to generate a transcript, and wherein the transcript is used to identify the vowel sound boundaries.

26. The computer-implemented system of claim 20, wherein a forced alignment procedure is used to identify the plurality of vowel sound boundaries.

33. The computer-implemented system of claim 20, wherein the non-native speech proficiency score is determined based on one or more non-vowel-space-based metrics.

34. The computer-implemented system of claim 20, wherein the non-native speech proficiency score is further based on a stress metric, an intonation metric, a vocabulary metric, or a grammar metric.

37. The computer-implemented system of claim 20, wherein one or more of the vowel space metrics are calculated based on a first vowel characteristic, formant F1, and a second vowel characteristic, formant F2;

wherein the vowel space metrics comprise an area of a vowel triangle defined by a mean F1 and F2 values of the AA, IY, and OW vowel sounds.

38. The computer-implemented system of claim 37, wherein the area is calculated according to:

area=√

{square root over (s(s−

D_{IY, AA} )(s−

D_{AA, OW} )(s−

D_{OW, IY}))},

where s=0.5*(D_{IY, AA}+D_{AA, OW}+D_{OW, IY}), V represents mean F1 and F2 values for vowel V, and D_x,y represents a Euclidean distance between two values in an F1-F2 plane;

D_x,y=√

{square root over ((F1_x−

F1_y)²+(F2_x−

F2_y)²)}{square root over ((F1_x−

F1_y)²+(F2_x−

F2_y)²)}.

39. The computer-implemented system of claim 20, wherein one or more of the vowel space metrics are calculated based on a first vowel space characteristic, formant F1, and a second vowel space characteristic, formant F2;

wherein the vowel space metrics comprise a within category vowel space dispersion.

40. The computer-implemented system of claim 39, wherein the within category vowel space dispersion is calculated according to:

21. A non-transitory computer-readable storage medium comprising instructions for which when executed cause a processing system to execute steps comprising:

processing a non-native speech sample to identify vowel sounds in the non-native speech sample and vowel sound boundaries corresponding to the vowel sounds in the non-native speech sample;

analyzing the vowel sounds within the vowel sound boundaries to extract vowel characteristics;

processing the vowel sounds and the vowel characteristics to identify a plurality of vowel space metrics for the non-native speech sample, wherein the vowel space metrics indicate an extent of proficient utilization of AA, IY, and OW vowel sounds in the non-native speech sample; and

generating a non-native speech proficiency score based upon the plurality of vowel space metrics for the non-native speech sample, wherein the non-native speech proficiency score provides a measure of a pronunciation proficiency of the non-native speech sample based upon the plurality of vowel space metrics, wherein the non-native speech proficiency score is generated by processing the plurality of vowel space metrics with a statistical computer model, the statistical computer model comprising multiple weighted features determined by training the statistical computer model relative to a plurality of training items, the training utilizing calculated vowel space metrics of the training items and human-scored non-native speech proficiency scores of the training items to determine values for the weighted features.

27. The non-transitory computer-readable storage medium of claim 21, wherein the vowel space metrics comprise a vowel space range, a vowel space area, a vowel space dispersion, a within category vowel dispersion, or a F2-F1 distance.

28. The non-transitory computer-readable storage medium of claim 27, wherein the vowel space metrics comprise a vowel space range, wherein the vowel space range, F1 Range, is calculated according to

F1Range=Max_F1(AA)−

Min_F1(IY),

where Max_F1(AA) is the maximum value of the first vowel formant measurements for all AA vowel sounds produced by a speaker, and Min_F1(IY) is the minimum value of the first vowel formant measurements for all IY vowel sounds produced by a speaker.

29. The non-transitory computer-readable storage medium of claim 28, wherein the vowel space metrics comprise a second vowel space range, wherein the second vowel space range, F2Range, is calculated according to

F2Range=Max_F2(IY)−

Min_F2(OW),

where Max_F2(IY) is the maximum value of the second vowel formant measurements for all IY vowel sounds produced by a speaker, and Min_F2(OW) is the minimum value of the second vowel formant measurements for all OW vowel sounds produced by the speaker,

wherein the non-native speech proficiency score is determined based on the F1Range and F2Range vowel space metrics.

30. The non-transitory computer-readable storage medium of claim 21, wherein the instructions cause the processing system to perform automatic speech recognition on the non-native speech sample to generate a transcript, and wherein the transcript is used to identify the vowel sound boundaries.

31. The non-transitory computer-readable storage medium of claim 21, wherein a forced alignment procedure is used to identify the plurality of vowel sound boundaries.

35. The non-transitory computer-readable storage medium of claim 21, wherein the non-native speech proficiency score is determined based on one or more non-vowel-space-based metrics.

36. The non-transitory computer-readable storage medium of claim 21, wherein the non-native speech proficiency score is further based on a stress metric, an intonation metric, a vocabulary metric, or a grammar metric.

41. The non-transitory computer-readable storage medium of claim 21, wherein one or more of the vowel space metrics are calculated based on a first vowel characteristic, formant F1, and a second vowel characteristic, formant F2;

wherein the vowel space metrics comprise an area of a vowel triangle defined by a mean F1 and F2 values of the AA, IY, and OW vowel sounds.

42. The non-transitory computer-readable storage medium of claim 41, wherein the area is calculated according to:

area=√

{square root over (s(s−

D_{IY, AA} )(s−

D_{AA, OW} )(s−

D_{OW, IY}))},

where s=0.5*(D_{IY, AA}+D_{AA, OW}+D_{OW, IY}), V represents mean F1 and F2 values for vowel V, and D_x,y represents a Euclidean distance between two values in an F1-F2 plane;

D_x,y=√

{square root over ((F1_x−

F1_y)²+(F2_x−

F2_y)²)}{square root over ((F1_x−

F1_y)²+(F2_x−

F2_y)²)}.

43. The non-transitory computer-readable storage medium of claim 21, wherein one or more of the vowel space metrics are calculated based on a first vowel space characteristic, formant F1, and a second vowel space characteristic, formant F2;

wherein the vowel space metrics comprise a within category vowel space dispersion.

44. The non-transitory computer-readable storage medium of claim 43, wherein the within category vowel space dispersion is calculated according to: