Utterance verification and pronunciation scoring by lattice transduction

• US 8,880,399 B2
• Filed: 09/27/2010
• Issued: 11/04/2014
• Est. Priority Date: 09/27/2010
• Status: Active Grant

• Current Assignee: Rosetta Stone Limited (Rosetta Stone Incorporated)
• Original Assignee: Rosetta Stone Limited (Rosetta Stone Incorporated)
• Inventors: Hagen, Andreas, Pellom, Bryan
• Primary Examiner(s): Lerner, Martin
• Application Number:
  US12/890,753
• Publication Number:
  US 20120078630A1
• Time in Patent Office: 1,499 Days
• Field of Search: 704/231, 704/236, 704/239, 704/242, 704/251, 704/254, 704/256.4, 704/271, 704/255, 434/185
• US Class Current: 704/236
• CPC Class Codes: G09B 19/06 Foreign languages with audible presentation of material to be studied G09B5/04
• CPC Class Codes: G10L 15/187 Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams
• CPC Class Codes: G10L 2015/025 Phonemes, fenemes or fenones being the recognition units

Abstract

In the field of language learning systems, proper pronunciation of words and phrases is an integral aspect of language learning, determining the proximity of the language learner'"'"'s pronunciation to a standardized, i.e. ‘perfect’, pronunciation is utilized to guide the learner from imperfect toward perfect pronunciation. In this regard, a phoneme lattice scoring system is utilized, whereby an input from a user is transduced into the perfect pronunciation example in a phoneme lattice. The cost of this transduction may be determined based on a summation of substitutions, deletions and insertions of phonemes needed to transducer from the input to the perfect pronunciation of the utterance.

Claims

1. A method comprising:

defining, at a computer, a phoneme lattice for an ideal utterance, the phoneme lattice including a plurality of phoneme instances;

tagging, at the computer, each phoneme instance from the plurality of phoneme instances with a begin time, an end time, and a score;

storing the phoneme lattice at the computer;

determining an ideal path through the phoneme lattice for the ideal utterance;

receiving, at the computer, an input utterance from a user;

transducing the input utterance into the ideal utterance utilizing the phoneme lattice;

calculating, at the computer, an ideal path transduction cost based on the ideal path and the begin time, the end time, and the score of each phoneme instance from the plurality of phoneme instances included in the ideal path;

transducing the input utterance into an out-of-grammar word sequence;

calculating, at the computer, an out-of-grammar transduction cost based on the out-of-grammar word sequence;

determining an accuracy of the input utterance based on the ideal path transduction cost and the out-of-grammar transduction cost; and

sending a signal to output from the computer an indication of the accuracy.

2. The method of claim 1, further comprising:

tagging, at the computer, each phoneme instance of the ideal utterance with an ideal duration;

measuring an actual duration of each phoneme of the input utterance; and

scoring the ideal duration versus the actual duration to produce the score, the determining the accuracy of the input utterance being based on the score.

3. The method of claim 1, wherein the determining the accuracy of the input utterance is based on a phoneme confusion matrix.

4. The method of claim 3, wherein the phoneme confusion matrix provides a substitution cost for each phoneme instance from the plurality of phoneme instances included in the ideal path depending on a likelihood that a first phoneme is confused with a second phoneme.

5. The method of claim 1, further comprising:

transforming the phoneme lattice into a directed acyclic graph.

6. The method of claim 1, further comprising:

classifying the input utterance of the user based on a phoneme alignment of the input utterance to the ideal utterance.

7. The method of claim 1, further comprising:

optimizing the transducing the input utterance into the ideal utterance using an accent adjusted acoustic model.

8. The method of claim 1, further comprising:

sending a signal representing an out-of-grammar message if the ideal path transduction cost is outside of a predetermined threshold of the out-of-grammar transduction cost.

9. The method of claim 1, wherein the calculating the out-of-grammar transduction cost is based on a phoneme confusion matrix.

10. The method of claim 1, wherein the calculating the ideal path transduction cost is based on an accent adjusted acoustic model, the calculating the out-of-grammar transduction cost is based on the accent adjusted acoustic model.

11. A non-transitory processor-readable medium storing code representing instructions to be executed by a processor, the code comprising code to cause the processor to:

tag each phoneme instance from a plurality of ideal phoneme instances for an ideal utterance with a begin time, an end time, and a score to generate an ideal phoneme lattice;

determine a target phoneme sequence based on the ideal phoneme lattice for the ideal utterance;

receive an input utterance from a user;

define an input phoneme lattice based on the input utterance;

transduce the input phoneme lattice into the target phoneme sequence;

calculate an ideal path transduction cost of the input utterance based on the begin time, the end time, and the score of each phoneme instance from the plurality of ideal phoneme instances;

tag each phoneme instance from a plurality of out-of-grammar phoneme instances for an out-of-grammar utterance with a begin time, an end time, and a score to generate an out-of-grammar phoneme lattice;

determine an out-of-grammar phoneme sequence based on the out-of-grammar phoneme lattice for the out-of-grammar utterance;

transduce the phoneme lattice of the input utterance into the out-of-grammar phoneme sequence;

calculate an out-of-grammar transduction cost of the input utterance based on the begin time, the end time, and the score of each phoneme instance from the plurality of out-of-grammar phoneme instances; and

send a signal to output from the computer an indication of an accuracy of the input utterance based on the ideal path transduction cost and the out-of-grammar transduction cost.

12. The non-transitory processor readable medium of claim 11, further comprising code to cause the processor to:

verify the input utterance is a valid pronunciation attempt.

13. The non-transitory processor readable medium of claim 11, wherein the code to cause the processor to calculate includes code to cause the processor to calculate the ideal path transduction cost based on a phoneme confusion matrix.

14. The non-transitory processor readable medium of claim 11, further comprising code to cause the processor to:

determine a nativeness of the input utterance based on the ideal path transduction cost.

15. The non-transitory processor readable medium of claim 11, further comprising code to cause the processor to:

classify the input utterance of the user based on a phoneme alignment of the input utterance to the target phoneme sequence.

16. The non-transitory processor readable medium of claim 11, further comprising code to cause the processor to:

transform the ideal phoneme lattice into a directed acyclic graph.

17. The non-transitory processor readable medium of claim 11, further comprising code to cause the processor to:

send a signal representing an out-of-grammar message if the ideal path transduction cost is outside of a predetermined threshold of the out-of-grammar transduction cost.

18. The non-transitory processor readable medium of claim 11, wherein the calculating the out-of-grammar transduction cost is based on a phoneme confusion matrix.

19. The non-transitory processor readable medium of claim 11, wherein the calculating the ideal path transduction cost is based on an accent adjusted acoustic model, the calculating the out-of-grammar transduction cost is based on the accent adjusted acoustic model.