System and method for teaching non-lexical speech effects

• US 8,972,259 B2
• Filed: 09/09/2010
• Issued: 03/03/2015
• Est. Priority Date: 09/09/2010
• Status: Active Grant

• Current Assignee: Rosetta Stone Limited (Rosetta Stone Incorporated)
• Original Assignee: Rosetta Stone Limited (Rosetta Stone Incorporated)
• Inventors: Tepperman, Joseph, Stanley, Theban, Hacioglu, Kadri
• Primary Examiner(s): ADESANYA, OLUJIMI A
• Application Number:
  US12/878,402
• Publication Number:
  US 20120065977A1
• Time in Patent Office: 1,636 Days
• Field of Search: 704/9, 434156-186
• US Class Current: 704/236
• CPC Class Codes: G10L 13/08 Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
• CPC Class Codes: G10L 13/10 Prosody rules derived from text; Stress or intonation
• CPC Class Codes: G10L 15/10 using distance or distortion measures between unknown speech and reference templates
• CPC Class Codes: G10L 17/08 Use of distortion metrics or a particular distance between probe pattern and reference templates
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices
• CPC Class Codes: G10L 2025/903 using a laryngograph

Abstract

A method and system for teaching non-lexical speech effects includes delexicalizing a first speech segment to provide a first prosodic speech signal and data indicative of the first prosodic speech signal is stored in a computer memory. The first speech segment is audibly played to a language student and the student is prompted to recite the speech segment. The speech uttered by the student in response to the prompt, is recorded.

Claims

1. A method, comprising:

delexicalizing a speech segment to generate a first prosodic speech signal;

encoding the first prosodic speech signal to generate a musically encoded prosodic speech signal;

storing data indicative of the musically encoded prosodic speech signal in a computer memory;

audibly playing the musically encoded prosodic speech signal to a language student;

prompting, in response to the audibly playing, the student to recite the speech segment from which the musically encoded prosodic speech signal originated;

recording audible speech uttered by the student in response to the prompt;

delexicalizing the audible speech to generate a second prosodic speech signal;

calculating at least one error signal based on a difference between;

a difference between a duration of a first syllable in the first prosodic speech signal and a duration of a second syllable in the first prosodic speech signal, the first syllable in the first prosodic speech signal being non-adjacent to the second syllable in the first prosodic speech signal; and

a difference between a duration of a first syllable in the second prosodic speech signal and a duration of a second syllable in the second prosodic speech signal, the first syllable in the second prosodic speech signal being non-adjacent to the second syllable in the second prosodic speech signal.

2. The method of claim 1, further comprising:

determining prosodic characteristics of the first prosodic speech signal and the second prosodic speech signal, the calculating the at least one error signal being based on the prosodic characteristics of the first prosodic speech signal and the second prosodic speech signal.

3. The method of claim 1, further comprising:

determining prosodic characteristics of the first prosodic speech signal and the second prosodic speech signal, the calculating the at least one error signal being based on the prosodic characteristics of the first prosodic speech signal and the second prosodic speech signal, the prosodic characteristics including rhythm, the prosodic characteristics further including at least one of prosody;

duration in time;

a total number of syllables;

or a pitch level of at least one syllable.

4. The method of claim 1, wherein

the at least one error signal is further based on at least one of a difference in prosody between the first prosodic speech signal and the second prosodic speech signal;

a difference in a total number of syllables between the first prosodic speech signal and the second prosodic speech signal;

a difference in duration in units of time between the first prosodic speech signal and the second prosodic speech signal;

a difference in pitch between respective syllables between the first prosodic speech signal and the second prosodic speech signal;

or a difference in intonation between the first prosodic speech signal and the second prosodic speech signal.

5. The method of claim 1, wherein the encoding includes implementing a musical instrument digital interface (MIDI) framework.

6. The method of claim 1, wherein the encoding includes converting a frequency of the first prosodic speech signal to a musical instrument digital interface (MIDI) pitch note number.

7. The method of claim 1, wherein the calculating the at least one error signal is further based on a sum of the difference between:

the difference between the duration of the first syllable in the first prosodic speech signal and the duration of the second syllable in the first prosodic speech signal, the first syllable in the first prosodic speech signal being non-adjacent to the second syllable in the first prosodic speech signal; and

the difference between the duration of the first syllable in the second prosodic speech signal and the duration of the second syllable in the second prosodic speech signal, the first syllable in the second prosodic speech signal being non-adjacent to the second syllable in the second prosodic speech signal.

8. The method of claim 7, wherein the calculating the at least one error signal is further based on a normalization of the sum of the difference.

9. A method, comprising:

audibly presenting a non-lexicalized musically encoded phrase in a target language to a student;

receiving, at a computer, a lexicalized version of the non-lexicalized musically encoded phrase in the target language from the student;

comparing, at the computer, non-lexical properties of the non-lexicalized musically encoded phrase with non-lexical properties of the lexicalized version of the non-lexicalized musically encoded phrase to calculate at least one error signal based on a difference between;

a difference between a duration of a first syllable in the non-lexicalized musically encoded phrase and a duration of a second syllable in the non-lexicalized musically encoded phrase, the first syllable in the non-lexicalized musically encoded phrase being non-adjacent to the second syllable in the non-lexicalized musically encoded phrase; and

a difference between a duration of a first syllable in the lexicalized version of the non-lexicalized musically encoded phrase and a duration of a second syllable in the lexicalized version of the non-lexicalized musically encoded phrase, the first syllable in the lexicalized version of the non-lexicalized musically encoded phrase being non-adjacent to the second syllable in the lexicalized version of the non-lexicalized musically encoded phrase; and

adjusting a future lesson for the student based upon the error signal.

10. The method of claim 9, wherein the non-lexical properties of the non-lexicalized musically encoded phrase and the lexicalized version of the non-lexicalized musically encoded phrase are visually presented to a user.

11. The method of claim 9, wherein the at least one error signal is based on a difference in rhythm between the non-lexicalized musically encoded phrase and the lexicalized version of the non-lexicalized musically encoded phrase, the at least one error signal further based on at least one of a difference in prosody between the non-lexicalized musically encoded phrase and the lexicalized version of the non-lexicalized musically encoded phrase;

a difference in the total number of syllables between the non-lexicalized musically encoded phrase and the lexicalized version of the non-lexicalized musically encoded phrase;

a difference in duration in units of time between the non-lexicalized musically encoded phrase and the lexicalized version of the non-lexicalized musically encoded phrase;

a difference in pitch between respective ones of the syllables between the non-lexicalized musically encoded phrase and the lexicalized version of the non-lexicalized musically encoded phrase;

or a difference in intonation between the non-lexicalized musically encoded phrase and the lexicalized version of the non-lexicalized musically encoded phrase.

12. The method of claim 9, wherein the non-lexical properties of the non-lexicalized musically encoded phrase and the non-lexical properties of the lexicalized version of the non-lexicalized musically encoded phrase include rhythm, and further include at least one of prosody;

duration in time;

a total number of syllables;

or a pitch level of at least one syllable.

13. The method of claim 9, wherein the at least one error signal further based on a sum of the difference between:

the difference between the duration of the first syllable in the non-lexicalized musically encoded phrase and the duration of the second syllable in the non-lexicalized musically encoded phrase, the first syllable in the non-lexicalized musically encoded phrase being non-adjacent to the second syllable in the non-lexicalized musically encoded phrase; and

the difference between the duration of the first syllable in the lexicalized version of the non-lexicalized musically encoded phrase and the duration of the second syllable in the lexicalized version of the non-lexicalized musically encoded phrase, the first syllable in the lexicalized version of the non-lexicalized musically encoded phrase being non-adjacent to the second syllable in the lexicalized version of the non-lexicalized musically encoded phrase.

14. A non-transitory processor-readable medium storing code representing instructions to be executed by a processor, the code comprising code to cause the processor to:

encode a first prosodic speech signal to generate a musically encoded first prosodic speech signal by mapping each syllable of the first prosodic speech signal to a musical note;

store the musically encoded first prosodic speech signal;

audibly play the musically encoded first prosodic speech signal to a language student;

prompt the student to recite the speech segment from which the musically encoded first prosodic speech signal originated;

record an utterance from the language student in response to the prompt;

delexicalize the utterance to generate a second prosodic speech signal; and

calculate at least one error signal based on a difference between;

a difference between a duration of a first syllable in the first prosodic speech signal and a duration of a second syllable in the first prosodic speech signal, the first syllable in the first prosodic speech signal being non-adjacent to the second syllable in the first prosodic speech signal; and

a difference between a duration of a first syllable in the second prosodic speech signal and a duration of a second syllable in the second prosodic speech signal, the first syllable in the second prosodic speech signal being non-adjacent to the second syllable in the second prosodic speech signal.

15. The non-transitory processor-readable medium of claim 14, further comprising code to cause the processor to:

determine prosodic characteristics of the first prosodic speech signal and the second prosodic speech signal, the code to cause the processor to calculate includes code to cause the processor to calculate the at least one error signal based on the prosodic characteristics of the first prosodic speech signal and the second prosodic speech signal.

16. The non-transitory processor-readable medium of claim 14, further comprising code to cause the processor to:

determine prosodic characteristics of the first prosodic speech signal and the second prosodic speech signals, the code to cause the processor to calculate includes code to cause the processor to calculate the at least one error signal based on the prosodic characteristics of the first prosodic speech signal and the second prosodic speech signal, the prosodic characteristics including rhythm, the prosodic characteristics further including at least one of prosody;

duration in time;

a total number of syllables;

or a pitch level of at least one syllable.

17. The non-transitory processor-readable medium of claim 14, wherein

the at least one error signal is further based on at least one of a difference in prosody between the first prosodic speech signal and the second prosodic speech signal;

a difference in a total number of syllables between the first prosodic speech signal and the second prosodic speech signal;

a difference in duration in units of time between the first prosodic speech signal and the second prosodic speech signal;

a difference in pitch between respective syllables between the first prosodic speech signal and the second prosodic speech signal;

or a difference in intonation between the first prosodic speech signal and the second prosodic speech signal.

18. The non-transitory processor-readable medium of claim 14, wherein the musically encoded first prosodic speech signal is encoded in a musical instrument digital interface (MIDI) framework.

19. The non-transitory processor-readable medium of claim 14, wherein the code to cause the processor to encode includes code to cause the processor to encode the first prosodic speech signal at least partially by converting a frequency of the first prosodic speech signal to a musical instrument digital interface (MIDI) pitch note number.

20. The non-transitory processor-readable medium of claim 14, wherein the at least one error signal further based on a sum of the difference between:

the difference between the duration of the first syllable in the first prosodic speech signal and the duration of the second syllable in the first prosodic speech signal, the first syllable in the first prosodic speech signal being non-adjacent to the second syllable in the first prosodic speech signal; and

the difference between the duration of the first syllable in the second prosodic speech signal and the duration of the second syllable in the second prosodic speech signal, the first syllable in the second prosodic speech signal being non-adjacent to the second syllable in the second prosodic speech signal.