FOREIGN LANGUAGE LEARNING APPARATUS AND METHOD FOR CORRECTING PRONUNCIATION THROUGH SENTENCE INPUT

• US 20150134338A1
• Filed: 11/22/2013
• Published: 05/14/2015
• Est. Priority Date: 11/13/2013
• Status: Active Grant

• Current Assignee: Weaversmind Inc.
• Original Assignee: Weaversmind Inc.
• Inventors: JUNG, SUNGEUN
• Granted Patent: US 9,520,143 B2
• Time in Patent Office: Days
• US Class Current: 704/260
• CPC Class Codes: G09B 19/06 Foreign languages with audible presentation of material to be studied G09B5/04
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 25/60 for measuring the quality of voice signals

Abstract

Provided are a foreign language learning apparatus and method using a function of reading an input sentence in voice via a Text To Speech (TTS) engine. The foreign language learning apparatus and method correct pronunciation through sentence input. The foreign language learning apparatus includes a sentence input unit for receiving a first sentence from a user; a linked letter detection unit for detecting at least one letter corresponding to at least one linking rule; a linked letter removal unit for removing the letter and generating a second sentence by inserting a linking code; a partial waveform generation unit for generating one or more partial waveforms using the TTS engine; an input waveform generation unit for converting a voice corresponding to the first sentence into an input waveform; and a matching degree calculation unit for calculating a matching degree and a partial matching degree.

Claims

1. A foreign language learning apparatus using a function of reading an input sentence in voice via a Text To Speech (TTS) engine, the foreign language learning apparatus correcting pronunciation through sentence input, comprising:

a sentence input unit for receiving a first sentence from a user;

a linked letter detection unit for detecting at least one letter corresponding to at least one linking rule by searching letters that form the first sentence received via the sentence input unit;

a linked letter removal unit for removing the letter corresponding to the linking rule and detected by the linked letter detection unit and generating a second sentence by inserting a linking code into a part from which the letter has been removed;

a partial waveform generation unit for generating one or more partial waveforms using the TTS engine with respect to a portion from a start point of the second sentence generated by the linked letter removal unit before the part into which the linking code has been inserted, a portion from the part into which the linking code has been inserted before a part into which a subsequent linking code has been inserted, and a portion from the part into which the subsequent linking code has been inserted to an end point of the second sentence;

an input waveform generation unit for converting a voice corresponding to the first sentence received via the sentence input unit into an input waveform when the voice is received from a user; and

a matching degree calculation unit for calculating a matching degree by comparing the input waveform generated by the input waveform generation unit with the one or more partial waveforms generated by the partial waveform generation unit and calculating a partial matching degree of a part having a highest matching degree in the partial waveforms by detecting the part having the highest matching degree while moving from a beginning of the input waveform to an end thereof in an order in which the one or more partial waveforms are disposed.

2. The foreign language learning apparatus of claim 1, wherein:

the linked letter detection unit comprises;

a word detection unit for detecting an identical word by determining whether or not a word including the letter corresponding to the linking rule is identical with a word previously stored in a data storage unit when the letter corresponding to the linking rule is detected by the linked letter detection unit; and

a word waveform insertion unit for inserting a word waveform corresponding to the previously stored word into a part corresponding to the word detected by the word detection unit;

wherein the word waveform comprises one or more phonemic waveforms that corresponds to phonemes for respective letters that form the word;

wherein the linked letter removal unit removes a phonemic waveform of the part corresponding to the letter that belongs to the letters of the word corresponding to the word waveform and that has been removed by the linked letter removal unit;

wherein the partial waveform generation unit generates the one or more partial waveforms using the TTS engine with respect to a portion from the start point of the second sentence generated by the linked letter removal unit to a letter prior to the part into which the word waveform has been inserted, a portion from a letter posterior to the part into which the word waveform has been inserted to a letter prior to a part into which a subsequent word waveform has been inserted, and a portion from a letter posterior to the part into which the word waveform has been inserted to the end point of the second sentence; and

wherein the matching degree calculation unit calculates a matching degree by comparing the input waveform generated by the input waveform generation unit with one or more word waveforms generated by the linked letter removal unit and then calculating a word matching degree of a part having a highest matching degree in each of the word waveforms by detecting the part having the highest matching degree while moving from the beginning of the input waveform to the end thereof in the order in which the one or more word waveforms are disposed.

3. The foreign language learning apparatus of claim 2, further comprising:

a waveform speed calculation unit for detecting a partial waveform or a word waveform corresponding to a highest matching degree of the matching degrees calculated by the matching degree calculation unit and then calculating a comparative speed by comparing a playback length of an input waveform of a corresponding part with a playback length of the partial waveform or the word waveform corresponding to the highest matching degree; and

a matching degree re-calculation unit for compressing or extending a total playback length of the input waveform based on the comparative speed calculated by the waveform speed calculation unit and then calculating the matching degree between each of the partial waveforms or the word waveforms and the input waveform of the corresponding part.

4. The foreign language learning apparatus of claim 1, wherein the linked letter detection unit comprises:

a first linking rule in which a letter corresponding to a middle consonant of three consonants is detected if the three consonants are consecutive in one word or two words;

a second linking rule in which “

p,”

“

t,”

or “

k”

is detected if “

p,”

“

t,”

or “

k”

is placed behind “

s”

;

a third linking rule in which a rearmost consonant of a front word and a foremost vowel of a rear word are detected if the two words are consecutive, the front word ends with a consonant and the rear word starts with a vowel; and

a fourth linking rule in which “

d”

or “

t”

is detected if “

d”

or “

t”

is placed between vowels.

5. The foreign language learning apparatus of claim 2, further comprising a disposition sequence determination unit for sending a re-input signal to an output unit if the partial matching degree and the word matching degree of the input waveform calculated by the matching degree calculation unit are not identical with a disposition order of the second sentence.

6. The foreign language learning apparatus of claim 2, wherein the data storage unit stores a standard pronunciation video corresponding to each of the word waveforms.

7. The foreign language learning apparatus of claim 2, wherein the matching degree calculation unit calculates a matching percentage for each word by comparing the input waveform with the word waveform on a word basis, calculates a matching percentage for each part by comparing the input waveform with the partial waveform on a partial waveform basis, calculates a matching percentage for each sentence by summing all the matching percentages for each word and all the matching percentages for each part and then dividing each of the sums by a total number of matching percentages for each word and matching percentages for each part, and sends the calculated matching percentage for each word, the calculated matching percentage for each part, and the calculated matching percentage for each sentence to an output unit.

8. The foreign language learning apparatus of claim 7, further comprising a learning achievement display unit for receiving a standard pronunciation video corresponding to a word waveform whose matching percentage for each word is lower than a predetermined reference matching degree from the data storage unit and outputting the received standard pronunciation video to the output unit.

9. The foreign language learning apparatus of claim 8, wherein after the one or more word waveforms have been output to the output unit, a voice corresponding to each of the one or more word waveforms is received from a user via an input unit, a matching percentage for each word corresponding to each of the word waveforms is generated, and the learning achievement display unit calculates an average value for each word on a word waveform basis and outputs the calculated average value for each word to the output unit on a word waveform basis.

10. The foreign language learning apparatus of claim 9, wherein the learning achievement display unit performs individual customized learning based on the average value for each word waveform.

11. The foreign language learning apparatus of claim 10, wherein:

the learning achievement display unit outputs a standard pronunciation video corresponding to some or all of word waveforms whose average values are lower than a reference matching degree, searches for sentence data including the word waveforms in order to improve a matching percentage for each word of each of the output word waveforms, and sends the retrieved sentence data to the output unit; and

the data storage unit stores one or more pieces of sentence data.

12. The foreign language learning apparatus of claim 1, further comprising a review unit for outputting previously output word waveforms again when a review request signal is received from the sentence input unit, that is, periodically outputting the word waveforms in a review cycle that corresponds to an Ebbinghaus forgetting curve.

13. A foreign language learning method using a function of reading an input sentence in voice through a Text To Speech (TTS) engine, the foreign language learning method correcting pronunciation through sentence input, comprising the steps of:

(a) receiving a first sentence from a user via a sentence input unit;

(b) detecting, by a linked letter detection unit, at least one letter corresponding to at least one linking rule by searching letters that form the first sentence received via the sentence input unit;

(c) removing, by a linked letter removal unit, the letter corresponding to the linking rule detected by the linked letter detection unit, and generating, by the linked letter removal unit, a second sentence by inserting a linking code into a part from which the letter has been removed;

(d) generating, by a partial waveform generation unit, one or more partial waveforms using the TTS engine with respect to a portion from a start point of the second sentence generated by the linked letter removal unit before a part into which the linking code has been inserted, a portion from the part into which the linking code has been inserted before a part into which a subsequent linking code has been inserted, and a portion from the part into which the subsequent linking code has been inserted to an end point of the second sentence;

(d) converting, by an input waveform generation unit, a voice corresponding to the first sentence received through the sentence input unit into an input waveform when the voice is received from a user; and

(e) calculating, by a matching degree calculation unit, a matching degree by comparing the input waveform generated by the input waveform generation unit with the one or more partial waveforms generated by the partial waveform generation unit, and calculating, by the matching degree calculation unit, a partial matching degree of a part having a highest matching degree in each of the partial waveforms by detecting the part having the highest matching degree while moving from a beginning of the input waveform to an end thereof in an order in which the one or more partial waveforms are disposed.

14. The foreign language learning method of claim 13, wherein:

step (b) comprises the steps of;

(b1) detecting, by a word detection unit, an identical word by determining whether or not a word including the letter corresponding to the linking rule is identical with a word previously stored in a data storage unit when the letter corresponding to the linking rule is detected by the linked letter detection unit; and

(b2) inserting, by a word waveform insertion unit, a word waveform corresponding to the previously stored word into a part corresponding to the word detected by the word detection unit;

the word waveform comprises one or more phonemic waveforms that corresponds to phonemes for respective letters that form the word;

the linked letter removal unit removes a phonemic waveform of the part corresponding to the letter that belongs to the letters of the word corresponding to the word waveform and that has been removed by the linked letter removal unit;

the partial waveform generation unit generates the one or more partial waveforms using the TTS engine with respect to a portion from the start point of the second sentence generated by the linked letter removal unit to a letter prior to the part into which the word waveform has been inserted, a portion from a letter posterior to the part into which the word waveform has been inserted to a letter prior to a part into which a subsequent word waveform has been inserted, and a portion from a letter posterior to the part into which the word waveform has been inserted to the end point of the second sentence; and

the matching degree calculation unit calculates a matching degree by comparing the input waveform generated by the input waveform generation unit with one or more word waveforms generated by the linked letter removal unit and then calculating a word matching degree of a part having a highest matching degree in each of the word waveforms by detecting the part having the highest matching degree while moving from the beginning of the input waveform to the end thereof in the order in which the one or more word waveforms are disposed.

15. The foreign language learning method of claim 13, further comprising steps of:

(h) detecting, by a waveform speed calculation unit, a partial waveform or a word waveform corresponding to a highest matching degree of the matching degrees calculated by the matching degree calculation unit, and then calculating, by the waveform speed calculation unit, a comparative speed by comparing a playback length of an input waveform of a corresponding part with a playback length of the partial waveform or the word waveform corresponding to the highest matching degree; and

(i) compressing or extending, by a matching degree re-calculation unit, a total playback length of the input waveform based on the comparative speed calculated by the waveform speed calculation unit, and then calculating, by the matching degree re-calculation unit, the matching degree between each of the partial waveforms or the word waveforms and the input waveform of the corresponding part.

16. The foreign language learning method of claim 13, wherein the linked letter detection unit comprises:

a first linking rule in which a letter corresponding to a middle consonant of three consonants is detected if the three consonants are consecutive in one word or two words;

a second linking rule in which “

p,”

“

t,”

or “

k”

is detected if “

p,”

“

t,”

or “

k”

is placed behind “

s”

;

a third linking rule in which a rearmost consonant of a front word and a foremost vowel of a rear word are detected if the two words are consecutive, the front word ends with a consonant and the rear word starts with a vowel; and

a fourth linking rule in which “

d”

or “

t”

is detected if “

d”

or “

t”

is placed between vowels.

17. The foreign language learning method of claim 14, further comprising step (g) of sending, by a disposition sequence determination unit, a re-input signal to an output unit if the partial matching degree and the word matching degree of the input waveform calculated by the matching degree calculation unit is not identical with a disposition order of the second sentence.

18. The foreign language learning method of claim 14, wherein the data storage unit stores a standard pronunciation video corresponding to each of the word waveforms.

19. The foreign language learning method of claim 13, wherein the matching degree calculation unit calculates a matching percentage for each word by comparing the input waveform with the word waveform on a word basis, calculates a matching percentage for each part by comparing the input waveform with the partial waveform on a partial waveform basis, calculates a matching percentage for each sentence by summing all the matching percentages for each word and all the matching percentages for each part and then dividing each of the sums by a total number of matching percentages for each word and matching percentages for each part, and sends the calculated matching percentage for each word, the calculated matching percentage for each part, and the calculated matching percentage for each sentence to an output unit.

20. The foreign language learning method of claim 19, further comprising step (j) of receiving, by a learning achievement display unit, a standard pronunciation video corresponding to a word waveform whose matching percentage for each word is lower than a predetermined reference matching degree from the data storage unit, and outputting, by the learning achievement display unit, the received standard pronunciation video to the output unit.