Automatic cognate detection in a computer-assisted language learning system

• US 9,400,781 B1
• Filed: 02/08/2016
• Issued: 07/26/2016
• Est. Priority Date: 02/08/2016
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Ward, Robert T., Navratil, Jiri, Roukos, Salim
• Primary Examiner(s): Yen, Eric
• Application Number:
  US15/018,014
• Time in Patent Office: 169 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/345 Summarisation for human users
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 40/216 using statistical methods
• CPC Class Codes: G06F 40/247 Thesauruses; Synonyms
• CPC Class Codes: G06F 40/253 Grammatical analysis; Style critique
• CPC Class Codes: G06F 40/268 Morphological analysis
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/44 Statistical methods, e.g. probability models
• CPC Class Codes: G06F 40/45 Example-based machine translation; Alignment
• CPC Class Codes: G06F 40/47 Machine-assisted translation, e.g. using translation memory
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G10L 15/10 using distance or distortion measures between unknown speech and reference templates
• CPC Class Codes: G10L 2015/081 Search algorithms, e.g. Baum-Welch or Viterbi

Abstract

According to an aspect, a first word in a first language and a second word in a second language in a bilingual corpus are stemmed. A probability for aligning the first stem and the second stem and a distance metric between the normalized first stem and the normalized second stem are calculated. The first word and the second word are identified as a cognate pair when the probability and the distance metric meet a threshold criterion and stored as a cognate pair in a set of cognates. A candidate sentence in the second language is retrieved from a corpus. The candidate sentence is filtered by the active vocabulary of a user in the second language and the set of cognates. A sentence quality score is calculated for the candidate sentence; and the candidate sentence is ranked for presentation to the user based on the sentence quality scorer.

Claims

1. A computer-implemented method for automatic cognate detection, the method comprising:

stemming, by a processor, a first word in a first language in a bilingual corpus to obtain a first stem and a second word in a second language in the bilingual corpus to obtain a second stem;

calculating, by the processor, a probability for aligning the first stem and the second stem;

normalizing, by the processor, the first stem and the second stem;

calculating, by the processor, a distance metric between the normalized first stem and the normalized second stem;

identifying, by the processor, the first word and the second word as a cognate pair when the probability and the distance metric meet a threshold criterion;

storing the cognate pair in a set of cognates;

retrieving, by the processor, a candidate sentence in the second language from a corpus;

filtering, by the processor, the candidate sentence by an active vocabulary of a user in the second language and the set of cognates;

calculating, by the processor, a sentence quality score for the candidate sentence;

ranking, by the processor, the candidate sentence based on the sentence quality score; and

presenting the ranked candidate sentence as a pure or combined audio, graphic, textual, or video stimulus to the user.

2. The method of claim 1, wherein the probability is calculated with a Viterbi algorithm.

3. The method of claim 1, wherein the probability is calculated with an expectation maximization algorithm.

4. The method of claim 1, wherein the distance metric comprises an edit distance.

5. The method of claim 1, wherein calculating the distance metric comprises

mapping between characters or groups of characters of the first and second languages, wherein the first language and second language have different alphabets;

or

phonetic transformation of the first stem and the second stem.

6. The method of claim 1, wherein stemming is performed using predetermined language-specific rules.

7. The method of claim 1, wherein the sentence quality score is based on user topic interest, richness in words stored in the set of cognates, richness in words in the active vocabulary in the second language, grammatical correctness, or parallelism.

8. The method of claim 1, wherein filtering a candidate sentence by the active vocabulary and the set of cognates comprises:

flagging each word in the candidate sentence which is present in the set of cognates; and

flagging all words in the candidate sentence which are present in the active vocabulary of the second language.

9. A system for automatic cognate detection, the system comprising:

a memory having computer readable instructions; and

one or more processors for executing the computer readable instructions, the one or more processors being configured to;

stem a first word in a first language in a bilingual corpus to obtain a first stem and a second word in a second language in the bilingual corpus to obtain a second stem;

calculate a probability for aligning the first stem and the second stem;

normalize the first stem and the second stem;

calculate a distance metric between the normalized first stem and the normalized second stem;

identify the first word and the second word as a cognate pair when the probability and the distance metric meet a threshold criterion;

store the cognate pair in a set of cognates;

retrieve a candidate sentence in the second language from a corpus;

filter the candidate sentence by an active vocabulary of a user in the second language and the set of cognates;

calculate a sentence quality score for the candidate sentence;

rank the candidate sentence based on the sentence quality score; and

present the ranked candidate sentence as a pure or combined audio, graphic, textual, or video stimulus to the user.

10. The system of claim 9, wherein the probability is calculated with a Viterbi algorithm.

11. The system of claim 9, wherein the probability is calculated with an expectation maximization algorithm.

12. The system of claim 9, wherein the distance metric comprises an edit distance.

13. The system of claim 9, wherein calculating the distance metric comprises

mapping between characters or groups of characters of the first and second languages, wherein the first language and second language have different alphabets;

or

phonetic transformation of the first stem and the second stem.

14. The system of claim 9, wherein stemming is performed using predetermined language-specific rules.

15. The system of claim 9, wherein the sentence quality score is based on user interest, richness in stored cognate pairs, richness in words in the active vocabulary in the second language, grammatical correctness, or parallelism.

16. The system of claim 9, wherein filtering a candidate sentence by the active vocabulary and the set of cognates comprises:

flagging each word in the candidate sentence which is present in the set of cognates; and

flagging all words in the candidate sentence which are present in the active vocabulary of the second language.

17. A computer program product for automatic cognate detection, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to perform:

stemming a first word in a first language in a bilingual corpus to obtain a first stem and a second word in a second language in the bilingual corpus to obtain a second stem;

calculating a probability for aligning the first stem and the second stem;

normalizing the first stem and the second stem;

calculating a distance metric between the normalized first stem and the normalized second stem;

identifying the first word and the second word as a cognate pair when the probability and the distance metric meet a threshold criterion;

storing the cognate pair in a set of cognates;

retrieving a candidate sentence in the second language from a corpus;

filtering the candidate sentence by an active vocabulary of a user in the second language and the set of cognates;

calculating a sentence quality score for the candidate sentence;

ranking the candidate sentence based on the sentence quality score; and

presenting the ranked candidate sentence as a pure or combined audio, graphic, textual, or video stimulus to the user.