Determining a natural language shift in a computer document

• US 5,913,185 A
• Filed: 12/20/1996
• Issued: 06/15/1999
• Est. Priority Date: 08/19/1996
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Paulsen, Robert Charles Jr., Martino, Michael John
• Primary Examiner(s): Thomas, Joseph
• Application Number:
  US08/772,213
• Time in Patent Office: 907 Days
• Field of Search: 704/1, 704/8, 704/9, 704/10, 707/531, 707/532, 707/533, 707/535, 707/536
• US Class Current: 704/8
• CPC Class Codes: G06F 40/216 using statistical methods
• CPC Class Codes: G06F 40/263 Language identification

Abstract

Language shift points in a computer document written in a plurality of natural languages are determined. An interval is defined on and moved through a text document in a computer memory, the interval contains a portion of the text in the document. As the interval is moved through the document for each position of the interval, a probability that the text in the interval is written in each of a plurality of candidate languages is determined for the position. For the first position of the interval, generally the beginning of the document, a first candidate language is classified as the current language if it has the highest probability of all the candidate languages within the interval. A language shift point in the document is identified where the relative probability of a second candidate language is higher than the current language at a new position of the interval. At this point, the second candidate language is classified as the current language in the document after the language shift point. The process continues to identify other language shift points in the document.

Claims

1. A method for detecting language shift points in a computer document written in a plurality of natural languages, comprising the steps of:

moving an interval through a text document in a computer memory, the interval containing a plurality of words in the document;

for each position of the interval, determining a probability that text in the interval is written in each of a plurality of candidate languages according to a respective number of matches of words in the interval with words in each of a plurality of word lists of a few common words selected from each respective candidate language;

for a first position of the interval, classifying a first candidate language having the highest probability as the current language within the interval;

finding a language shift point in the document where the probability that a second candidate language is higher than the current language for a new position of the interval; and

classifying the second candidate language as the current language in the document after the language shift point.

2. The method as recited in claim 1 wherein the determining step comprises the steps of:

comparing words in the interval with word lists comprised of common words in each of the candidate languages;

counting a number of words in the interval which match the common words in each respective word list; and

determining the relative probability of each candidate language based on a cumulative count of words in the interval which match for each respective candidate language.

3. The method as recited in claim 2 wherein the count for each word is weighted by a factor in the cumulative count.

4. The method as recited in claim 2 wherein the common words selected for the word lists are chosen so that strong aliases between candidate languages are not included.

5. The method as recited in claim 4 wherein the candidate languages to be compared to the document are selected before the language shift determination from all the candidates languages for which word lists are stored, and wherein the strong aliases between the selected candidate languages are removed from master word lists for the selected candidate languages composed of common words in each selected candidate language.

6. The method as recited in claim 2 wherein partial words are used in the comparing step.

7. The method as recited in claim 1 further comprising the steps of:

searching the document for syntax cues proximate to the found language shift point; and

responsive to finding syntax cues, shifting the language shift point if indicated by the found syntax cues.

8. The method as recited in claim 7, wherein the method operates in batch mode, so that a plurality of language shift points are found in a first pass through the document and syntax cues proximate to the language shift points are searched for in a second pass through the document.

9. The method as recited in claim 1 further comprising the steps of:

performing a language related operation on the document according to a first dictionary for the first candidate language prior to the language change point; and

performing the language related operation on the document according to a second dictionary for the second candidate language after the language change point.

10. A system including processor and memory for detecting language shift points in a computer document written in a plurality of natural languages, comprising:

interval defining means for moving an interval through a computer document in the memory, the interval containing a plurality of words in the document;

language comparing means for determining for each position of the interval a probability that text in the interval is written in each of a plurality of candidate languages according to a respective number of matches of words in the interval with words in each of a plurality of word lists of a few common words selected from each respective candidate language;

language determining means for determining for each position of the interval the candidate language having the highest relative probability within the interval;

language shift determining means for finding language shift points in the document according to the positions of the interval where the candidate language having the highest relative probability changes from prior positions.

11. The system as recited in claim 10 wherein the language comparing means comprises:

means for comparing words in the interval with word lists comprised of common words in each of the candidate languages;

means for determining a number of words in the interval which match the common words in each respective word list; and

means for determining the probability of each candidate language based on the number of words in the interval which match for each respective candidate language.

12. The system as recited in claim 11 wherein the common words in the word lists are chosen do not include strong aliases between candidate languages being compared.

13. The system as recited in claim 12 wherein the candidate languages to be compared to the document are selected from all the candidate languages for which word lists are stored, and wherein the strong aliases between the selected candidate languages are removed from master word lists for the selected candidate languages before language shift determination takes place.

14. The system as recited in claim 11 wherein the words in the word lists are stored as bits in N×

N bit tables, where each bit represents a given letter pair at a particular place in a word in one of the candidate languages.

15. The system as recited in claim 14 wherein the bit tables are arranged in tableaus, each tableau for storing words of a particular length for a given candidate language.

16. The system as recited in claim 11 wherein the interval contains non adjacent words.

17. The system as recited in claim 10 further comprising dictionaries for performing a language related operation on the document for the candidate languages.

18. A computer program product in a computer readable medium for detecting language shift points in a computer document written in a plurality of natural languages, comprising:

means for moving an interval through a computer document in a computer memory, the interval containing a plurality of words in the document;

means for determining a probability that text in the interval is written in each of a plurality of candidate languages within the interval according to a respective number of matches of words in the interval with words in each of a plurality of word lists of a few common words selected from each respective candidate language; and

means for finding language shift points in the document according to changes in the relative probabilities of the candidate language as the interval changes positions in the document.

19. The product as recited in claim 18 further comprising:

means for comparing words in the interval with word lists comprised of common words in each of the candidate languages;

means for determining a number of words in the interval which match the common words in each respective word list; and

means for determining the probability of each candidate language based on the number of words in the interval which match for each respective candidate language.

20. The product as recited in claim 19 wherein the common words in the word lists are chosen do not include strong aliases between candidate languages being compared.

21. The product as recited in claim 20 further comprising:

means for selecting the candidate languages to be compared to the document from a plurality of candidate languages for which word lists are stored;

a master word list for each candidate language; and

means for removing the strong aliases between the selected candidate languages from the master word lists for the selected candidate languages before language shift determination takes place.

22. The product as recited in claim 18 further comprising dictionaries for performing a language related operation on the document for the candidate languages.

23. The product as recited in claim 18 further comprising:

means for searching the document for syntax cues proximate to the language shift points; and

means responsive to finding syntax cues for shifting the language shift point if indicated by the found syntax cues.

24. A method for detecting language shift points in a computer document written in a plurality of natural languages, comprising the steps of:

selecting successive pluralities of words located at successive locations in the document;

for each plurality of words, recognizing no words or one or more words, but not all words, as being members of each of a respective candidate language;

for each plurality of words, classifying the plurality of words as being written in a respective language according to a greatest number of words in the respective plurality of words being recognized as members of the respective language; and

finding language shift points in the document according to where the respective languages of successive pluralities of words change according to the classifying step.

25. The method as recited in claim 24, wherein each successive plurality of words shares a predetermined number of words with a predecessor plurality of words and a successor plurality of words.

26. A method for detecting language shift points in a computer document written in a plurality of natural languages, comprising the steps of:

selecting successive pluralities of words located at successive locations in the document, wherein each plurality of words includes a first number of words of which a second number of words is shared by a next plurality of words;

for each plurality of words, classifying the plurality of words as being written in a respective language according to a greatest number of words in the respective plurality of words being recognized as members of the respective language; and

finding language shift points in the document according to where the respective languages of successive pluralities of words change according to the classifying step.