Method of identifying the language of a textual passage using short word and/or n-gram comparisons
First Claim
1. A method of determining the language of a textual passage, the method comprising the steps of:
- (a) parsing said textual passage into plural n-grams;
(b) comparing each of said plural n-grams with a plurality of databases, wherein each of said databases comprises multiple n-grams associated with a specific language;
(c) determining an initial weight for each of said plural n-grams, per language, by calculating the frequency with which each of said plural n-grams appears in each of said databases and dividing said frequency by the total number of the multiple n-grams in said respective database;
(d) determining the number of said databases within which each of said plural n-grams appear;
(e) altering said initial weight for each of said plural n-grams by multiplying said initial weight with the inverse of said number of databases within which each of said plural n-grams appear to produce an altered weight for each of said plural n-grams;
(f) multiplying, for each of the plural n-grams, a number of times that n-gram appears in the textual passage by the altered weight of that n-gram from step (e) to produce a product for each n-gram, per language, and summing those products to produce a language passage weight for each language for the textual passage;
(g) determining, based upon a comparison of the language passage weights from step (f) the most likely language or languages for the textual passage.
2 Assignments
0 Petitions
Accused Products
Abstract
A method and system identifying the language of a textual passage is disclosed. The method and system includes parsing the textual passage into n-grams and assigning an initial weight to each n-gram, and adjusting the weight initially assigned to a word or n-gram parsed from the textual passage. The initially assigned weight is adjusted in a manner proportionate to the inverse of the number of languages within which such words or n-grams appear. Reducing the weight assigned to such words or n-grams diminishes—without completely eliminating—their importance in comparison to other words or n-grams parsed from the same textual passage when determining the language of a passage. The method and system of the present invention appropriately weighs the short words or n-grams common to multiple languages without affecting the short words or n-grams that are uncommon to several languages.
-
Citations
19 Claims
-
1. A method of determining the language of a textual passage, the method comprising the steps of:
-
(a) parsing said textual passage into plural n-grams; (b) comparing each of said plural n-grams with a plurality of databases, wherein each of said databases comprises multiple n-grams associated with a specific language; (c) determining an initial weight for each of said plural n-grams, per language, by calculating the frequency with which each of said plural n-grams appears in each of said databases and dividing said frequency by the total number of the multiple n-grams in said respective database; (d) determining the number of said databases within which each of said plural n-grams appear; (e) altering said initial weight for each of said plural n-grams by multiplying said initial weight with the inverse of said number of databases within which each of said plural n-grams appear to produce an altered weight for each of said plural n-grams; (f) multiplying, for each of the plural n-grams, a number of times that n-gram appears in the textual passage by the altered weight of that n-gram from step (e) to produce a product for each n-gram, per language, and summing those products to produce a language passage weight for each language for the textual passage; (g) determining, based upon a comparison of the language passage weights from step (f) the most likely language or languages for the textual passage. - View Dependent Claims (2, 3, 4, 5, 6, 7, 16)
-
-
8. A method of determining the language of a textual passage, the method comprising the steps of:
-
(a) filtering plural short words from a textual passage; (b) comparing each of said plural short words against a plurality of databases, wherein each of said databases comprises multiple short words associated with a different language; (c) determining an initial weight for each of said plural short words, per language, by calculating the frequency with which each of said plural short words appears in each of said databases and dividing said frequency by the total number of the multiple short words in said respective database; (d) determining the number of said databases within which each of said plural short words appear; (e) altering said initial weight for each of said plural short words by multiplying said initial weight with the inverse of said number of databases within which each of said plural short words appear to produce an altered weight for each of said plural short words; (f) multiplying, for each of the plural short words, a number of times that short word appears in the textual passage by the altered weight of that short word from step (e) to produce a product for each short word, per language, and summing those products to produce a language passage weight for each language for the textual passage; (g) determining, based upon a comparison of the language passage weights from step (f) the most likely language or languages for the textual passage. - View Dependent Claims (17)
-
-
9. A method of determining the language of a textual passage, the method comprising the steps of:
-
(a) filtering plural short words from a textual passage and parsing said textual passage into plural n-grams; (b) comparing each of said plural n-grams and said plural short words against a plurality of databases, wherein each of said databases comprises multiple n-grams and short words associated with a different language; (c) determining an initial weight for each of said plural n-grams and said plural short words, per language; (d) determining the number of said databases within which each of said plural n-grams and said plural short words appear; (e) altering said initial weight for each of said plural n-grams and said plural short words by multiplying said initial weight with the inverse of said number of databases within which each of said plural n-grams and said plural short words appear to produce an altered weight for each of said plural n-grams and said plural short words; (f) multiplying, for each of the plural n-grams and each of the plural short words, a number of times that n-gram or short word appears in the textual passage by the altered weight of that n-gram or short word from step (e) to produce a product for each n-gram and each short word, per language, and summing those products to produce a language passage weight for each language for the textual passage; (g) determining, based upon a comparison of the language passage weights from step (f) the most likely language or languages for the textual passage. - View Dependent Claims (18)
-
-
10. A system for determining the language of a textual passage, said system comprising a central processing unit coupled to a memory system and a display, wherein said central processing unit operates according to a program retrieved from said memory system, wherein said program is configured to cause the central processing unit to execute steps comprising:
-
(a) receiving a textual passage; (b) parsing said textual passage into plural n-grams; (c) comparing each of said plural n-grams against a plurality of databases, wherein each of said databases comprises multiple n-grams associated with a different language; (d) assigning an initial weight to each of said plural n-grams, per language, by calculating the frequency with which each of said plural n-grams appears in each of said databases and dividing said frequency by the total number of n-grams in said respective database; (e) calculating the number of said databases within which each of said plural n-grams appear; (f) altering said initial weight assigned to each of said plural n-grams by multiplying said initial weight with the inverse of said number of databases within which each of said plural n-grams appear to produce an altered weight for each of said plural n-grams; (g) multiplying, for each of the plural n-grams, a number of times that n-gram appears in the textual passage by the altered weight of that n-gram from step (f) to produce a product for each n-gram, per language, and summing those products to produce a language passage weight for each language for the textual passage; (h) determining, based upon a comparison of the language passage weights from step (g) the most likely language or languages for the textual passage. - View Dependent Claims (11, 12, 13, 14, 15, 19)
-
Specification