Lemmatizing, stemming, and query expansion method and system

• US 8,473,279 B2
• Filed: 06/01/2009
• Issued: 06/25/2013
• Est. Priority Date: 05/30/2008
• Status: Active Grant

• Current Assignee: Eiman Al-Shammari
• Original Assignee: Eiman Al-Shammari
• Inventors: Al-Shammari, Eiman Tamah
• Primary Examiner(s): Desir, Pierre-Louis
• Assistant Examiner(s): KOVACEK, DAVID M
• Application Number:
  US12/476,238
• Publication Number:
  US 20100082333A1
• Time in Patent Office: 1,485 Days
• Field of Search: 704 1- 10, 704E11001-E11007, 704E13001-E13014, 707705-747, 707/E17.001, 707/E17.143
• US Class Current: 704/9
• CPC Class Codes: G06F 16/3335 Syntactic pre-processing, e.g. stopword elimination, stemming
• CPC Class Codes: G06F 16/3338 Query expansion
• CPC Class Codes: G06F 40/242 Dictionaries
• CPC Class Codes: G06F 40/268 Morphological analysis

Abstract

A method of stemming text and system therefore are described. The method comprises removing stop words from a document based on at least one stop word entry in an array of stop words and flagging as nouns words determined to be attached to definite articles and preceded by a noun array entry in an array of stop words preceding at least one noun; adding flagged nouns to a noun dictionary; flagging as verbs words determined to be preceded by an verb array entry in an array of stop words preceding at least one verb; adding flagged verbs to a verb dictionary; searching the document for nouns and verbs based on the flagged nouns and the flagged verbs; removing remaining stop words subsequent to searching the document; applying light stemming on the flagged nouns; applying a root-based stemming on the flagged verbs; and storing the stemmed document.

Claims

1. A computer-implemented method of stemming Arabic text comprising:

removing stop words from a document comprising Arabic text based on at least one stop word entry in an array of stop words;

flagging as nouns words determined to be attached to definite articles and preceded by a noun array entry in an array of stop words preceding at least one noun;

adding flagged nouns to a noun dictionary;

flagging as verbs words determined to be preceded by a verb array entry in an array of stop words preceding at least one verb;

adding flagged verbs to a verb dictionary;

searching the document for nouns and verbs based on the flagged nouns and the flagged verbs;

removing remaining stop words subsequent to searching the document;

applying light stemming on the flagged nouns only;

Applying a root-based stemming on the flagged verbs only; and

storing the stemmed document.

2. The method as claimed in claim 1, further comprising:

prior to the first step of removing of stop words, categorizing the stop words into useful and useless stop words.

3. The method as claimed in claim 1, further comprising:

generating the array of stop words using three methods;

English stop words translation into Arabic, identification of common words in arbitrary Arabic documents, and manual search of synonyms to previously identified stop words.

4. A system for stemming Arabic text comprising:

a processor;

a memory, storing an input document comprising Arabic text, a stemming process, a noun stop word array of stop words preceding at least one noun, and a verb stop word array of stop words preceding at least one verb, and coupled with the processor, wherein the stemming process comprises a sequence of one or more instructions which, when executed by the processor, causes the processor to;

remove stop words from the input document based on at least one noun stop word entry in the noun stop word array;

flag as nouns words determined to be attached to definite articles and preceded by a noun stop word array entry;

flag as verbs words determined to be preceded by a verb stop word array entry;

search the document for nouns and verbs based on the flagged nouns and the flagged verbs;

remove remaining stop words subsequent to searching the document to create a stemmed document;

apply light stemming on the flagged nouns only;

apply a root-based stemming on the flagged verbs only; and

store the stemmed document to the memory.

5. The system as claimed in claim 4, further comprising instructions to cause the processor to categorize the stop words into useful and useless stop words prior to the first removing of stop words.

6. The system as claimed in claim 4, further comprising instructions to cause the processor to generate the array of stop words using three methods:

English stop words translation into Arabic, identification of common words in arbitrary Arabic documents, and manual search of synonyms to previously identified stop words.

7. A non-transitory computer-readable medium storing instruction which, when executed by a processor, cause the processor to perform a method comprising:

removing stop words from a document comprising Arabic text based on at least one stop word entry in an array of stop words;

flagging as nouns words determined to be attached to definite articles and preceded by a noun array entry in an array of stop words preceding at least one noun;

adding flagged nouns to a noun dictionary;

flagging as verbs words determined to be preceded by a verb array entry in an array of stop words preceding at least one verb;

adding flagged verbs to a verb dictionary;

searching the document for nouns and verbs based on the flagged nouns and the flagged verbs;

removing remaining stop words subsequent to searching the document;

applying light stemming on the flagged nouns only;

applying a root-based stemming on the flagged verbs only; and

storing the stemmed document.