System and method for disambiguating non diacritized arabic words in a text

• US 8,041,559 B2
• Filed: 12/09/2005
• Issued: 10/18/2011
• Est. Priority Date: 12/10/2004
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: El-Shishiny, Hisham
• Primary Examiner(s): Jackson; Jakieda R
• Application Number:
  US11/299,220
• Publication Number:
  US 20060129380A1
• Time in Patent Office: 2,139 Days
• Field of Search: 704/9, 704/10
• US Class Current: 704/9
• CPC Class Codes: G06F 40/232 Orthographic correction, e.g. spell checking or vowelisation
• CPC Class Codes: G06F 40/53 Processing of non-Latin text kana-to-kanji conversion G06F40/129; vowelisation G06F40/232

Abstract

The present invention proposes a solution to the problem of word lexical disambiguation in Arabic texts. This solution is based on text domain-specific knowledge, which facilitates the automatic vowel restoration of modern standard Arabic scripts. Texts similar in their contents, restricted to a specific field or sharing a common knowledge can be grouped in a specific category or in a specific domain (examples of specific domains; sport, art, economic, science . . . ). The present invention discloses a method, system and computer program for lexically disambiguating non diacritized Arabic words in a text based on a learning approach that exploits; Arabic lexical look-up, and Arabic morphological analysis, to train the system on a corpus of diacritized Arabic text pertaining to a specific domain. Thereby, the contextual relationships of the words related to a specific domain are identified, based on the valid assumption that there is less lexical variability in the use of the words and their morphological variants within a domain compared to an unrestricted text.

Claims

1. A method in a specific language, for lexically disambiguating non diacritized words in a text and restoring vowels, said method comprising the steps of:

automatically generating, by a lexicon generation sub-system, a domain specific lexicon based on a diacritized training corpus pertaining to a specific domain, including, for each word in the diacritized training corpus;

removing the diacritics from the word;

obtaining all possible valid vowelization patterns for the word, each vowelization pattern belonging to a different stem;

selecting from the obtained vowelization patterns, the vowelization pattern that matches the vowelization pattern of the word before the diacritics have been removed from the word;

identifying in a generic lexicon, a stem associated with the vowelization pattern that matches vowelization pattern of the word; and

disambiguating, by a lexicon generation sub-system, non diacritized words in a text pertaining to the specific domain and restoring vowels by means of the previously generated domain specific lexicon thereby converting non diacritized words in the text to diacritized words.

2. The method according to claim 1 wherein the step for each word in the diacritized training corpus, of identifying in a generic lexicon, a stem associated with the vowelization pattern that matches the vowelization pattern of the word, comprises the further steps of:

recording how many time the identified stem is encountered in the diacritized training corpus.

3. The method according to claim 2 wherein the step of generating a domain specific lexicon based on a diacritized training corpus pertaining to a specific domain, comprises the further steps of:

keeping in the generic lexicon;

stems that have the highest number of morphological variants identified in the diacritized training corpus, when multiple stems with identical character sequences exist;

stems that have no other identical sequence of characters;

any stem randomly selected among stems that;

are not related to words in the diacritized training corpus, and

have multiple identical character sequences,

deleting from the generic lexicon all other stems and therefore generating a domain specific lexicon from the generic lexicon.

4. The method according to claim 3 wherein the step of disambiguating non diacritized words in a text and restoring vowels, comprises the further steps of:

for each word in the non diacritized text;

segmenting the word into a stem, and affixes if any;

disambiguating the word using the domain specific lexicon, said domain specific lexicon comprising a unique vowelization pattern for each stem;

determining the vowelization pattern of the stem;

diacritizing the word by concatenating the diacritized stem with diacritized affixes if any.

5. The method according to claim 4 wherein the step of generating a domain specific lexicon based on a diacritized training corpus pertaining to a specific domain, comprises the further step of:

determining for each verb encountered in the diacritized training corpus, the most frequent voice.

6. The method according to claim 5 wherein the step, for each word in the diacritized training corpus, of identifying in a generic lexicon, a stem associated with the vowelization pattern that matches the vowelization pattern of the word, comprises the further steps of:

if the identified stem is for a verb;

determining the voice of the verb stem from the word vowelization pattern; and

,

recording how many time the voice of the verb stem is encountered in the diacritized training corpus.

7. The method according to claim 6 wherein the step of disambiguating non diacritized verbs by means, for each verb, of the most frequent voice encountered in the diacritized training corpus, comprises the preliminary step of:

generating a list comprising the verb stems encountered in the diacritized training corpus with for each of the stems, an identification of the most frequent voice.

8. The method according to claim 1 wherein the step of disambiguating non diacritized words in a text pertaining to the specific domain and restoring vowels by means of the previously generated domain specific lexicon, comprises the further step of:

disambiguating non diacritized verbs by means, for each verb, of the most frequent voice encountered in the diacritized training corpus.

9. The method according to claim 8 wherein said specific language is Arabic, said non diacritized text is an Arabic text, said diacritized training corpus is in Arabic.

10. A system comprising means adapted for carrying out the steps of the method according to claim 1.