Method and system for natural language dictionary generation

• US 8,849,651 B2
• Filed: 12/21/2012
• Issued: 09/30/2014
• Est. Priority Date: 06/27/2007
• Status: Expired due to Fees

• Current Assignee: ABBYY Production LLC (ABBYY Software)
• Original Assignee: ABBYY InfoPoisk LLC
• Inventors: Selegey, Vladimir, Marachim, Alexey
• Primary Examiner(s): Jackson, Jakieda
• Application Number:
  US13/723,814
• Publication Number:
  US 20130110504A1
• Time in Patent Office: 648 Days
• Field of Search: 704/9
• US Class Current: 704/9
• CPC Class Codes: G06F 40/242 Dictionaries
• CPC Class Codes: G06F 40/268 Morphological analysis
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30

Abstract

A method and computer system for analyzing a text corpus in a natural language is provided. An initial morphological description having word inflection rules for various groups of words in the natural language is created by a linguist. A plurality of text corpuses are analyzed to obtain information on the occurrence of a plurality of word forms for each word token in each text corpus. A morphological dictionary which contains information about each base form and word inflection rules for each word token with verified hypothesis is generated.

Claims

1. A computer system to create a morphological electronic dictionary for a natural language, the computer system comprising:

a processor;

an electronic memory configured with electronic instructions to cause the computer system to perform steps, the electronic instructions including;

identify each word token in the text corpus;

apply paradigm rules to each word token in the text corpus;

generate one or more hypotheses about a part of speech for base forms of each word token;

select other word inflected forms corresponding to the base form of each word token;

verify each hypothesis of the one or more hypotheses for each base form of each word token based on ratings;

add grammatical values and inflection paradigms to each base form of each word token for each verified hypothesis;

obtain information about one or more morphological descriptions for each word token with a verified hypothesis; and

add the base form of each word token with the morphological descriptions to the electronic morphological dictionary of the natural language for each verified hypothesis.

2. The computer system of claim 1, the instructions further including:

creating an initial morphological description having word inflection rules for groups of words in the natural language.

3. The computer system of claim 1, wherein each of the morphological descriptions comprises one or more word inflection rules.

4. The computer system of claim 1, wherein the morphological descriptions comprise one or more word formation rules.

5. The computer system of claim 1, wherein the morphological descriptions comprise a grammatical system of the natural language.

6. The computer system of claim 5, wherein the grammatical system of the natural language comprises a set of grammatical categories and the grammemes thereof.

7. The computer system of claim 1, further comprising:

using additional ratings for one or more hypotheses where verification is initially unsuccessful.

8. The computer system of claim 7, wherein the addition additional ratings are obtained from checking the base form of each word token against a word list of a dictionary.

9. The computer system of claim 7, wherein the addition additional ratings are obtained from checking the base form of each word token with a spelling checker component.

10. The computer system of claim 1, wherein the one or more paradigm rules applied to each word token comprise rules for changing each word token, the rules selected from the group consisting of:

adding an affix, removing an affix, adding an ending, removing an ending, and combinations thereof.

11. The computer system of claim 1, wherein the grammatical values added to the base form of each word token comprise information about a part of speech for each word token.

12. The computer system of claim 1, wherein the inflection paradigms added to the base form of each word token comprise inflection rules for each word token.

13. The computer system of claim 1, wherein the natural language comprises a natural language selected from the group consisting of:

English, French, German, Italian, Russian, Spanish, Ukrainian, Dutch, Danish, Swedish, Finnish, Portuguese, Slovak, Polish, Czech, Hungarian, Lithuanian, Latvian, Estonian, Greek, Bulgarian, Turkish, Tatar, Hindi, Serbian, Croatian, Romanian, Slovenian, Macedonian, Japanese, Korean, Arabic, Hebrew, and Swahili.

14. A computer system to generate a morphological electronic dictionary for a natural language, the computer system comprising:

a processor;

an electronic memory configured with electronic instructions to cause the computer system to perform steps, the electronic instructions including;

create an initial morphological description having word inflection rules for groups of words in the natural language;

analyze by the computer system a plurality of text corpuses in the natural language, including;

identifying each word token in each text corpus of the natural language;

applying one or more paradigm rules to each word token in each text corpus;

generating one or more hypotheses about parts of speech for base forms of each word token;

searching for other word inflected forms corresponding to the base form of each word token;

verifying each hypothesis of the one or more hypotheses for each base form of each word token based on ratings to identify verified hypotheses;

adding grammatical values and inflection paradigms to each base form of each word token for each verified hypothesis; and

obtaining information about one or more morphological descriptions for each word token with a verified hypothesis; and

add the base form of each word token with the morphological descriptions to the morphological electronic dictionary for each verified hypothesis.

15. The computer system of claim 14, wherein the morphological descriptions comprise one or more word inflection rules.

16. The computer system of claim 14, wherein the morphological descriptions comprise one or more word formation rules.

17. The computer system of claim 14, wherein the morphological descriptions comprise a grammatical system of the natural language.

18. The computer system of claim 14, wherein the natural language comprises a natural language selected from the group consisting of:

English, French, German, Italian, Russian, Spanish, Ukrainian, Dutch, Danish, Swedish, Finnish, Portuguese, Slovak, Polish, Czech, Hungarian, Lithuanian, Latvian, Estonian, Greek, Bulgarian, Turkish, Tatar, Hindi, Serbian, Croatian, Romanian, Slovenian, Macedonian, Japanese, Korean, Arabic, Hebrew, and Swahili.