METHOD AND SYSTEM FOR NATURAL LANGUAGE DICTIONARY GENERATION

• US 20090006078A1
• Filed: 06/27/2007
• Published: 01/01/2009
• Est. Priority Date: 06/27/2007
• Status: Active Grant

• Current Assignee: ABBYY Development LLC
• Original Assignee: ABBYY InfoPoisk LLC
• Inventors: Maramchin, Alexey, Selegey, Vladimir
• Granted Patent: US 8,812,296 B2
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06F 40/242 Dictionaries
• CPC Class Codes: G06F 40/268 Morphological analysis
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30

Abstract

A method and computer system for analyzing a text corpus in a natural language is provided. An initial morphological description having word inflection rules for various groups of words in the natural language is created by a linguist. A plurality of text corpuses are analyzed to obtain information on the occurrence of a plurality of word forms for each word token in each text corpus. A morphological dictionary which contains information about each base form and word inflection rules for each word token with verified hypothesis is generated.

Claims

1. A method of analyzing a text corpus in a natural language, comprising:

identifying each word token in the text corpus;

applying one or more paradigm rules to each word token in the text corpus;

generating one or more hypotheses for base forms of each word token;

searching for other word inflected forms corresponding to the base form of each word token;

verifying each hypothesis of the one or more hypotheses for each base form of each word token to identify verified hypothesis;

adding grammatical values and inflection paradigms to each base form of each word token for each verified hypothesis; and

obtaining information on its morphological descriptions for each word token with verified hypothesis.

2. The method of claim 1, further comprising creating by a linguist an initial morphological description having word inflection rules for groups of words in the natural language.

3. The method of claim 1, further comprising adding the base form of each word token with the morphological descriptions to a morphological dictionary of the natural language for each verified hypothesis.

4. The method of claim 1, wherein the morphological descriptions comprises one or more word inflection rules.

5. The method of claim 1, wherein the morphological descriptions comprises one or more word formation rules.

6. The method of claim 1, wherein the morphological descriptions comprises a grammatical system of the natural language.

7. The method of claim 6, wherein the grammatical system of the natural language comprises a set of grammatical categories and the grammemes thereof.

8. The method of claim 1, further comprising:

using additional ratings for one or more hypotheses where verification is initially unsuccessful.

9. The method of claim 8, wherein the addition ratings are obtained from checking the base form of each word token against a word list of a dictionary.

10. The method of claim 8, wherein the addition ratings are obtained from checking the base form of each word token with a spelling checker system.

11. The method of claim 1, wherein the one more paradigm rules applied to each word token comprises rules for changing each word token, the rules are selected from the group consisting of adding an affix, removing an affix, adding an ending, removing an ending, and combinations thereof.

12. The method of claim 1, wherein the grammatical values added to the base form of each word token comprise information about part of speech for each word token.

13. The method of claim 1, wherein the inflection paradigms added to the base form of each word token comprises inflection rules for each word token.

14. The method of claim 1, wherein the natural language comprises a natural language selected from the group consisting of English, French, German, Italian, Russian, Spanish, Ukrainian, Dutch, Danish, Swedish, Finnish, Portuguese, Slovak, Polish, Czech, Hungarian, Lithuanian, Latvian, Estonian, Greek, Bulgarian, Turkish, Tatar, Hindi, Serbian, Croatian, Romanian, Slovenian, Macedonian, Japanese, Korean, Arabic, Hebrew, and Swahili.

15. A method of generating a morphological dictionary for a natural language, comprising:

creating an initial morphological description having word inflection rules for groups of words in the natural language;

analyzing a plurality of text corpuses in the natural language, including;

identifying each word token in each text corpus of the natural language;

applying one or more paradigm rules to each word token in each text corpus;

generating one or more hypotheses for base forms of each word token;

searching for other word inflected forms corresponding to the base form of each word token;

verifying each hypothesis of the one or more hypotheses for each base form of each word token to identify verified hypothesis;

adding grammatical values and inflection paradigms to each base form of each word token for each verified hypothesis; and

obtaining information on its morphological descriptions for each word token with verified hypothesis; and

adding the base form of each word token with the morphological descriptions to the morphological dictionary for each verified hypothesis.

16. The method of claim 15, wherein the morphological descriptions comprises one or more word inflection rules.

17. The method of claim 15, wherein the morphological descriptions comprises one or more word formation rules.

18. The method of claim 15, wherein the morphological descriptions comprises a grammatical system of the natural language.

19. The method of claim 15, wherein the natural language comprises a natural language selected from the group consisting of English, French, German, Italian, Russian, Spanish, Ukrainian, Dutch, Danish, Swedish, Finnish, Portuguese, Slovak, Polish, Czech, Hungarian, Lithuanian, Latvian, Estonian, Greek, Bulgarian, Turkish, Tatar, Hindi, Serbian, Croatian, Romanian, Slovenian, Macedonian, Japanese, Korean, Arabic, Hebrew, and Swahili.

20. A computer readable medium comprising instructions for causing a computing system to carry out steps for analyzing a text corpus in a natural language, the steps comprising:

identifying each word token in the text corpus;

applying one or more paradigm rules to each word token in the text corpus;

generating one or more hypotheses for base forms of each word token;

searching for other word inflected forms corresponding to each base form of each word token;

verifying each hypothesis of the one or more hypotheses for each base form of each word token to identify verified hypothesis;

adding grammatical values and inflection paradigms to each base form of each word token for each verified hypothesis; and

obtaining information on its morphological descriptions for each word token with verified hypothesis.

21. The computer readable medium of claim 20, wherein the natural language comprises a natural language selected from the group consisting of English, French, German, Italian, Russian, Spanish, Ukrainian, Dutch, Danish, Swedish, Finnish, Portuguese, Slovak, Polish, Czech, Hungarian, Lithuanian, Latvian, Estonian, Greek, Bulgarian, Turkish, Tatar, Hindi, Serbian, Croatian, Romanian, Slovenian, Macedonian, Japanese, Korean, Arabic, Hebrew, and Swahili.

22. A computer readable medium comprising instructions for causing a computing system to carry out steps for of generating a morphological dictionary for a natural language, the steps comprising:

analyzing a plurality of text corpuses in the natural language, including;

identifying each word token in each text corpus of the natural language;

applying one or more paradigm rules to each word token in each text corpus;

generating one or more hypotheses for base forms of each word token;

searching for other word inflected forms corresponding to each base form of the word token;

verifying each hypothesis of the one or more hypotheses for each base form of each word token to identify verified hypothesis;

adding grammatical values and inflection paradigms to each base form of each word token for each verified hypothesis; and

obtaining information on its morphological descriptions for each word token with verified hypothesis; and

adding the base form of each word token with the morphological descriptions to the morphological dictionary for each verified hypothesis.

23. The computer readable medium of claim 22, wherein the natural language comprises a natural language selected from the group consisting of English, French, German, Italian, Russian, Spanish, Ukrainian, Dutch, Danish, Swedish, Finnish, Portuguese, Slovak, Polish, Czech, Hungarian, Lithuanian, Latvian, Estonian, Greek, Bulgarian, Turkish, Tatar, Hindi, Serbian, Croatian, Romanian, Slovenian, Macedonian, Japanese, Korean, Arabic, Hebrew, and Swahili.