CROSS-LANGUAGE TEXT CLASSIFICATION

• US 20120271627A1
• Filed: 06/28/2012
• Published: 10/25/2012
• Est. Priority Date: 10/10/2006
• Status: Active Grant

• Current Assignee: ABBYY Development LLC
• Original Assignee: ABBYY Software
• Inventors: Danielyan, Tatiana, Zuev, Konstantin, Anisimovich, Konstantin, Selegey, Vladimir
• Granted Patent: US 9,588,958 B2
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 40/211 Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
• CPC Class Codes: G06F 40/237 Lexical tools
• CPC Class Codes: G06F 40/268 Morphological analysis
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/55 Rule-based translation

Abstract

Methods are described for performing classification (categorization) of text documents written in various languages. Language-independent semantic structures are constructed before classifying documents. These structures reflect lexical, morphological, syntactic, and semantic properties of documents. The methods suggested are able to perform cross-language text classification which is based on document properties reflecting their meaning. The methods are applicable to genre classification, topic detection, news analysis, authorship analysis, etc.

Claims

1. A method for a computer to analyze, across languages, a text written in one or more natural languages, the method comprising:

performing an analysis of a sentence of the text, wherein the analysis includes performing steps including;

performing lexical-morphological analysis of the sentence of the text;

performing a syntactical analysis of the sentence of the text; and

performing a semantic analysis of the sentence of the text;

generating a set of features, where at least one feature is based on the results of the said analysis; and

performing a text classification based on said set of features, wherein the text classification includes assigning the text to one or more categories.

2. The method of claim 1, wherein the analysis of the sentence of the text includes:

performing a lexical-morphological analysis of the sentence of the text in the source language;

performing a first syntactic analysis of the sentence of the text, the first syntactic analysis comprising;

generating possible constituents for each element of the sentence; and

generating a graph of the generalized constituents to describe possible syntactic relations in sentence of the text;

performing a second syntactic analysis of the sentence of the text on the graph of the generalized constituents, the second syntactic analysis including;

generating one or more syntactic trees for the sentence of the text;

generating one or more hypotheses about related to the overall syntactic structures of the sentence of the text;

verifying the one or more hypotheses related to the overall syntactic structures of the sentence of the text; and

generating a syntactic structure for the sentence of the text; and

generating a language-independent semantic structure for the sentence of the text based upon the selected syntactic structure for the sentence of the text.

3. The method of claim 1, wherein analysis of the sentence of the text further includes generating a statistic for at least one grammatical feature of the sentence of the text.

4. The method of claim 1, wherein analysis of the sentence of the text further includes generating a statistic for at least one lexical feature of the sentence of the text.

5. The method of claim 1, wherein analysis of the sentence of the text further includes generating a statistic for at least one syntactic feature of the sentence of the text.

6. The method of claim 1, wherein analysis of the sentence on the text further includes generating a statistic for at least one semantic feature of the sentence of the text.

7. The method of claim 1, wherein analysis of the sentence on the text further includes generating a statistic for at least one syntactic structure of the sentence of the text.

8. The method of claim 1, wherein analysis of the sentence of the text further includes generating a statistic for at least one language independent semantic structure of the sentence of the text.

9. The method of claim 1, wherein analysis of the sentence on the text further includes generating a statistic for at least one semantic class of a semantic hierarchy related to the sentence of the text.

10. The method of claim 1, wherein a set of features for each text includes generating a statistic of at least one extracted feature.

11. The method of claim 1, wherein performing the text classification includes use of rules for estimating values of a feature statistic.

12. The method of claim 1, wherein the category or categories, to which the sentence of the text is assigned, are language independent categories.

13. The method of claim 1, wherein the text classification includes the analysis of training data.

14. A non-transitory computer readable medium comprising instructions for causing a computing system to carry out steps comprising:

performing a feature extraction on the text, wherein the feature extraction includes performing steps including;

defining a lexical-morphological feature of a sentence of the text;

defining a syntactical feature of the sentence of the text;

defining a semantic feature of the sentence of the text; and

generating a set of features for the sentence of the text related to the lexical-morphological feature, the syntactical feature and the semantic feature;

performing a text classification based on said set of features, wherein the text classification includes assigning the text to one or more categories.

15. The non-transitory computer readable medium of claim 14, wherein the analysis of the sentence of the text includes:

performing a lexical-morphological analysis of the sentence of the text in the source language;

performing a first syntactic analysis of the sentence of the text, the first syntactic analysis comprising;

generating possible constituents for each element of the sentence; and

generating a graph of the generalized constituents to describe possible syntactic relations in sentence of the text;

performing a second syntactic analysis of the sentence of the text on the graph of the generalized constituents, the second syntactic analysis including;

generating one or more syntactic trees for the sentence of the text;

generating one or more hypotheses about related to the overall syntactic structures of the sentence of the text;

verifying the one or more hypotheses related to the overall syntactic structures of the sentence of the text; and

generating a syntactic structure for the sentence of the text; and

generating a language-independent semantic structure for the sentence of the text based upon the selected syntactic structure for the sentence of the text.

16. The non-transitory computer readable medium of claim 14, wherein feature extraction further includes generating a statistic for at least one grammatical feature of the sentence of the text.

17. The non-transitory computer readable medium of claim 14, wherein feature extraction further includes generating a statistic for at least one lexical feature of the sentence of the text.

18. The non-transitory computer readable medium of claim 14, wherein feature extraction further includes generating a statistic for at least one syntactic feature of the sentence of the text.

19. The non-transitory computer readable medium of claim 14, wherein feature extraction further includes generating a statistic for at least one semantic feature of the sentence of the text.

20. The non-transitory computer readable medium of claim 14, wherein feature extraction further includes generating a statistic for at least one syntactic structure of the sentence of the text.

21. The non-transitory computer readable medium of claim 14, wherein feature extraction further includes generating a statistic for at least one language independent semantic structure of the sentence of the text.

22. The non-transitory computer readable medium of claim 14, wherein feature extraction further includes generating a statistic for at least one semantic class.

23. The non-transitory computer readable medium of claim 14, wherein a set of features for each text includes generating a statistic of at least one extracted feature.

24. The non-transitory computer readable medium of claim 14, wherein performing the text classification includes use of rules for estimating values of a feature statistic.

25. The non-transitory computer readable medium of claim 14, wherein the category or categories, to which the sentence of the text is assigned, are language independent categories.

26. A computer system adapted to assign to a language-independent category a source sentence in a source language, the computer system comprising:

a feature extractor adapted to perform steps including;

defining a lexical-morphological feature of the sentence of the text;

defining a syntactical feature of the sentence of the text;

defining a semantic feature of the sentence of the text; and

generating a set of features for the sentence of the text related to the lexical-morphological feature, the syntactical feature and the semantical feature; and

a text classifier adapted to perform steps including;

classify text based on said set of features, wherein the text classification includes assigning the text to one or more categories.

27. The computer system of claim 26, wherein the computer system further comprises:

a first syntactic analyzer adapted to perform steps including;

performing a lexical-morphological analysis of the sentence of the text in the source language;

performing a first syntactic analysis of the sentence of the text, the first syntactic analysis comprising;

generating possible constituents for each element of the sentence; and

generating a graph of the generalized constituents to describe possible syntactic relations in the sentence of the text;

a second syntactic analyzer adapted to perform steps including;

generating one or more syntactic trees for the sentence of the text based on the graph of the generalized constituents;

generating one or more hypotheses about related to the overall syntactic structures of the sentence of the text;

verifying the one or more hypotheses related to the overall syntactic structures of the sentence of the text; and

generating a syntactic structure for the sentence of the text; and

a semantic structure generator adapted to perform steps including;

generating a language-independent semantic structure for the sentence of the text based upon the selected syntactic structure for the sentence of the text.

28. The computer system of claim 26, wherein the feature extractor is further adapted to perform steps including:

generating a statistic for at least one grammatical feature of the sentence of the text.

29. The computer system of claim 26, wherein the feature extractor is further adapted to perform steps including:

generating a statistic for at least one lexical feature of the sentence of the text.

30. The computer system of claim 26, wherein the feature extractor is further adapted to perform steps including:

generating a statistic for at least one syntactic feature of the sentence of the text.

31. The computer system of claim 26, wherein the feature extractor is further adapted to perform steps including:

generating a statistic for at least one semantic feature of the sentence of the text.

32. The computer system of claim 26, wherein the feature extractor is further adapted to perform steps including:

generating a statistic for at least one syntactic structure of the sentence of the text.

33. The computer system of claim 26, wherein the semantic structure generator is further adapted to perform steps including:

generating a statistic for at least one language independent semantic structure of the sentence of the text.

34. The computer system of claim 26, wherein the feature extractor is further adapted to perform steps including:

generating a statistic for at least one semantic class.

35. The computer system of claim 26, wherein the feature extractor is further adapted to perform steps including:

generating a statistic of at least one extracted feature.

36. The computer system of claim 26, wherein the text classifier is adapted to use rules for estimating values of a feature statistic.

37. The computer system of claim 26, wherein the category or categories, to which the sentence of the text is assigned, are language independent categories.