CROSS-LANGUAGE TEXT CLUSTERING

• US 20130041652A1
• Filed: 10/10/2012
• Published: 02/14/2013
• Est. Priority Date: 10/10/2006
• Status: Active Grant

• Current Assignee: ABBYY Production LLC (ABBYY Software)
• Original Assignee: ABBYY InfoPoisk LLC
• Inventors: Zuev, Konstantin, Danielyan, Tatiana
• Granted Patent: US 9,495,358 B2
• Time in Patent Office: Days
• US Class Current: 704/8
• CPC Class Codes: G06F 40/211 Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
• CPC Class Codes: G06F 40/268 Morphological analysis
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/55 Rule-based translation

Abstract

Methods are described for performing clustering or classification of texts of different languages. Language-independent semantic structures (LISS) are constructed before clustering is performed. These structures reflect lexical, morphological, syntactic, and semantic properties of texts. The methods suggested are able to perform cross-language text clustering which is based on the meaning derived from texts. The methods are applicable to genre classification, topic detection, news analysis, authorship analysis, internet searches, and creating corpora for other tasks, etc.

Claims

1. A method for a computing device to analyze, across languages, a set of texts in one or more natural languages, the method comprising for each text:

electronically analyzing the text, wherein the analysis includes performing steps including;

performing a syntactic analysis of at least one sentence of the text; and

creating a language-independent semantic structure (LISS) by performing a semantic analysis of the sentence of the text;

generating a set of features for the text, where at least one feature is based on the results of the said analysis; and

performing text clustering based on said set of features, wherein the text clustering includes assigning the text to one or more clusters.

2. The method of claim 1, wherein said analyzing comprises resolving lexical ambiguities.

3. The method of claim 1, wherein said analyzing comprises resolving anaphoras.

4. The method of claim 1, wherein said set of features includes lexical features.

5. The method of claim 1, wherein said set of features includes syntactic features.

6. The method of claim 1, wherein said set of features includes grammatical features.

7. The method of claim 1, wherein said set of features includes semantic features.

8. The method of claim 1, wherein said set of features includes semantic classes.

9. The method of claim 1, wherein the method further includes creating at least one index.

10. The method of claim 1 further includes creating at least one index of semantic features.

11. The method of claim 1, wherein said clustering uses a similarity measure, wherein said similarity measure is based on a result of said semantic analysis.

12. The method of claim 11, wherein said similarity measure depends on distances between semantic classes in a semantic hierarchy.

13. The method of claim 12, wherein said similarity measure depends on a frequency of words related to a common ancestor of said semantic classes in said semantic hierarchy.

14. The method of claim 1, wherein analyzing the sentence of the text further includes generating a statistic for at least one grammatical feature of the sentence of the text.

15. The method of claim 1, wherein analyzing the sentence of the text further includes generating a statistic for at least one lexical feature of the sentence of the text.

16. The method of claim 1, wherein analyzing the sentence of the text further includes generating a statistic for at least one syntactic feature of the sentence of the text.

17. The method of claim 1, wherein analyzing the sentence of the text further includes generating a statistic for at least one semantic feature of the sentence of the text.

18. The method of claim 1, wherein analyzing the sentence of the text further includes generating a statistic for at least one language independent semantic structure (LISS) of the sentence of the text.

19. The method of claim 1, wherein analyzing the sentence of the text further includes generating a statistic for at least one semantic class of a semantic hierarchy related to the sentence of the text.

20. The method of claim 1, wherein the set of features for each text includes generating a statistic of at least one extracted feature.

21. The method of claim 1, wherein the method further comprises making one or more of the clusters accessible to another computing device.

22. The method of claim 21, wherein clusters are located across a plurality of computing devices.

23. The method of claim 1, wherein members of the set of texts are located on a plurality of computing devices, wherein the plurality of computing devices are accessible through one or more network protocols.

24. The method of claim 19, wherein clusters comprise texts of different languages.

25. The method of claim 19, wherein clusters comprise texts of mixed languages.