Mining Multilingual Topics

• US 20110258229A1
• Filed: 04/15/2010
• Published: 10/20/2011
• Est. Priority Date: 04/15/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Hu, Jian, Chen, Zheng, Ni, Xiaochuan, Sun, Jian-Tao
• Granted Patent: US 8,825,648 B2
• Time in Patent Office: Days
• US Class Current: 707/776
• CPC Class Codes: G06F 16/24 Querying
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/36 Creation of semantic tools, e.g. ontology or thesauri
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates

Abstract

Techniques for utilizing data mining technology to extract universal topics with multilingual representations from a multilingual database, and to organize existing or new documents in different languages by analyzing their respective topic distributions.

Claims

1. A method comprising:

identifying multiple concept-units from a multi-language document corpus, a particular concept-unit including a set of documents in different languages describing a particular concept;

modeling the concept-units of the multi-language document corpus to create a generative model, wherein the generative model represents at least;

(a) a plurality of universal topics, each of the universal topics being defined by a plurality of topic word distributions corresponding respectively to the different languages;

(b) a topic distribution for each concept-unit, wherein the documents of any single concept-unit are constrained within the generative model to share a common topic distribution;

inferring the plurality of universal topics from the documents of the concept-units based on the generative model.

2. A method as recited in claim 1, wherein the inferring comprises performing a latent Dirichlet allocation analysis.

3. A method as recited in claim 1, wherein the generative model further represents:

(c) the concept-units;

(d) the documents of the concept-units; and

(e) word distributions of the documents.

4. A method as recited in claim 1, wherein identifying multiple concept-units comprises identifying hyperlinks within the documents that identify other documents in other languages relating to common concepts.

5. A method as recited in claim 1, further comprising comparing a new document in a given language to the topic word distributions corresponding to the given language to estimate a topic distribution of the new document.

6. A method as recited in claim 1, further comprising comparing new documents of different languages to the topic distributions to identify one or more groups of the new documents sharing common topics.

7. A method as recited in claim 1, further comprising:

comparing documents of an unclassified document corpus and documents of a classified document corpus to the topic distributions to identify topics of the documents;

classifying documents of the unclassified document corpus according to classifications of documents in the classified document corpus having common topics.

8. A method as recited in claim 1, further comprising:

comparing a reference document of a first language and a plurality of documents of a second language to the topic distributions to identify topics of the documents;

recommending documents of the second language that are related to the reference document based on their identified topics.

20. A method as recited in claim 1, wherein identifying the multiple concept-units comprises identifying hyperlinks within the documents that identify other documents in other languages relating to common concepts.

9. A method comprising:

identifying multiple concept-units from a multi-language document corpus, a particular concept-unit including a set of documents in different languages describing a particular concept; and

inferring a plurality of universal topics from the multiple concept-units, each of the universal topics being defined by a plurality of topic word distributions corresponding respectively to the different languages.

10. A method as recited in claim 9, wherein the inferring comprises performing a latent Dirichlet allocation analysis.

11. A method as recited in claim 9, wherein the inferring comprises performing a probabilistic latent semantic analysis.

12. A method as recited in claim 9, wherein the inferring comprises performing a latent Dirichlet allocation analysis while constraining the documents within a single concept-unit to share a common topic distribution.

13. A method as recited in claim 9, wherein identifying multiple concept-units comprises identifying hyperlinks within the documents that identify other documents in other languages relating to common concepts.

14. A method as described in claim 9, further comprising comparing a new document in a given language to the topic word distributions corresponding to the given language to estimate a topic distribution of the new document.

15. A method as recited in claim 9, further comprising comparing new documents of different languages to the topic distributions to identify one or more groups of the new documents sharing common topics.

16. A method as recited in claim 9, further comprising:

comparing documents of an unclassified document corpus and documents of a classified document corpus to the topic distributions to identify topics of the documents; and

classifying documents of the unclassified document corpus according to classifications of documents in the classified document corpus having common topics.

17. A method as recited in claim 9, further comprising:

comparing a reference document of a first language and a plurality of documents of a second language to the topic distributions to identify topics of the documents; and

recommending documents of the second language that are related to the reference document based on their identified topics.

18. A method comprising:

identifying multiple concept-units from a multi-language document corpus, a particular concept-unit including a set of documents in different languages describing a particular concept; and

deriving a universal topic space from the concept-units; and

analyzing new documents of different languages to place them within the universal topic space.

19. A method as recited in claim 18, wherein deriving the universal topic space comprises inferring a plurality of universal topics from the multiple concept-units, each of the universal topics being defined by a plurality of topic word distributions corresponding respectively to the different languages.