Modeling Topics Using Statistical Distributions

• US 20090094233A1
• Filed: 10/01/2008
• Published: 04/09/2009
• Est. Priority Date: 10/05/2007
• Status: Active Grant

• Current Assignee: Fujitsu Limited
• Original Assignee: Fujitsu Limited
• Inventors: Adler, B. Thomas, Labrou, Yannis, Marvit, David L., Stergiou, Stergios, Jain, Jawahar, Gilman, Alex, Sidorowich, John J.
• Granted Patent: US 9,317,593 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06F 16/355 Class or cluster creation or modification

Abstract

In one embodiment, modeling topics includes accessing a corpus comprising documents that include words. Words of a document are selected as keywords of the document. The documents are clustered according to the keywords to yield clusters, where each cluster corresponds to a topic. A statistical distribution is generated for a cluster from words of the documents of the cluster. A topic is modeled using the statistical distribution generated for the cluster corresponding to the topic.

Claims

1. A method comprising:

accessing a corpus stored in one or more tangible media, the corpus comprising a plurality of documents, a document comprising a plurality of words;

selecting one or more words of each document as one or more keywords of the each document;

clustering the documents according to the keywords to yield a plurality of clusters, each cluster corresponding to a topic;

generating a statistical distribution for each cluster from a subset of the words of the documents of the each cluster to yield a plurality of statistical distributions; and

modeling each topic using the statistical distribution generated for the cluster corresponding to the each topic.

2. The method of claim 1, the selecting the one or more words of the each document further comprising:

ranking the words of the each document according to a ranking technique; and

selecting one or more highly ranked words as the one or more keywords.

3. The method of claim 1, the clustering the documents according to the keywords to yield the plurality of clusters further comprising:

removing one or more clusters that fail to satisfy a size threshold.

4. The method of claim 1, the generating the statistical distribution for the each cluster further comprising:

calculating a term frequency of each word of the subset of the words to yield a plurality of term frequencies; and

generating a term distribution from the term frequencies.

5. The method of claim 1, the generating the statistical distribution for the each cluster further comprising:

calculating a number of documents that include each word of the subset of the words; and

generating a term distribution from the numbers of documents.

6. The method of claim 1, the generating the statistical distribution for the each cluster further comprising:

determining a co-occurrence value indicating a co-occurrence of the topic of the each cluster with the topics of the other clusters; and

generating a co-occurrence distribution from the co-occurrence values.

7. The method of claim 1, further comprising:

identifying at least two clusters with similar statistical distributions; and

consolidating the at least two clusters.

8. The method of claim 1, further comprising:

organizing the clusters according to the statistical distributions; and

assigning the topics of the organized clusters to the documents in the organized clusters.

19. The method of claim 1, further comprising:

selecting the subset of the words according to a term frequency-inverse document frequency (TF-IDF) analysis.

20. The method of claim 1, the consolidating the statistical distributions to yield the consolidated statistical distribution further comprising:

adding the statistical distributions to yield the consolidated statistical distribution.

21. The method of claim 1, further comprising:

selecting a topic for the document according to the consolidated statistical distribution.

9. One or more computer-readable tangible media encoding software operable when executed to:

access a corpus stored in one or more tangible media, the corpus comprising a plurality of documents, a document comprising a plurality of words;

select one or more words of each document as one or more keywords of the each document;

cluster the documents according to the keywords to yield a plurality of clusters, each cluster corresponding to a topic;

generate a statistical distribution for each cluster from a subset of the words of the documents of the each cluster to yield a plurality of statistical distributions; and

model each topic using the statistical distribution generated for the cluster corresponding to the each topic.

10. The computer-readable tangible media of claim 9, further operable to select the one or more words of the each document by:

ranking the words of the each document according to a ranking technique; and

selecting one or more highly ranked words as the one or more keywords.

11. The computer-readable tangible media of claim 9, further operable to cluster the documents according to the keywords to yield the plurality of clusters by:

removing one or more clusters that fail to satisfy a size threshold.

12. The computer-readable tangible media of claim 9, further operable to generate the statistical distribution for the each cluster by:

calculating a term frequency of each word of the subset of the words to yield a plurality of term frequencies; and

generating a term distribution from the term frequencies.

13. The computer-readable tangible media of claim 9, further operable to generate the statistical distribution for the each cluster by:

calculating a number of documents that include each word of the subset of the words; and

generating a term distribution from the numbers of documents.

14. The computer-readable tangible media of claim 9, further operable to generate the statistical distribution for the each cluster by:

determining a co-occurrence value indicating a co-occurrence of the topic of the each cluster with the topics of the other clusters; and

generating a co-occurrence distribution from the co-occurrence values.

15. The computer-readable tangible media of claim 9, further operable to:

identify at least two clusters with similar statistical distributions; and

consolidate the at least two clusters.

16. The computer-readable tangible media of claim 9, further operable to:

organize the clusters according to the statistical distributions; and

assign the topics of the organized clusters to the documents in the organized clusters.

17. A system comprising:

means for accessing a corpus stored in one or more tangible media, the corpus comprising a plurality of documents, a document comprising a plurality of words;

means for selecting one or more words of each document as one or more keywords of the each document;

means for clustering the documents according to the keywords to yield a plurality of clusters, each cluster corresponding to a topic;

means for generating a statistical distribution for each cluster from a subset of the words of the documents of the each cluster to yield a plurality of statistical distributions; and

means for modeling each topic using the statistical distribution generated for the cluster corresponding to the each topic.

18. A method comprising:

accessing a document and a plurality of clusters stored in one or more tangible media, the document comprising a plurality of words, the clusters associated with a plurality of topics;

performing the following for each word of a subset of the words to yield a plurality of statistical distributions;

establishing a frequency of the each word in each cluster of the clusters to yield a plurality of frequencies; and

generating a statistical distribution from the frequencies; and

consolidating the statistical distributions to yield a consolidated statistical distribution, the consolidated statistical distribution indicating the frequencies of the subset of words in the topics.

22. One or more computer-readable tangible media encoding software operable when executed to:

access a document and a plurality of clusters stored in one or more tangible media, the document comprising a plurality of words, the clusters associated with a plurality of topics;

perform the following for each word of a subset of the words to yield a plurality of statistical distributions;

establishing a frequency of the each word in each cluster of the clusters to yield a plurality of frequencies; and

generating a statistical distribution from the frequencies; and

consolidate the statistical distributions to yield a consolidated statistical distribution, the consolidated statistical distribution indicating the frequencies of the subset of words in the topics.

23. The computer-readable tangible media of claim 22, further operable to:

select the subset of the words according to a term frequency-inverse document frequency (TF-IDF) analysis.

24. The computer-readable tangible media of claim 22, further operable to consolidate the statistical distributions to yield the consolidated statistical distribution by:

adding the statistical distributions to yield the consolidated statistical distribution.

25. The computer-readable tangible media of claim 22, further operable to:

select a topic for the document according to the consolidated statistical distribution.

26. A system comprising:

means for accessing a document and a plurality of clusters stored in one or more tangible media, the document comprising a plurality of words, the clusters associated with a plurality of topics;

means for performing the following for each word of a subset of the words to yield a plurality of statistical distributions;

establishing a frequency of the each word in each cluster of the clusters to yield a plurality of frequencies; and

generating a statistical distribution from the frequencies; and

means for consolidating the statistical distributions to yield a consolidated statistical distribution, the consolidated statistical distribution indicating the frequencies of the subset of words in the topics.