APPARATUS FOR CLUSTERING A PLURALITY OF DOCUMENTS

• US 20130212106A1
• Filed: 02/14/2013
• Published: 08/15/2013
• Est. Priority Date: 02/14/2012
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Inagaki, Takeshi
• Granted Patent: US 9,342,591 B2
• Time in Patent Office: Days
• US Class Current: 707/737
• CPC Class Codes: G06F 16/355 Class or cluster creation or modification

Abstract

According to an aspect, there are provided an apparatus, a program for causing a computer to function as such an apparatus, and a method, wherein the apparatus includes a selection section for selecting a plurality of sample documents from a plurality of documents and a first parameter generation section for analyzing the plurality of sample documents to generate an initial parameter matrix expressing a probability that each of a plurality of words included in the plurality of sample documents is included in each of a plurality of topics. The apparatus also includes a second parameter generation section for analyzing the plurality of documents by using each value included in the initial parameter matrix as an initial value to generate a parameter matrix expressing a probability that each of a plurality of words included in the plurality of documents is included in each of a plurality of topics.

Claims

1. An apparatus comprising:

a selection section for selecting a plurality of sample documents from a plurality of documents;

a first parameter generation section for analyzing the plurality of sample documents to generate an initial parameter matrix expressing a probability that each of a plurality of words included in the plurality of sample documents is included in each of a plurality of topics; and

a second parameter generation section for analyzing the plurality of documents by using each value included in the initial parameter matrix as an initial value to generate a parameter matrix expressing a probability that each of a plurality of words included in the plurality of documents is included in each of a plurality of topics.

2. The apparatus according to claim 1, wherein the first parameter generation section and the second parameter generation section generate the initial parameter matrix and the parameter matrix by means of Latent Dirichlet Allocation.

3. The apparatus according to claim 2, wherein the second parameter generation section includes:

a division section for dividing the plurality of documents into a plurality of groups;

a plurality of element parameter generation sections, each of which is provided to correspond to each of the plurality of groups to analyze a plurality of documents included in a corresponding group by using each value in the initial parameter matrix as an initial value in order to generate an element parameter matrix expressing a probability that each of a plurality of words included in the plurality of documents included in the corresponding group is included in each of a plurality of topics; and

a synthesis section for synthesizing a plurality of element parameter matrices to generate the parameter matrix.

4. The apparatus according to claim 3, wherein the division section includes:

a pre-clustering section for clustering the plurality of documents into a plurality of clusters based on the initial parameter matrix; and

an allocation section for allocating each of documents in the plurality of clusters to each of the plurality of groups.

5. The apparatus according to claim 4, wherein the allocation section allocates the plurality of documents to the plurality of groups in such a manner that each of the plurality of clusters obtained by clustering the plurality of documents based on the initial parameter matrix is included in equal proportions among groups.

6. The apparatus according to claim 3, wherein the synthesis section feeds the calculated parameter matrix back to each of the plurality of element parameter generation sections,

each of the plurality of element parameter generation sections uses, as an initial value, each value in the parameter matrix fed back to analyze a plurality of documents included in a corresponding group in order to generate the element parameter matrix again, and

the synthesis section synthesizes the plurality of element parameter matrices generated by using, as the initial value, each value in the parameter matrix fed back to generate the parameter matrix again.

7. The apparatus according to claim 6, wherein

the synthesis section feeds the calculated parameter matrix back to the division section,

the division section divides the plurality of documents into the plurality of groups again in such a manner that each of the plurality of clusters obtained by clustering the plurality of documents based on the parameter matrix fed back is included in equal proportions among groups, and

each of the plurality of element parameter generation sections analyzes a plurality of documents included in a corresponding group divided again to generate the element parameter matrix.

8. The apparatus according to claim 2, wherein the second parameter generation section generates the parameter matrix including a topic that is not included in the initial parameter matrix.

9. The apparatus according to claim 8, wherein the second parameter generation section sets an initial value of the topic, which is not included in the initial parameter matrix, the initial value of the topic being smaller than an average of initial values of a plurality of topics included in the initial parameter matrix to analyze the plurality of documents in order to generate the parameter matrix.

10. The apparatus according to claim 9, wherein the second parameter generation section sets an initial value of a word, which is not included in the initial parameter matrix, the initial value of the word being smaller than an average of initial values of a plurality of words included in the initial parameter matrix to analyze the plurality of documents in order to generate the parameter matrix.

11. The apparatus according to claim 1, further comprising

a setting section for setting the number of selected sample documents plural times while increasing the number of selections in a step-by-step manner, wherein

the selection section selects sample documents corresponding in number to a number of selections from the plurality of documents each time the number of selections is set,

the first parameter generation section analyzes the plurality of sample documents selected each time the number of selections is set to generate the initial parameter and, upon setting the number of selections from the second time on, the first parameter generation section analyzes the plurality of sample documents selected by using, as an initial value, each value included in the previously generated initial parameter matrix to generate the initial parameter matrix, and

the second parameter generation section analyzes the plurality of documents by using, as the initial value, each value included in the initial parameter matrix generated by setting the number of selections a predetermined number of times to generate the parameter matrix.

12. The apparatus according to claim 1, further comprising a clustering section for clustering the plurality of documents based on the parameter matrix generated by the second parameter generation section.

13. An apparatus for clustering a plurality of documents, comprising:

a selection section for selecting a plurality of sample documents from the plurality of documents;

a parameter generation section for analyzing the plurality of sample documents to generate a parameter matrix expressing a probability that each of a plurality of words included in the plurality of sample documents is included in each of a plurality of topics; and

a clustering section for clustering the plurality of documents into a plurality of clusters based on the parameter matrix.

14. A method comprising:

selecting a plurality of sample documents from a plurality of documents;

analyzing the plurality of sample documents to generate an initial parameter matrix expressing a probability that each of a plurality of words included in the plurality of sample documents is included in each of a plurality of topics; and

analyzing the plurality of documents by using each value included in the initial parameter matrix as an initial value to generate a parameter matrix expressing a probability that each of a plurality of words included in the plurality of documents is included in each of a plurality of topics.

15. A computer program product for clustering a plurality of documents, the computer program product comprising:

a storage medium readable by a processing circuit and storing instructions for execution by the processing circuit for performing a method comprising;

selecting a plurality of sample documents from a plurality of documents;

analyzing the plurality of sample documents to generate an initial parameter matrix expressing a probability that each of a plurality of words included in the plurality of sample documents is included in each of a plurality of topics; and

analyzing the plurality of documents by using each value included in the initial parameter matrix as an initial value to generate a parameter matrix expressing a probability that each of a plurality of words included in the plurality of documents is included in each of a plurality of topics.

16. The computer program product according to claim 15, wherein analyzing the plurality of sample documents further comprises generating the initial parameter matrix by means of Latent Dirichlet Allocation.

17. The computer program product according to claim 16, wherein analyzing the plurality of documents further comprises generating the parameter matrix by means of Latent Dirichlet Allocation.

18. The computer program product according to claim 17, wherein analyzing the plurality of documents further comprises:

dividing the plurality of documents into a plurality of groups, wherein a plurality of element parameter generation sections are provided, each of which corresponds to each of the plurality of groups to analyze a plurality of documents included in a corresponding group by using each value in the initial parameter matrix as an initial value in order to generate an element parameter matrix expressing a probability that each of a plurality of words included in the plurality of documents included in the corresponding group is included in each of a plurality of topics; and

synthesizing a plurality of element parameter matrices to generate the parameter matrix.

19. The computer program product according to claim 18, wherein dividing the plurality of documents further comprises:

clustering the plurality of documents into a plurality of clusters based on the initial parameter matrix; and

allocating each of documents in the plurality of clusters to each of the plurality of groups.

20. The computer program product according to claim 19, wherein allocating each of documents further comprises allocating the plurality of documents to the plurality of groups in such a manner that each of the plurality of clusters obtained by clustering the plurality of documents based on the initial parameter matrix is included in equal proportions among groups.