Apparatus for clustering a plurality of documents

• US 9,342,591 B2
• Filed: 02/14/2013
• Issued: 05/17/2016
• Est. Priority Date: 02/14/2012
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Inagaki, Takeshi
• Primary Examiner(s): Reyes, Mariela
• Assistant Examiner(s): Jacobs, Edward
• Application Number:
  US13/766,889
• Publication Number:
  US 20130212106A1
• Time in Patent Office: 1,188 Days
• Field of Search: 707/738
• US Class Current: 1/1
• CPC Class Codes: G06F 16/355 Class or cluster creation or modification

Abstract

According to an aspect, there are provided an apparatus, a program for causing a computer to function as such an apparatus, and a method, wherein the apparatus includes a selection section for selecting a plurality of sample documents from a plurality of documents and a first parameter generation section for analyzing the plurality of sample documents to generate an initial parameter matrix expressing a probability that each of a plurality of words included in the plurality of sample documents is included in each of a plurality of topics. The apparatus also includes a second parameter generation section for analyzing the plurality of documents by using each value included in the initial parameter matrix as an initial value to generate a parameter matrix expressing a probability that each of a plurality of words included in the plurality of documents is included in each of a plurality of topics.

Claims

1. An apparatus comprising:

a selection section for selecting a plurality of sample documents from a plurality of documents;

a first parameter generation section for analyzing the plurality of sample documents using preset values as initial values to generate an initial parameter matrix expressing a probability that each of a plurality of words included in the plurality of sample documents is included in each of a plurality of topics; and

a second parameter generation section for analyzing the plurality of documents by using each value included in the initial parameter matrix as an initial value to generate a parameter matrix expressing a probability that each of a plurality of words included in the plurality of documents is included in each of a plurality of topics,

wherein the second parameter generation section includes;

a division section for dividing the plurality of documents into a plurality of groups, the division section including;

a pre-clustering section for clustering the plurality of documents into a plurality of clusters based on the initial parameter matrix; and

an allocation section for allocating each of documents in the plurality of clusters to each of the plurality of groups, wherein the allocation section allocates the plurality of documents to the plurality of groups in such a manner that each of the plurality of clusters obtained by clustering the plurality of documents based on the initial parameter matrix is included in equal proportions among groups;

a plurality of element parameter generation sections, each of which is provided to correspond to each of the plurality of groups to analyze a plurality of documents included in a corresponding group by using each value in the initial parameter matrix as an initial value in order to generate an element parameter matrix expressing a probability that each of a plurality of words included in the plurality of documents included in the corresponding group is included in each of a plurality of topics; and

a synthesis section for synthesizing a plurality of element parameter matrices to generate the parameter matrix, wherein the synthesis section;

feeds the calculated parameter matrix back to each of the plurality of element parameter generation sections, each of the plurality of element parameter generation sections uses, as an initial value, each value in the parameter matrix fed back to analyze a plurality of documents included in a corresponding group in order to generate the element parameter matrix again,

synthesizes the plurality of element parameter matrices generated by using, as the initial value, each value in the parameter matrix fed back to generate the parameter matrix again, and

feeds the calculated parameter matrix back to the division section,

wherein the division section divides the plurality of documents into the plurality of groups again in such a manner that each of the plurality of clusters obtained by clustering the plurality of documents based on the parameter matrix fed back is included in equal proportions among groups, and

each of the plurality of element parameter generation sections analyzes a plurality of documents included in a corresponding group divided again to generate the element parameter matrix.

2. The apparatus according to claim 1, wherein the first parameter generation section and the second parameter generation section generate the initial parameter matrix and the parameter matrix by means of Latent Dirichlet Allocation.

3. The apparatus according to claim 2, wherein the second parameter generation section generates the parameter matrix including a topic that is not included in the initial parameter matrix.

4. The apparatus according to claim 3, wherein the second parameter generation section sets an initial value of the topic, which is not included in the initial parameter matrix, the initial value of the topic being smaller than an average of initial values of a plurality of topics included in the initial parameter matrix to analyze the plurality of documents in order to generate the parameter matrix.

5. The apparatus according to claim 4, wherein the second parameter generation section sets an initial value of a word, which is not included in the initial parameter matrix, the initial value of the word being smaller than an average of initial values of a plurality of words included in the initial parameter matrix to analyze the plurality of documents in order to generate the parameter matrix.

6. The apparatus according to claim 1, further comprising

a setting section for setting the number of selected sample documents plural times while increasing the number of selections in a step-by-step manner, wherein

the selection section selects sample documents corresponding in number to a number of selections from the plurality of documents each time the number of selections is set,

the first parameter generation section analyzes the plurality of sample documents selected each time the number of selections is set to generate the initial parameter and, upon setting the number of selections from the second time on, the first parameter generation section analyzes the plurality of sample documents selected by using, as an initial value, each value included in the previously generated initial parameter matrix to generate the initial parameter matrix, and

the second parameter generation section analyzes the plurality of documents by using, as the initial value, each value included in the initial parameter matrix generated by setting the number of selections a predetermined number of times to generate the parameter matrix.

7. The apparatus according to claim 1, further comprising a clustering section for clustering the plurality of documents based on the parameter matrix generated by the second parameter generation section.

8. A method comprising:

selecting a plurality of sample documents from a plurality of documents;

analyzing the plurality of sample documents using preset values as initial values to generate an initial parameter matrix expressing a probability that each of a plurality of words included in the plurality of sample documents is included in each of a plurality of topics;

analyzing the plurality of documents by using each value included in the initial parameter matrix as an initial value to generate a parameter matrix expressing a probability that each of a plurality of words included in the plurality of documents is included in each of a plurality of topics, the analyzing including;

dividing the plurality of documents into a plurality of groups, the dividing including;

clustering the plurality of documents into a plurality of clusters based on the initial parameter matrix; and

allocating each of documents in the plurality of clusters to each of the plurality of groups, wherein the plurality of documents are allocated to the plurality of groups in such a manner that each of the plurality of clusters obtained by clustering the plurality of documents based on the initial parameter matrix is included in equal proportions among groups;

for each of the plurality of groups;

analyzing a plurality of documents included in a corresponding group by using each value in the initial parameter matrix as an initial value in order to generate an element parameter matrix expressing a probability that each of a plurality of words included in the plurality of documents included in the corresponding group is included in each of a plurality of topics; and

synthesizing a plurality of element parameter matrices to generate the parameter matrix, wherein the synthesizing includes;

feeding the calculated parameter matrix back to each of the plurality of element parameter generation sections, each of the plurality of element parameter generation sections uses, as an initial value, each value in the parameter matrix fed back to analyze a plurality of documents included in a corresponding group in order to generate the element parameter matrix again;

synthesizing the plurality of element parameter matrices generated by using, as the initial value, each value in the parameter matrix fed back to generate the parameter matrix again; and

feeding the calculated parameter matrix back to a division section for dividing the plurality of documents into the plurality of groups again in such a manner that each of the plurality of clusters obtained by clustering the plurality of documents based on the parameter matrix fed back is included in equal proportions among groups; and

analyzing a plurality of documents included in a corresponding group divided again to generate the element parameter matrix.

9. A computer program product for clustering a plurality of documents, the computer program product comprising:

a non-transitory storage medium readable by a processing circuit and storing instructions for execution by the processing circuit for performing a method comprising;

selecting a plurality of sample documents from a plurality of documents;

analyzing the plurality of sample documents using preset values as initial values to generate an initial parameter matrix expressing a probability that each of a plurality of words included in the plurality of sample documents is included in each of a plurality of topics; and

analyzing the plurality of documents by using each value included in the initial parameter matrix as an initial value to generate a parameter matrix expressing a probability that each of a plurality of words included in the plurality of documents is included in each of a plurality of topics, the analyzing including;

dividing the plurality of documents into a plurality of groups, the dividing including;

clustering the plurality of documents into a plurality of clusters based on the initial parameter matrix; and

allocating each of documents in the plurality of clusters to each of the plurality of groups, wherein the plurality of documents are allocated to the plurality of groups in such a manner that each of the plurality of clusters obtained by clustering the plurality of documents based on the initial parameter matrix is included in equal proportions among groups;

for each of the plurality of groups;

analyzing a plurality of documents included in a corresponding group by using each value in the initial parameter matrix as an initial value in order to generate an element parameter matrix expressing a probability that each of a plurality of words included in the plurality of documents included in the corresponding group is included in each of a plurality of topics; and

synthesizing a plurality of element parameter matrices to generate the parameter matrix, wherein the synthesizing includes;

feeding the calculated parameter matrix back to each of the plurality of element parameter generation sections, each of the plurality of element parameter generation sections uses, as an initial value, each value in the parameter matrix fed back to analyze a plurality of documents included in a corresponding group in order to generate the element parameter matrix again;

synthesizing the plurality of element parameter matrices generated by using, as the initial value, each value in the parameter matrix fed back to generate the parameter matrix again; and

feeding the calculated parameter matrix back to a division section for dividing the plurality of documents into the plurality of groups again in such a manner that each of the plurality of clusters obtained by clustering the plurality of documents based on the parameter matrix fed back is included in equal proportions among groups; and

analyzing a plurality of documents included in a corresponding group divided again to generate the element parameter matrix.

10. The computer program product according to claim 9, wherein analyzing the plurality of sample documents further comprises generating the initial parameter matrix by means of Latent Dirichlet Allocation.

11. The computer program product according to claim 10, wherein analyzing the plurality of documents further comprises generating the parameter matrix by means of Latent Dirichlet Allocation.