REFINING TOPIC REPRESENTATIONS

• US 20170124190A1
• Filed: 10/28/2015
• Published: 05/04/2017
• Est. Priority Date: 10/28/2015
• Status: Active Grant

• Current Assignee: Fujitsu Limited
• Original Assignee: Fujitsu Limited
• Inventors: WANG, Jun, UCHINO, Kanji
• Granted Patent: US 10,146,874 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A method to refine topic representations is described. The method may include identifying an original topic representation for a set of electronic documents stored at least temporarily in a non-transitory storage media. The method may include performing, by a processor, an informativeness analysis on the original topic representation. The method may also include performing a topic consistency analysis on the original topic representation. The method may include performing a topic redundancy analysis on the original topic representation in view of a second topic representation. The method may also include generating a refined topic representation based on the informativeness analysis, the consistency analysis and the redundancy analysis.

Claims

1. A method comprising:

identifying, by a processor, an original topic representation for a set of electronic documents stored at least temporarily in a non-transitory storage media;

performing, by the processor, an informativeness analysis on the original topic representation;

performing, by the processor, a topic consistency analysis on the original topic representation;

performing, by the processor, a topic redundancy analysis on the original topic representation in view of a second topic representation; and

generating, by the processor, a refined topic representation based on the informativeness analysis, the consistency analysis, and the redundancy analysis.

2. The method of claim 1, wherein identifying the original topic representation for the set of electronic documents comprises:

crawling the set of electronic documents;

extracting a plurality of knowledge points from the set of electronic documents;

performing a topic model analysis on the knowledge points from the set of electronic documents; and

selecting the original topic representation for the set of electronic documents based on the topic model analysis.

3. The method of claim 2, wherein each extracted knowledge point includes a word or a phrase that exists at least once in the set of electronic documents.

4. The method of claim 1, wherein performing the informativeness analysis on the original topic representation comprises:

identifying a set of knowledge points in the original topic representation;

calculating an informativeness feature of each knowledge point in the set of knowledge points;

calculating a weighted informativeness value of the original topic representation based on the informativeness feature of each knowledge point in the set of knowledge points;

in response to determining that the weighted informativeness value is below a first threshold, removing the original topic representation; and

in response to determining that the weighted informativeness value is above the first threshold, keeping the original topic representation.

5. The method of claim 1, wherein performing the topic consistency analysis on the original topic representation comprises:

identifying a set of knowledge points in the original topic representation;

identifying a semantic vector representation of each knowledge point of the set of knowledge points;

calculating a centroid of the original topic representation;

calculating an average semantic distance to the centroid for the original topic representation;

in response to determining that the average semantic distance to the centroid for the original topic representation is above a second threshold, removing the original topic representation; and

in response to determining that the average semantic distance to the centroid for the original topic representation is below the second threshold, keeping the original topic representation.

6. The method of claim 1, wherein performing a topic redundancy analysis on the original topic representation in view of a second topic representation comprises:

identifying a first set of knowledge points in the original topic representation and a second set of knowledge points in the second topic representation;

identifying a semantic vector representation of each knowledge point of the first set of knowledge points and second set of knowledge points;

calculating a first centroid of the original topic representation and a second centroid of the second topic representation;

calculating an average semantic distance between the first centroid and the second centroid;

in response to determining that the average semantic distance between the first centroid and the second centroid is below a third threshold, merging the original topic representation and the second topic representation; and

in response to determining that the average semantic distance between the first centroid and the second centroid is above the third threshold, keeping the original topic representation and the second topic representation.

7. The method of claim 6, wherein merging the original topic representation and the second topic representation comprises:

(a) assigning the original topic representation or the second topic representation to the resulting merged topic representation, or (b) generating a third topic representation.

8. A non-transitory computer-readable medium having encoded therein programming code executable by a processor to perform or control performance of operations comprising:

identifying, by the processor, an original topic representation for a set of electronic documents stored at least temporarily in a non-transitory storage media;

performing, by the processor, an informativeness analysis on the original topic representation;

performing, by the processor, a topic consistency analysis on the original topic representation;

performing, by the processor, a topic redundancy analysis on the original topic representation in view of a second topic representation; and

generating, by the processor, a refined topic representation based on the informativeness analysis, the consistency analysis, and the redundancy analysis.

9. The non-transitory computer-readable medium of claim 8, wherein identifying the original topic representation for the set of electronic documents comprises:

crawling the set of electronic documents;

extracting a plurality of knowledge points from the set of electronic documents;

performing a topic model analysis on the knowledge points from the set of electronic documents; and

selecting the original topic representation for the set of electronic documents based on the topic model analysis.

10. The non-transitory computer-readable medium of claim 9, wherein each extracted knowledge point includes a word or a phrase that exists at least once in the set of electronic documents.

11. The non-transitory computer-readable medium of claim 8, wherein performing the informativeness analysis on the original topic representation comprises:

identifying a set of knowledge points in the original topic representation;

calculating an informativeness feature of each knowledge point in the set of knowledge points;

calculating a weighted informativeness value of the original topic representation based on the informativeness feature of each knowledge point in the set of knowledge points;

in response to determining that the weighted informativeness value is below a first threshold, removing the original topic representation; and

in response to determining that the weighted informativeness value is above the first threshold, keeping the original topic representation.

12. The non-transitory computer-readable medium of claim 8, wherein performing the topic consistency analysis on the original topic representation comprises:

identifying a set of knowledge points in the original topic representation;

identifying a semantic vector representation of each knowledge point of the set of knowledge points;

calculating a centroid of the original topic representation;

calculating an average semantic distance to the centroid for the original topic representation;

in response to determining that the average semantic distance to the centroid for the original topic representation is above a second threshold, removing the original topic representation; and

in response to determining that the average semantic distance to the centroid for the original topic representation is below the second threshold, keeping the original topic representation.

13. The non-transitory computer-readable medium of claim 8, wherein performing a topic redundancy analysis on the original topic representation in view of a second topic representation comprises:

identifying a first set of knowledge points in the original topic representation and a second set of knowledge points in the second topic representation;

identifying a semantic vector representation of each knowledge point of the first set of knowledge points and second set of knowledge points;

calculating a first centroid of the original topic representation and a second centroid of the second topic representation;

calculating an average semantic distance between the first centroid and the second centroid;

in response to determining that the average semantic distance between the first centroid and the second centroid is below a third threshold, merging the original topic representation and the second topic representation; and

in response to determining that the average semantic distance between the first centroid and the second centroid is above the third threshold, keeping the original topic representation and the second topic representation.

14. The non-transitory computer-readable medium of claim 13, wherein merging the original topic representation and the second topic representation comprises (a) assigning the original topic representation or the second topic representation to the resulting merged topic representation or (b) generating a third topic representation.

15. A system comprising:

a memory; and

a processing device operatively coupled to the memory, the processing device being configured to execute operations comprising;

identify an original topic representation for a set of electronic documents stored at least temporarily in a non-transitory storage media;

perform, by a processor, an informativeness analysis on the original topic representation;

perform a topic consistency analysis on the original topic representation;

perform a topic redundancy analysis on the original topic representation in view of a second topic representation; and

generate a refined topic representation based on the informativeness analysis, the consistency analysis, and the redundancy analysis.

16. The system of claim 15, wherein when identifying the original topic representation for the set of electronic documents, the processing device is configured to:

crawl the set of electronic documents;

extract a plurality of knowledge points from the set of electronic documents;

perform a topic model analysis on the knowledge points from the set of electronic documents; and

select the original topic representation for the set of electronic documents based on the topic model analysis.

17. The system of claim 15, wherein when performing the informativeness analysis on the original topic representation, the processing device is configured to:

identify a set of knowledge points in the original topic representation;

calculate an informativeness feature of each knowledge point in the set of knowledge points;

calculate a weighted informativeness value of the original topic representation based on the informativeness feature of each knowledge point in the set of knowledge points;

in response to a determination that the weighted informativeness value is below a first threshold, remove the original topic representation; and

in response to a determination that the weighted informativeness value is above the first threshold, keep the original topic representation.

18. The system of claim 15, wherein when performing the topic consistency analysis on the original topic representation, the processing device is configured to:

identify a set of knowledge points in the original topic representation;

identify a semantic vector representation of each knowledge point of the set of knowledge points;

calculate a centroid of the original topic representation;

calculate an average semantic distance to the centroid for the original topic representation;

in response to determining that the average semantic distance to the centroid for the original topic representation is above a second threshold, remove the original topic representation; and

in response to determining that the average semantic distance to the centroid for the original topic representation is below the second threshold, keep the original topic representation.

19. The system of claim 15, wherein when performing a topic redundancy analysis on the original topic representation in view of a second topic representation, the processing device is configured to:

identify a first set of knowledge points in the original topic representation and a second set of knowledge points in the second topic representation;

identify a semantic vector representation of each knowledge point of the first set of knowledge points and second set of knowledge points;

calculate a first centroid of the original topic representation and a second centroid of the second topic representation;

calculate an average semantic distance between the first centroid and the second centroid;

in response to determining that the average semantic distance between the first centroid and the second centroid is below a third threshold, merge the original topic representation and the second topic representation; and

in response to determining that the average semantic distance between the first centroid and the second centroid is above the third threshold, keep the original topic representation and the second topic representation.

20. The system of claim 15, wherein when merging the original topic representation and the second topic representation, the processing device is configured to (a) assign the original topic representation or the second topic representation to the resulting merged topic representation, or (b) generate a third topic representation.