IDENTIFYING CATEGORIES WITHIN TEXTUAL DATA

• US 20160232226A1
• Filed: 02/05/2016
• Published: 08/11/2016
• Est. Priority Date: 02/06/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: McManis, Charles E. Jr., Smith, Douglas A.
• Granted Patent: US 10,157,178 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/353 into predefined classes
• CPC Class Codes: G06F 16/381 using identifiers, e.g. barcodes, RFIDs for URLs G06F16/9554
• CPC Class Codes: G06F 40/216 using statistical methods
• CPC Class Codes: G06F 40/263 Language identification
• CPC Class Codes: G06F 40/30 Semantic analysis

Abstract

A computer-implemented method according to one embodiment includes identifying a plurality of documents associated with a predetermined subject, where each of the plurality of documents contains textual data, analyzing the textual data of each of the plurality of documents to identify one or more categories within the plurality of the documents, and returning the one or more categories identified within the plurality of the documents.

Claims

1. A computer-implemented method, comprising:

identifying a plurality of documents associated with a predetermined subject, where each of the plurality of documents contains textual data;

analyzing the textual data of each of the plurality of documents to identify one or more categories within the plurality of the documents; and

returning the one or more categories identified within the plurality of the documents.

2. The computer-implemented method of claim 1, wherein the plurality of documents include one or more of web page content and scanned document content.

3. The computer-implemented method of claim 1, wherein the plurality of documents is included within a grouping that includes identifiers of a location of each of the plurality of documents.

4. The computer-implemented method of claim 1, wherein analyzing the textual data for each of the plurality of documents includes performing automatic language detection on the textual data.

5. The computer-implemented method of claim 1, wherein analyzing the textual data for each of the plurality of documents includes refining the textual data by removing one or more words from the textual data.

6. The computer-implemented method of claim 1, wherein analyzing the textual data for each of the plurality of documents includes removing any duplicate documents within the plurality of documents.

7. The computer-implemented method of claim 1, wherein the one or more categories each include one or more topic vectors, where each of the one or more topic vectors include one or more identified keywords within the textual data as well as a frequency of the one or more identified keywords within the textual data.

8. The computer-implemented method of claim 1, wherein the one or more categories are used to perform classification of additional textual data.

9. The computer-implemented method of claim 1, wherein analyzing the textual data for each of the plurality of documents may include performing a latent dirichlet allocation (LDA) analysis on the textual data to identify the one or more categories.

10. The computer-implemented method of claim 9, wherein performing the LDA analysis on the textual data includes transforming the textual data into an array and determining the one or more categories from the array.

11. A computer program product for identifying one or more categories within textual data of each of a plurality of documents, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, wherein the computer readable storage medium is not a transitory signal per se, the program instructions executable by a processor to cause the processor to perform a method comprising:

identifying, by the processor, a plurality of documents associated with a predetermined subject, where each of the plurality of documents contains the textual data;

analyzing, by the processor, the textual data of each of the plurality of documents to identify the one or more categories within the plurality of the documents; and

returning, by the processor, the one or more categories identified within the plurality of the documents.

12. The computer program product of claim 11, wherein the plurality of documents include one or more of web page content and scanned document content.

13. The computer program product of claim 11, wherein the plurality of documents is included within a grouping that includes identifiers of a location of each of the plurality of documents.

14. The computer program product of claim 11, wherein analyzing, by the processor, the textual data for each of the plurality of documents includes performing, by the processor, automatic language detection on the textual data.

15. The computer program product of claim 11, wherein analyzing, by the processor, the textual data for each of the plurality of documents includes refining, by the processor, the textual data by removing one or more words from the textual data.

16. The computer program product of claim 11, wherein analyzing, by the processor, the textual data for each of the plurality of documents includes removing, by the processor, any duplicate documents within the plurality of documents.

17. The computer program product of claim 11, wherein the one or more categories each include one or more topic vectors, where each of the one or more topic vectors include one or more identified keywords within the textual data as well as a frequency of the one or more identified keywords within the textual data.

18. The computer program product of claim 11, wherein the one or more categories are used to perform classification of additional textual data.

19. The computer program product of claim 11, wherein analyzing, by the processor, the textual data for each of the plurality of documents may include performing, by the processor, a latent dirichlet allocation (LDA) analysis on the textual data to identify the one or more categories.

20. A system, comprising:

a processor and logic integrated with and/or executable by the processor, the logic being configured to;

identify a plurality of documents associated with a predetermined subject, where each of the plurality of documents contains textual data;

analyze the textual data of each of the plurality of documents to identify one or more categories within the plurality of the documents; and

return the one or more categories identified within the plurality of the documents.