Concept Driven Automatic Section Identification

• US 20150088888A1
• Filed: 09/26/2013
• Published: 03/26/2015
• Est. Priority Date: 09/26/2013
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Carrier, Scott R., McCloskey, Daniel J., Brennan, Paul E., McAteer, Seamus R.
• Granted Patent: US 9,058,374 B2
• Time in Patent Office: Days
• US Class Current: 707/737
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/38 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/258 Heading extraction; Automatic titling; Numbering

Abstract

Mechanisms are provided for generating section metadata for an electronic document. These mechanisms receive a document and analyze the document to identify concepts present within textual content of the document. The mechanisms correlate concepts within the textual content with one another to identify concept groups based on the application of one or more rules defining related concepts or concept patterns. The mechanisms determine sections of text within the textual content based on the correlation of concepts within the textual content. Based on results of the determining, the mechanisms generate section metadata for the document and store the section metadata in association with the document for use by a document processing system.

Claims

1. A method, in a data processing system comprising a processor and a memory, for generating section metadata for an electronic document, the method comprising:

receiving, by the data processing system, an electronic document for processing;

analyzing, by the data processing system, the electronic document to identify concepts present within textual content of the electronic document;

correlating, by the data processing system, concepts within the textual content with one another to identify concept groups within the textual content based on the application of one or more rules defining related concepts or concept patterns;

determining, by the data processing system, at least one section of text within the textual content based on the correlation of concepts within the textual content;

generating, by the data processing system, based on results of the determining, section metadata for the electronic document to thereby identify the at least one section in the electronic document; and

storing, by the data processing system, the section metadata in association with the electronic document for use by a document processing system.

2. The method of claim 1, wherein determining at least one section of text within the textual content further comprises utilizing a knowledge base specifying a plurality of concepts or concept groups and determining a measure of affinity between each concept or concept group in the knowledge base and portions of the textual content.

3. The method of claim 1, wherein determining at least one section of text within the textual content further comprises determining a measure of affinity between a concept in a candidate section of text in the textual content to a section header in the textual content.

4. The method of claim 1, wherein determining at least one section of text within the textual content further comprises:

identifying a candidate section heading and a proposed section of text within the electronic document;

determining whether the proposed section of text has a threshold measure of affinity to the candidate section heading based on one or more concept elements within the proposed section of text; and

selecting the proposed section of text to be an actual section of text associated with the candidate section heading in response to the proposed section of text having at least the threshold measure of affinity to the candidate section heading.

5. The method of claim 4, wherein the candidate section heading is an inferred section heading inferred from a concept or concept group associated with the proposed section of text.

6. The method of claim 1, wherein determining at least one section of text within the textual content based on the correlation of concepts within the textual content comprises performing a statistical analysis of the concepts within the textual content, wherein the statistical analysis comprises at least one of concept density analysis, inverse document frequency analysis, or concept commonality analysis amongst a plurality of proposed sections of text within the textual content.

7. The method of claim 1, wherein the textual content does not have associated structural annotations, and wherein determining at least one section of text within the textual content based on the correlation of concepts within the textual content is performed without regard to structural annotations being associated with the textual content.

8. The method of claim 1, wherein determining at least one section of text within the textual content based on the correlation of concepts within the textual content further comprises determining the at least one section of text within the textual content also based on structural annotations associated with the textual content.

9. The method of claim 1, wherein the document processing system is a natural language processing system, and wherein the method further comprises performing natural language processing on the electronic document based on the section metadata associated with the electronic document.

10. The method of claim 1, wherein the document processing system is a question and answer system and wherein the electronic document is part of a corpus of documents ingested by the question and answer system, and wherein the method further comprises performing a question answering operation on an input question using the section metadata associated with the electronic document, wherein the section metadata is used by the question and answer system to modify a score associated with a candidate answer obtained from the electronic document.

11. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a data processing system, causes the data processing system to:

receive, an electronic document for processing;

analyze the electronic document to identify concepts present within textual content of the electronic document;

correlate concepts within the textual content with one another to identify concept groups within the textual content based on the application of one or more rules defining related concepts or concept patterns;

determine at least one section of text within the textual content based on the correlation of concepts within the textual content;

generate, based on results of the determining, section metadata for the electronic document to thereby identify the at least one section in the electronic document; and

store the section metadata in association with the electronic document for use by a document processing system.

12. The computer program product of claim 11, wherein the computer readable program causes the data processing system to determine at least one section of text within the textual content further at least by utilizing a knowledge base specifying a plurality of concepts or concept groups and determining a measure of affinity between each concept or concept group in the knowledge base and portions of the textual content.

13. The computer program product of claim 11, wherein the computer readable program causes the data processing system to determine at least one section of text within the textual content further at least by determining a measure of affinity between a concept in a candidate section of text in the textual content to a section header in the textual content.

14. The computer program product of claim 11, wherein the computer readable program causes the data processing system to determine at least one section of text within the textual content further at least by:

identifying a candidate section heading and a proposed section of text within the electronic document;

determining whether the proposed section of text has a threshold measure of affinity to the candidate section heading based on one or more concept elements within the proposed section of text; and

selecting the proposed section of text to be an actual section of text associated with the candidate section heading in response to the proposed section of text having at least the threshold measure of affinity to the candidate section heading.

15. The computer program product of claim 14, wherein the candidate section heading is an inferred section heading inferred from a concept or concept group associated with the proposed section of text.

16. The computer program product of claim 11, wherein the computer readable program causes the data processing system to determine at least one section of text within the textual content based on the correlation of concepts within the textual content at least by performing a statistical analysis of the concepts within the textual content, wherein the statistical analysis comprises at least one of concept density analysis, inverse document frequency analysis, or concept commonality analysis amongst a plurality of proposed sections of text within the textual content.

17. The computer program product of claim 11, wherein the textual content does not have associated structural annotations, and wherein determining at least one section of text within the textual content based on the correlation of concepts within the textual content is performed without regard to structural annotations being associated with the textual content.

18. The computer program product of claim 11, wherein the computer readable program causes the data processing system to determine at least one section of text within the textual content based on the correlation of concepts within the textual content further at least by determining the at least one section of text within the textual content also based on structural annotations associated with the textual content.

19. The computer program product of claim 11, wherein the document processing system is a question and answer system and wherein the electronic document is part of a corpus of documents ingested by the question and answer system, and wherein the computer readable program further causes the data processing system to perform a question answering operation on an input question using the section metadata associated with the electronic document, wherein the section metadata is used by the question and answer system to modify a score associated with a candidate answer obtained from the electronic document.

20. An apparatus comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to;

receive, an electronic document for processing;

analyze the electronic document to identify concepts present within textual content of the electronic document;

correlate concepts within the textual content with one another to identify concept groups within the textual content based on the application of one or more rules defining related concepts or concept patterns;

determine at least one section of text within the textual content based on the correlation of concepts within the textual content;

generate, based on results of the determining, section metadata for the electronic document to thereby identify the at least one section in the electronic document; and

store the section metadata in association with the electronic document for use by a document processing system.