Method For Preserving Conceptual Distance Within Unstructured Documents

• US 20160098398A1
• Filed: 03/09/2015
• Published: 04/07/2016
• Est. Priority Date: 10/07/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Bufe, John P., Winkler, Timothy P.
• Granted Patent: US 9,424,299 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 40/247 Thesauruses; Synonyms
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30

Abstract

A method, system and computer-usable medium are disclosed for preserving conceptual distance within unstructured documents by characterizing conceptual relationships. Natural language processing is applied to content in a plurality of documents to identify topics and subjects. Analytic analysis is then applied to the identified topics and subjects to identify concepts. The content in each of the plurality of documents is partitioned into a first structured hierarchy, preserving at least one structure in each document inherent in the each document. Access is then provided to the content through a first index based upon utilizing the first structured hierarchy and through a second index utilizing a second structured hierarchy. The conceptual relationship criteria are based upon a directed graph with weights based upon a similarity and a distance based upon concepts.

Claims

1. A computer-implemented method for characterizing content of documents by conceptual relationships, comprising:

applying natural language processing (NLP) to content in a plurality of documents to identify topics and subjects;

applying analytic analysis to the topics and subjects to identify a conceptual relationships of the content in the plurality of documents;

partitioning the content in each of the plurality of documents into a first structured hierarchy, preserving at least one structure in each document inherent in the each document; and

providing access to content through a first index based upon utilizing the first structured hierarchy and through a second index utilizing a second structured hierarchy.

2. The method of claim 1, wherein:

the conceptual relationship is based upon a directed graph with weights based upon a similarity and a distance based upon concepts.

3. The method of claim 2, wherein:

the distance is based upon a topic hierarchy.

4. The method of claim 1, wherein the content is characterized by optimizing a vector space model representation of the documents, the optimization performed by a system capable of answering questions, wherein:

the content from the plurality of documents is ingested by the system;

natural language processing is applied to the content in the plurality of documents to identify terms, topics, subjects and concepts;

the content is partitioned according to a semantic parse distance to identify a context for partitioned content;

the content and context is represented, by the system, utilizing a vector space model;

entries in the vector space model are eliminated based on a difference criteria; and

an iterative genetic algorithm is applied to optimize features of the vector space model.

5. The method of claim 4, wherein:

a ground truth is an optimized feature.

6. The method of claim 5, wherein:

the genetic algorithm determines which features are used during the ingesting and has weighting based on semantic distance.

7-20. -20. (canceled)