Ingestion planning for complex tables

• US 9,483,455 B1
• Filed: 02/18/2016
• Issued: 11/01/2016
• Est. Priority Date: 10/23/2015
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Bastide, Paul R., Broomhall, Matthew E., Byron, Donna K., Loredo, Robert E.
• Primary Examiner(s): Bashore, William
• Assistant Examiner(s): Faber, David
• Application Number:
  US15/046,541
• Time in Patent Office: 257 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/148 File search processing
• CPC Class Codes: G06F 16/2282 Tablespace storage structures; Management thereof
• CPC Class Codes: G06F 16/243 Natural language query formulation
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06F 16/3326 using relevance feedback from the user, e.g. relevance feedback on documents, documents sets, document terms or passages
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 16/345 Summarisation for human users
• CPC Class Codes: G06F 16/41 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 40/151 Transformation
• CPC Class Codes: G06F 40/177 of tables; using ruled lines
• CPC Class Codes: G06F 40/20 Natural language analysis semantic analysis of natural language G06F40/30
• CPC Class Codes: G06F 40/56 Natural language generation

Abstract

Embodiments of the present invention disclose a method, computer program product, and system for generating a plan for document processing. A plurality of electronic documents are received from a data store. The plurality of electronic documents are analyzed. Textual data within the identified tabular data are identified, by performing a first natural language search of the analyzed plurality of electronic documents. Textual hints are generated, where the generated textual hints are mapped to a lookup set. References are identified, and a count of identified references are determined. A priority score is calculating based on the count of identified references. In response to receiving a priority score modifying value, a modified priority score is calculated. Ingestion plans are generated based on the modified priority score. Generated ingestion plans are communicated by the computer using the network.

Claims

1. A computer-implemented method for generating a plan for document processing, the method comprising:

receiving a plurality of electronic documents from a data store, by a computer using a network;

analyzing the plurality of electronic documents, using the computer to identify a plurality of tabular data by performing a search for one or more table markers, based on the analyzed plurality of electronic documents;

identifying textual data within the identified tabular data, by performing a first natural language search of the analyzed plurality of electronic documents;

generating textual hints, based on the identified textual data within the identified tabular data by associating identified textual data into a set using a second natural language search;

mapping the generated textual hints to a lookup set;

identifying references, wherein references are based on mapped textual hints with associated identified textual data in the received plurality of electronic documents;

determining a count of identified references;

calculating a priority score based on the count of identified references, wherein the calculating further comprises multiplying the count of identified references by a predetermined scale value;

in response to receiving a priority score modifying value, wherein the priority score modifying value is a numerical value, calculating a modified priority score, wherein the calculating further comprises multiplying the priority score by the received priority score modifying value;

generating one or more ingestion plans, the one or more ingestion plans comprising an ordered list of the identified references and associated tabular data, where ordering of the ordered list is based at least in part on the modified priority score; and

communicating the one or more generated ingestion plans by the computer using the network.