Ingestion plan based on table uniqueness

• US 9,679,198 B2
• Filed: 11/05/2015
• Issued: 06/13/2017
• Est. Priority Date: 11/05/2015
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Bastide, Paul R., Broomhall, Matthew E., Byron, Donna K., Loredo, Robert E., Pikovsky, Alexander
• Primary Examiner(s): Dang, Duy M
• Application Number:
  US14/933,020
• Publication Number:
  US 20170132463A1
• Time in Patent Office: 586 Days
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 18/2323 based on graph theory, e.g. minimum spanning trees [MST] or graph cuts
• CPC Class Codes: G06V 30/412 Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables
• CPC Class Codes: G06V 30/413 Classification of content, e.g. text, photographs or tables
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: G06V 30/416 Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors

Abstract

Embodiments of the present invention disclose a method, computer program product, and system for a computer implemented method for processing tabular data. In various embodiments, an electronic document is received through a network, along with associated metadata. A plurality of table markers, or tabular data markers, are identified, in response to analyzing the received electronic document for said markers. References and citations associated with the plurality of tabular data markers are identified. A graphical representation of the relationship between identified tabular data markers and the identified references is generated. A uniqueness score is calculated, based on the generated graph and an ingestion plan is generated for the received electronic documents based on the calculated uniqueness score value.

Claims

1. A computer implemented method for processing tabular data, the method comprising:

receiving an electronic document on a computer through a network;

receiving metadata associated with the received electronic document;

identifying a plurality of tabular data markers, in response to analyzing the received electronic document and associated metadata;

identifying references for association with the identified plurality of tabular data markers;

generating a graphical representation of the relationship between the identified tabular data markers and identified references;

calculating a uniqueness score value based on the generated graphical representation; and

generating an ingestion plan for the received electronic documents based on the calculated uniqueness score value.

2. The method of claim 1, wherein identifying references further comprises:

identifying HMTL links in the received document;

identifying textual data within the received document as related to identified tabular data in the received document, in response to a natural language analysis of the received document; and

identifying tabular data authorship, in response to a natural language analysis of the received metadata.

3. The method of claim 1, wherein the graphical representation includes, tabular data as vertices, and received documents as edges, wherein the edges are directional and based on a count of identified references.

4. The method of claim 3, wherein the uniqueness score is modified based on one or more of:

a first count based on the directional edges to a vertex in the graphical representation;

multiplying the uniqueness score by zero, in response to the count of direction edges not exceeding a threshold; and

a second count based on the of vertexes in the graphical representation exceeding a threshold.

5. The method of claim 1, further comprising:

in response to receiving a second electronic document, applying the generated ingestion plan to the received second electronic document.

6. The method of claim 1, wherein a uniqueness score is modified by a predetermined value, in response to receiving a user input.

7. A computer program product for processing tabular data, the computer program product comprising:

a computer-readable storage media having program instructions stored on the computer-readable storage media, the program instructions, executable by a device, comprising;

instructions to receive an electronic document through a network;

instructions to receive metadata associated with the received electronic document;

instructions to identify a plurality of tabular data markers, in response to analyzing the received electronic document and associated metadata;

instructions to identify references for association with the identified plurality of tabular data markers;

instructions to generate a graphical representation of the relationship between the identified tabular data markers and identified references;

instructions to calculate a uniqueness score value based on the generated graphical representation; and

instructions to generate an ingestion plan for the received electronic documents based on the calculated uniqueness score value.

8. The computer program product of claim 7, wherein instructions to identify references further comprises:

instructions to identify HMTL links in the received document;

instructions to identify textual data within the received document as related to identified tabular data in the received document, in response to a natural language analysis of the received document; and

instructions to identify tabular data authorship, in response to a natural language analysis of the received metadata.

9. The computer program product of claim 7, wherein the graphical representation includes, tabular data as vertices, and received documents as edges, wherein the edges are directional and based on a count of identified references.

10. The computer program product of claim 9, wherein the uniqueness score is modified based on one or more of:

a first count based on the directional edges to a vertex in the graphical representation;

multiplying the uniqueness score by zero, in response to the count of direction edges not exceeding a threshold; and

a second count based on the of vertexes in the graphical representation exceeding a threshold.

11. The computer program product of claim 7, further comprising:

in response to receiving a second electronic document, instructions to apply the generated ingestion plan to the received second electronic document.

12. The computer program product of claim 7, wherein a uniqueness score is modified by a predetermined value, in response to receiving a user input.

13. A computer system for processing tabular data, the computer system comprising:

one or more computer processors;

one or more computer-readable storage media;

program instructions stored on the computer-readable storage media for execution by at least one of the one or more processors, the program instructions comprising;

instructions to A computer implemented method for processing tabular data, the method comprising;

instructions to receive an electronic document through a network;

instructions to receive metadata associated with the received electronic document;

instructions to identify a plurality of tabular data markers, in response to analyzing the received electronic document and associated metadata;

instructions to identify references for association with the identified plurality of tabular data markers;

instructions to generate a graphical representation of the relationship between the identified tabular data markers and identified references;

instructions to calculate a uniqueness score value based on the generated graphical representation; and

instructions to generate an ingestion plan for the received electronic documents based on the calculated uniqueness score value.

14. The system of claim 13, wherein instructions to identify references further comprises:

instructions to identify HMTL links in the received document;

instructions to identify textual data within the received document as related to identified tabular data in the received document, in response to a natural language analysis of the received document; and

instructions to identify tabular data authorship, in response to a natural language analysis of the received metadata.

15. The system of claim 13, wherein the graphical representation includes, tabular data as vertices, and received documents as edges, wherein the edges are directional and based on identified references.

16. The system of claim 15, wherein the uniqueness score is modified based on one or more of:

a first count based on the directional edges to a vertex in the graphical representation;

multiplying the uniqueness score by zero, in response to the count of direction edges not exceeding a threshold; and

a second count based on the of vertexes in the graphical representation exceeding a threshold.

17. The system of claim 13, further comprising:

in response to receiving a second electronic document, instructions to apply the generated ingestion plan to the received second electronic document.

18. The system of claim 13, wherein a uniqueness score is modified by a predetermined value, in response to receiving a user input.