Error correction in tables using discovered functional dependencies

• US 9,569,417 B2
• Filed: 09/22/2014
• Issued: 02/14/2017
• Est. Priority Date: 06/24/2013
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Byron, Donna K., Gerard, Scott N., Pikovsky, Alexander, Winkler, Timothy P.
• Primary Examiner(s): LE, HUNG D
• Application Number:
  US14/492,172
• Publication Number:
  US 20150046785A1
• Time in Patent Office: 876 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 40/177 of tables; using ruled lines
• CPC Class Codes: G06F 40/18 of spreadsheets form-filling G06F40/174

Abstract

Mechanisms are provided for performing tabular data correction in a document. Tabular data is received and analyzed to identify at least one portion of the tabular data having an erroneous/missing data value. A functional dependency of the at least one portion of the tabular data on one or more other portions of the tabular data is determined. A correct data value for the erroneous or missing data value of the at least one portion of the tabular data is determined based on the functional dependency of the at least one portion. In addition, the tabular data is modified to replace the erroneous or missing data value with the correct data value and thereby generate a modified table data. A processing operation is then performed on the modified table data to generate a resulting output.

Claims

1. A method, in a data processing system comprising a processor and a memory, for performing tabular data correction in a document, the method comprising:

receiving, by the data processing system, a natural language document comprising tabular data;

analyzing, by the data processing system, the tabular data within the natural language document to identify at least one portion of the tabular data comprising an erroneous or missing data value;

determining, by the data processing system, for the at least one portion, a functional dependency of the at least one portion of the tabular data on one or more other portions of the tabular data by reference to the natural language document;

determining, by the data processing system, for the at least one portion, a correct data value for the erroneous or missing data value of the at least one portion of the tabular data based on the determined functional dependency of the at least one portion;

modifying, by the data processing system, the tabular data to replace the erroneous or missing data value with the correct data value and thereby generate a modified tabular data structure; and

performing, by the data processing system, a processing operation on the modified tabular data structure to generate an resulting output.

2. The method of claim 1, wherein the natural language document is an original document of a corpus of information, and wherein the modified tabular data structure is part of a modified document that replaces the original document in the corpus of information.

3. The method of claim 2, wherein the processing operation is a natural language processing operation that is applied to the modified document.

4. The method of claim 3, wherein the natural language processing operation is part of a question answering operation of a Question and Answer (QA) system.

5. The method of claim 1, wherein the natural language document is received as part of at least one of an optical character reading operation, a natural language processing operation, or a user input of the tabular data.

6. The method of claim 1, wherein analyzing the tabular data to identify at least one portion of the tabular data comprising an erroneous or missing data value comprises identifying the at least one portion of the tabular data as comprising an erroneous or missing data value due to an absence of a homogenous cell block criteria for the at least one portion of the tabular data.

7. The method of claim 1, wherein analyzing the tabular data to identify at least one portion of the tabular data comprising an erroneous or missing data value comprises:

analyzing the at least one portion of the tabular data to determine whether the at least one portion of tabular data comprises a regular structure having a repeated pattern of data or an identifiable dependency between cells in the at least one portion of the tabular data; and

in response to a regular structure being identified in the at least one portion of the tabular data, performing further analysis to determine if one or more cells within the at least one portion of the tabular data do not conform with the regular structure.

8. The method of claim 1, wherein determining for the at least one portion, a correct data value for the erroneous or missing data value of the at least one portion of the tabular data based on the functional dependency of the at least one portion comprises at least one of inverting the functional dependency of the at least one portion to generate the corrected data value or approximating the corrected data value using an iterative procedure.

9. The method of claim 1, wherein analyzing the tabular data to identify at least one portion of the tabular data comprising an erroneous or missing data value comprises identifying the at least one portion of the tabular data as comprising an erroneous or missing data value due to a discrepancy between a data value type specified by information in a header of one or more columns or rows associated with the at least one portion of the tabular data and the erroneous or missing data value.

10. The method of claim 1, further comprising:

outputting a notification to a user indicating the portion of the tabular data comprising the erroneous or missing data value and the corrected data value as a suggested replacement for the erroneous or missing data value, wherein the modifying of the tabular data is performed in response to receiving user input approving the corrected data value for replacing the erroneous or missing data value.