Methods and systems for data cleaning

• US 10,545,932 B2
• Filed: 02/07/2014
• Issued: 01/28/2020
• Est. Priority Date: 02/07/2013
• Status: Active Grant

• Current Assignee: Qatar Foundation
• Original Assignee: Qatar Foundation
• Inventors: Tang, Nan, Wang, Jiannan
• Primary Examiner(s): Spieler, William
• Application Number:
  US14/765,985
• Publication Number:
  US 20150379051A1
• Time in Patent Office: 2,181 Days
• Field of Search: 707691, 707692, 715205, 715234, 715771
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/217 Database tuning G06F16/2282 takes precedence; database performance monitoring G06F11/3409

Abstract

A method for cleaning data stored in a database, the method comprising providing a set of fixing rules. Each fixing rule incorporates a set of attribute values that capture an error in a plurality of semantically related attribute values, and a deterministic correction which is operable to replace one of the set of attribute values with a correct attribute value to correct the error. The method further comprises comparing at least two of the fixing rules with one another to check that the error correction carried out by one fixing rule is consistent with the error correction carried out by another fixing rule.

Claims

1. A computer-implemented method for cleaning data stored in a database, the method comprising:

providing a data fixing rule to capture an error, the data fixing rule incorporating;

a first set of attributes and respective first set of attribute values;

a second attribute and respective second set of attribute values, wherein the second set of attribute values are erroneous values; and

a correct value;

applying the data fixing rule to the database to capture the error, wherein the error is captured when the first set of attributes and attribute values, and the second attribute and at least one of the erroneous values of the second attribute match a record in the database; and

replacing the at least one erroneous value in the record with the correct value.

2. The method of claim 1, further comprising applying a plurality of the data fixing rules to the tuples stored in the database, the method applying each data fixing rule only once to a respective tuple.

3. The method of claim 1, further comprising:

applying a plurality of the data fixing rules to the tuples stored in the database;

applying each data fixing rule only once to a respective tuple; and

allocating an attribute to each tuple which indicates each data fixing rule that has been applied to the tuple.

4. The method of claim 3, further comprising:

applying a plurality of the data fixing rules to the tuples stored in the database;

applying each data fixing rule only once to a respective tuple;

allocating an attribute to each tuple which indicates each data fixing rule that has been applied to the tuple; and

incrementing at least one hash counter to record when a data fixing rule is applied to a tuple.

5. The method of claim 1, further comprising generating an inverted list of a plurality of data fixing rules, the inverted list comprising the plurality of data fixing rules indexed according to at least one attribute value of each respective data fixing rule.

6. The method of claim 1, further comprising generating an inverted list of a plurality of data fixing rules only once, the inverted list comprising the plurality of data fixing rules indexed according to at least one attribute value of each respective data fixing rule.

7. The method of claim 1, wherein the data fixing rule comprises at least one similarity operator which is operable to detect variants of the first set and second set of attribute values in the database.

8. The method of claim 1, wherein the data fixing rule is operable to use a wildcard attribute value in the first set and second set of attribute values.

9. The method of claim 1, wherein the data fixing rule is operable to detect the negation of an attribute value.

10. The method of claim 1, further comprising providing a plurality of data fixing rules and applying at least one of the plurality of data fixing rules to the database.

11. A system for cleaning data stored in a database, the system comprising a processor configured to:

provide a data fixing rule to capture an error, the data fixing rule incorporating;

a first set of attributes and respective first set of attribute values;

a second attribute and respective second set of attribute values, wherein the second set of attribute values are erroneous values; and

a correct value;

applying the data fixing rule to the database to capture the error, wherein the error is captured when the first set of attributes and attribute values, and the second attribute and at least one of the erroneous values of the second attribute match a record in the database; and

replacing the at least one erroneous value in the record with the correct value.

12. A non-transitory tangible computer readable storage medium comprising instructions for performing a process to be executed on a computer, the process comprising:

providing a data fixing rule to capture an error, the data fixing rule incorporating;

a first set of attributes and respective first set of attribute values;

a second attribute and respective second set of attribute values, wherein the second set of attribute values are erroneous values; and

a correct value;

applying the data fixing rules to the database to capture the error, wherein the error is captured when the first set of attributes and attribute values, and the second attribute and at least one of the erroneous values of the second attribute match a record in the database; and

replacing the at least one erroneous value in the record with the correct value.