METHODS AND SYSTEMS FOR DATA CLEANING

• US 20150379051A1
• Filed: 02/07/2014
• Published: 12/31/2015
• Est. Priority Date: 02/07/2013
• Status: Active Grant

• Current Assignee: Qatar Foundation
• Original Assignee: Qatar Foundation
• Inventors: TANG, Nan, WANG, Jiannan, HOARTON, Lloyd
• Granted Patent: US 10,545,932 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/217 Database tuning G06F16/2282 takes precedence; database performance monitoring G06F11/3409

Abstract

A method for cleaning data stored in a database, the method comprising providing a set of fixing rules. Each fixing rule incorporates a set of attribute values that capture an error in a plurality of semantically related attribute values, and a deterministic correction which is operable to replace one of the set of attribute values with a correct attribute value to correct the error. The method further comprises comparing at least two of the fixing rules with one another to check that the error correction carried out by one fixing rule is consistent with the error correction carried out by another fixing rule.

Claims

1. A method for cleaning data stored in a database, the method comprising:

providing a set of fixing rules, each fixing rule incorporating;

a set of attribute values that capture an error in a plurality of semantically related attribute values, and

a deterministic correction which is operable to replace one of the set of attribute values with a correct attribute value to correct the error,

wherein the method further comprises;

comparing at least two of the fixing rules with one another to check that the error correction carried out by one fixing rule is consistent with the error correction carried out by another fixing rule.

2. The method of claim 1, wherein the method comprises comparing all fixing rules in the set of fixing rules pairwise with one another.

3. The method of claim 1, wherein the method comprises applying at least two of the fixing rules to a tuple of attribute values to check whether the at least two fixing rules apply different corrections to the tuple, thereby indicating that the at least two fixing rules are not consistent with one another.

4. The method of claim 1, wherein the method comprises identifying a tuple of attribute values that satisfies two of the fixing rules and applying the two fixing rules to the tuple alternately in different orders to determine if the two fixing rules apply different error corrections to the tuple when the fixing rules are applied to the tuple in different orders, thereby indicating that the fixing rules are not consistent with one another.

5. The method of claim 1, wherein the method comprises applying at least two of the fixing rules to a tuple of attribute values to check whether the at least two fixing rules apply different corrections to the tuple, thereby indicating that the at least two fixing rules are not consistent with one another;

and

combining at least part of two inconsistent fixing rules with one another to produce one or more modified fixing rules which are consistent with one another.

7. The method of claim 1, wherein the method comprises applying at least two of the fixing rules to a tuple of attribute values to check whether the at least two fixing rules apply different corrections to the tuple, thereby indicating that the at least two fixing rules are not consistent with one another;

and

outputting at least two fixing rules that are not consistent with one another to a user so that the user can amend or delete at least one of the fixing rules to remove the inconsistency.

8. The method of claim 1, wherein the method further comprises:

applying at least one of the fixing rules to a plurality of tuples stored in a database to detect if at least one of the tuples comprises the respective set of attribute values that captures the error and, if the respective set of attribute values is detected, applying the deterministic correction to correct the error in the at least one tuple.

21. A tangible computer readable storage medium comprising instructions which, when executed, cause an apparatus to perform the method of claim 1.

6. (canceled)

9. A method for cleaning data stored in a database, the method comprising:

providing a set of fixing rules, each fixing rule incorporating;

a set of attribute values that capture an error in a plurality of semantically related attribute values, and

a deterministic correction which is operable to replace one of the set of attribute values with a correct attribute value to correct the error,

wherein the method comprises;

applying at least one of the fixing rules to a plurality of tuples stored in a database to detect if at least one of the tuples comprises the respective set of attribute values that captures the error and, if the respective set of attribute values is detected, applying the deterministic correction to correct the error in the at least one tuple.

10. The method of claim 9, wherein the method comprises applying a plurality of the fixing rules to the tuples stored in the database, the method applying each fixing rule only once to a respective tuple.

11. The method of claim 9, wherein the method comprises applying a plurality of the fixing rules to the tuples stored in the database;

applying each fixing rule only once to a respective tuple; and

allocating an attribute to each tuple which indicates each fixing rule that has been applied to the tuple.

12. The method of claim 11, wherein the method comprises applying a plurality of the fixing rules to the tuples stored in the database;

applying each fixing rule only once to a respective tuple;

allocating an attribute to each tuple which indicates each fixing rule that has been applied to the tuple; and

incrementing at least one counter to record when a fixing rule is applied to a tuple.

13. The method of claim 12, wherein each counter is a hash counter which records the number of tuples that correspond to each fixing rule.

14. The method of claim 9, wherein the method further comprises generating an inverted list of a plurality of fixing rules, the inverted list comprising the plurality of fixing rules indexed according to at least one attribute value of each respective fixing rule.

15. The method of claim 9, wherein the method further comprises generating an inverted list of a plurality of fixing rules only once, the inverted list comprising the plurality of fixing rules indexed according to at least one attribute value of each respective fixing rule.

16. The method of claim 9, wherein the fixing rule comprises at least one similarity operator which is operable to detect variants of attribute values.

17. The method of claim 9, wherein the fixing rule is operable to use a wildcard attribute value in the set of attribute values.

18. The method of claim 9, wherein the fixing rule is operable to detect the negation of an attribute value.

19. The method of claim 9, wherein the method comprises providing a plurality of fixing rules and applying at least one of the plurality of fixing rules to the database.

20. A system for cleaning data stored in a database, the system being operable to perform the method of claim 9.