METHOD, SYSTEM, PROGRAM AND DATA STRUCTURE FOR CLEANING A DATABASE TABLE

• US 20060195489A1
• Filed: 05/11/2006
• Published: 08/31/2006
• Est. Priority Date: 09/21/1999
• Status: Abandoned Application

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Mandel, Richard Henry III, Cesare, Mark Anthony, Jerves, Julie Ann, Christopher, Tom Robert
• Application Number:
  US11/382,894
• Publication Number:
  US 20060195489A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99936 Pattern matching access
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

Disclosed is a method, system, program, and data structure for performing a clean operation on an input table. The input table to clean is indicated in an input data table name. At least one rule definition is processed to clean the input table. Each rule definition indicates a find criteria, a replacement value, and an input data column in the input table. The rule definition comprises a type of rule that is a member of the set of rules consisting of: find and replace, discretization, and numeric clip, and at least two rule definitions are comprised of different rule types. For each rule definition, the input data column is searched for any fields that match the find criteria. The replacement value for the particular rule definition is inserted in the fields in the input data column that match the find criteria. Subsequent applications of additional rule definitions applied to the same input data column operate on replacement values inserted in the input data column during previously applied rule definitions.

Claims

1. A computer implemented method for performing a clean operation on an input table having an input table name, comprising:

receiving at least one rule definition, wherein each rule definition indicates a find criteria, a replacement value, and an input data column in the input table, wherein the rule definition comprises a type of rule that is a member of the set of rules consisting of;

find and replace, discretization, and numeric clip, and wherein at least two rule definitions are comprised of different rule types;

searching, for each rule definition, the input data column for any fields that match the find criteria;

determining whether the rule definition specifies an output table; and

in response to determining that the rule definition does not specify an output table, directly inserting, for each rule definition, the replacement value in the fields in the input data column that match the find criteria, wherein subsequent applications of additional rule definitions applied to the same input data column operate on replacement values inserted in the input data column in previously applied rule definitions.

2. The method of claim 1, wherein at least one rule definition includes multiple find criteria and a corresponding replacement value for each find criteria, wherein searching the input data column comprises applying each of the multiple find criteria to one field until one of:

(i) a match occurs and (ii) none of the multiple find criteria are found to match the field content, and wherein inserting the replacement value comprises inserting the replacement value corresponding to one find criteria that matched the field content.

3. The method of claim 1, wherein the find criteria for at least one rule definition comprises an upper bound and lower bound, wherein searching comprises searching for any fields that have values within the upper and lower bounds.

4. The method of claim 3, wherein searching comprises searching for any fields that have values outside of one of the upper and lower bounds.

5. The method of claim 1, wherein each rule definition includes a row clean flag, and wherein at least one rule definition has the row clean flag set, further comprising removing any row including a field matching the search criteria from the input table when the row clean flag is set.

6. A computer implemented system for performing a clean operation on an input table having an input data table name, comprising;

means for receiving at least one rule definition, wherein each rule definition indicates a find criteria, a replacement value, and an input data column in the input table, wherein the rule definition comprises a type of rule that is a member of the set of rules consisting of;

find and replace, discretization, and numeric clip, and wherein at least two rule definitions are comprised of different rule types;

means for searching, for each rule definition, the input data column for any fields that match the find criteria;

means for determining whether the rule definition specifies an output table; and

means for, in response to determining that the rule definition does not specify an output table, directly inserting, for each rule definition, the replacement value in the fields in the input data column that match the find criteria, wherein subsequent applications of additional rule definitions applied to the same input data column operate on replacement values inserted in the input data column in previously applied rule definitions.

7. The system of claim 6, wherein at least one rule definition includes multiple find criteria and a corresponding replacement value for each find criteria, wherein the means for searching the input data column comprises applying each of the multiple find criteria to one field until one of:

(i) a match occurs and (ii) none of the multiple find criteria are found to match the field content, and wherein the means for inserting the replacement value comprises inserting the replacement value corresponding to one find criteria that matched the field content.

8. The system of claim 6, wherein the find criteria for at least one rule definition comprises an upper bound and lower bound, and wherein searching comprises searching for any fields that have values within the upper and lower bounds.

9. The system of claim 8, wherein the means for searching comprises searching for any fields that have values outside of one of the upper and lower bounds.

10. The system of claim 6, wherein each rule definition includes a row clean flag, and wherein at least one rule definition has the row clean flag set, further comprising means for removing any row including a field matching the search criteria from the input table when the row clean flag is set.

11. An article of manufacture for use performing a clean operation on an input table in a database having an input data table name, the article of manufacture comprising computer usable media including at least one computer program embedded therein that causes the computer to perform:

receiving at least one rule definition, wherein each rule definition indicates a find criteria, a replacement value, and an input data column in the input table, wherein the rule definition comprises a type of rule that is a member of the set of rules consisting of;

find and replace, discretization, and numeric clip, and wherein at least two rule definitions are comprised of different rule types;

searching, for each rule definition, the input data column for any fields that match the find criteria;

determining whether the rule definition specifies an output table; and

in response to determining that the rule definition does not specify an output table, directly inserting, for each rule definition, the replacement value in the fields in the input data column that match the find criteria, wherein subsequent applications of additional rule definitions applied to the same input data column operate on replacement values inserted in the input data column in previously applied rule definitions.

12. The article of manufacture of claim 11, wherein at least one rule definition includes multiple find criteria and a corresponding replacement value for each find criteria, wherein searching the input data column comprises applying each of the multiple find criteria to one field until one of:

(i) a match occurs and (ii) none of the multiple find criteria are found to match the field content, and wherein inserting the replacement value comprises inserting the replacement value corresponding to one find criteria that matched the field content.

13. The article of manufacture of claim 11, wherein the find criteria for at least one rule definition comprises an upper bound and lower bound, and wherein searching comprises searching for any fields that have values within the upper and lower bounds.

14. The article of manufacture of claim 13, wherein searching comprises searching for any fields that have values outside of one of the upper and lower bounds.

15. The article of manufacture of claim 11, wherein each rule definition includes a row clean flag, and wherein at least one rule definition has the row clean flag set, further comprising removing any row including a field matching the search criteria from the input table when the row clean flag is set.

16. A memory device including a command data structure, comprising:

an input data table name parameter indicating the input table subject to the clean operation; and

at least one rule definition, wherein each rule definition includes a find criteria, a replacement value, and an input data column in the input table, wherein, for each rule definition, the input data column is searched for any fields that match the find criteria, wherein, for each rule definition, it is determined whether the rule definition specifies an output table, and in response to determining that the rule definition does not specify an output table, the replacement value is directly inserted in the fields in the input data column that match the find criteria, wherein subsequent applications of additional rule definitions applied to the same input data column operate on replacement values inserted in the input data column in previously applied rule definitions, wherein the rule definition comprises a type of rule that is a member of the set of rules consisting of;

find and replace, discretization, and numeric clip, and wherein at least two rule definitions are comprised of different rule types.

17. The memory device of claim 16, wherein at least one rule definition further includes:

multiple find criteria and a corresponding replacement value for each find criteria, wherein the input data column is searched by applying each of the multiple find criteria to one field until one of;

(i) a match occurs and (ii) none of the multiple find criteria are found to match the field content, and wherein the replacement value corresponding to the matching find criteria is inserted into the field.

18. The memory device of claim 16, wherein the find criteria for at least one rule definition comprises an upper bound and lower bound, and wherein the input data column is searched for any fields that have values within the upper and lower bounds.

19. The memory device of claim 18, wherein the input data column is searched for any fields that have values outside of one of the upper and lower bounds.

20. The memory device of claim 16, wherein each rule definition includes a row clean flag, and wherein at least one rule definition has the row clean flag set, and wherein any row including a field matching the search criteria is removed from the input table when the row clean flag is set.