Creating and processing a data rule for data quality

• US 8,949,166 B2
• Filed: 08/25/2011
• Issued: 02/03/2015
• Est. Priority Date: 12/16/2010
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Nelke, Sebastian, Oberhofer, Martin, Saillet, Yannick, Seifert, Jens
• Primary Examiner(s): CHEN, ALAN S
• Application Number:
  US13/217,389
• Publication Number:
  US 20120158625A1
• Time in Patent Office: 1,258 Days
• US Class Current: 706/47
• CPC Class Codes: G06F 16/20 of structured data, e.g. relational data
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06N 5/025 Extracting rules from data

Abstract

A data rule is created and processed by receiving an expression defining a logic of a rule and at least one logical variable, creating a rule definition including the expression and the at least one logical variable for binding each logical variable of the rule with at least one column, associating a characteristic enabling comparison of columns with a first logical variable of the rule definition, and storing the characteristic as part of the rule definition.

Claims

1. A computer-implemented method of creating and processing a data rule using an electronic computing device, the method comprising:

receiving an expression defining a logic of a rule and at least one logical variable;

creating a rule definition including the expression and the at least one logical variable for binding each logical variable of the rule with at least one column;

associating a characteristic enabling comparison of columns with a logical variable of the rule definition; and

storing the characteristic as part of the rule definition.

2. The method according to claim 1, wherein the associating comprises:

receiving information about the binding between the logical variable and a column; and

reading the characteristic from metadata associated with the column.

3. The method according to claim 1, wherein the associating is achieved by a user and comprises providing the characteristic enabling the comparison of columns.

4. The method according to claim 1, further comprising:

receiving information about a binding between the logical variable and a column; and

validating the binding based on comparing a characteristic of the column with the stored characteristic.

5. The method according to claim 1, further comprising:

checking whether a column can be found that has a characteristic similar to the characteristic of the rule definition;

handling a combination of the column and the rule definition as a rule candidate;

testing the rule candidate on a data sample;

removing the rule candidate from a list of rule candidates when the number of failed records exceeds a predefined threshold in the validation step; and

presenting the list of rule candidates to the user for review after the rule candidates have been tested.

6. The method according to claim 1, further comprising

generating a data rule based on the rule definition;

and

selectively applying the data rule to records in a table comprising a column

depending on whether values in the column match the characteristic.

7. A system for creating and processing a data rule comprising:

a processor configured to perform the operations of;

providing a user interface to manage I/O functions of the system, including a rule editor configured to create a plurality of rule definitions including an expression defining a logic of a rule and at least one logical variable for binding each logical variable of the rule with at least one column;

storing within a rule repository the plurality of rule definitions; and

providing a data processing engine in electronic communication with the user interface and the rule repository to associate a characteristic enabling comparison of columns with a logical variable of the rule definition, and to store the characteristic as part of the rule definition in the rule repository.

8. The system according to claim 7, wherein the processor is further configured to control the data processing engine to receive information about the binding between the logical variable and a column from the user interface, and the processor is further configured to control the data processing engine to read the characteristic from meta-data associated with the column.

9. The system according to claim 7, wherein the processor is further configured to control the data processing engine to receive the characteristic enabling the comparison of columns from the user interface.

10. The system according to claim 7, wherein the processor is further configured to control the data processing engine to receive information about a binding between the logical variable and a column from the user interface, and the processor is further configured to control the data processing engine to validate the binding based on comparing characteristic of the column with the stored characteristic.

11. The system according to claim 7, wherein the processor is further configured to control the data processing engine to check whether a column can be found that has a characteristic similar to the characteristic of the rule definition, to handle a combination of the column and the rule definition as a rule candidate, to test the rule candidate on a data sample, to remove the rule candidate from a list of rule candidates when the number of failed records exceeds a predefined threshold in the validation step, and to present the list of rule candidates to the user interface for review after the rule candidates have been tested.

12. The system according to claim 7, wherein the processor is further configured to control the data processing engine to generate a data rule based on the rule definition, and to selectively apply the data rule to records in a table comprising a column depending on whether values in the column match the characteristic.

13. A computer program product for creating and processing a data rule comprising:

a computer readable storage device having computer-readable program code embodied therewith, the computer readable program code configured to perform operations of;

receiving an expression defining a logic of a rule and at least one logical variable;

creating a rule definition including the expression and the at least one logical variable for binding each logical variable of the rule with at least one column;

associating a characteristic enabling comparison of columns with a logical variable of the rule definition; and

storing the characteristic as part of the rule definition.

14. The computer program product of claim 13, wherein the associating comprises:

receiving information about the binding between the logical variable and a column; and

reading the characteristic from metadata associated with the column.

15. The computer program product of claim 13, wherein the associating is performed by providing the characteristic enabling the comparison of columns.

16. The computer program product of claim 13, wherein the operations performed by the computer readable program code further comprise:

receiving information about a binding between the logical variable and a column; and

validating the binding based on comparing characteristic of the column with the stored characteristic.

17. The computer program product of claim 13, wherein the operations performed by the computer readable program code further comprise:

checking whether a column can be found that has a characteristic similar to the characteristic of the rule definition;

handling a combination of the column and the rule definition as a rule candidate;

testing the rule candidate on a data sample;

removing the rule candidate from a list of rule candidates when the number of failed records exceeds a predefined threshold in the validation step; and

presenting the list of rule candidates to the user for review after the rule candidates have been tested.

18. The computer program product of claim 13, wherein the operations performed by the computer readable program code further comprise:

generating a data rule based on the rule definition; and

selectively applying the data rule to records in a table comprising a column depending upon whether values in the column match the characteristic.