METHOD AND SYSTEM FOR ACCELERATED DATA QUALITY ENHANCEMENT

• US 20110138312A1
• Filed: 05/13/2010
• Published: 06/09/2011
• Est. Priority Date: 12/07/2009
• Status: Active Grant

• Current Assignee: Accenture Global Services Limited (Accenture PLC)
• Original Assignee: Accenture Global Services Limited (Accenture PLC)
• Inventors: PURI, Colin Anil, YEH, Peter Zei-Chan
• Granted Patent: US 8,700,577 B2
• Time in Patent Office: Days
• US Class Current: 715/771
• CPC Class Codes: G06F 16/24565 Triggers; Constraints
• CPC Class Codes: G06F 16/2465 Query processing support for facilitating data mining operations in structured databases

Abstract

Embodiments of the present invention solve the technical problem of identifying, collecting, and managing rules that improve poor quality data on enterprise initiatives ranging from data governance to business intelligence. In a specific embodiment of the present invention, a method is provided for producing data quality rules for a data set. A set of candidate conditional functional dependencies are generated comprised of candidate seeds of attributes that are within a certain degree of relatedness in the ontology of the data set. The candidate conditional functional dependencies are then applied to the data refined until they reach a quiescent state where they have not been refined even though the data they have been applied to has been stable. The resulting refined candidate conditional functional dependencies are the data enhancement rules for the data set and other related data sets. In another specific embodiment of the present invention, a computer system for the development of data quality rules is provided having a rule repository, a data quality rules discovery engine, and a user interface.

Claims

1. A computer-implemented method for producing data quality rules for a data set, comprising:

generating a set of candidate conditional functional dependencies based on a set of candidate seeds by using an ontology of said data set, said candidate seeds being comprised of a subset of attributes drawn from a set of the attributes of said data set that have a predetermined degree of separation in said ontology;

applying said candidate conditional functional dependencies individually to said data set to obtain a set of corresponding result values for said candidate conditional functional dependencies;

refining said candidate conditional functional dependencies individually and repeating said applying if said set of corresponding result values does not have a result signature that meets a predetermined expectation;

terminating said applying and refining of said candidate conditional functional dependencies individually when said candidate conditional functional dependencies individually reach a quiescent state; and

selecting a relevant set of said candidate conditional functional dependencies to be used as said data quality rules for said data set.

2. The computer-implemented method for producing data quality rules for a data set from claim 1, wherein said refining is achieved by eliminating a high entropy attribute of an individual one of said candidate conditional functional dependencies.

3. The computer-implemented method for producing data quality rules for a data set from claim 1, wherein:

a first number of conditions in each of said candidate conditional functional dependencies can be adjusted by a user prior to generating said set of candidate conditional functional dependencies; and

a second number of attributes in a candidate seed of said set of candidate seeds can be adjusted by a user prior to generating said set of candidate seeds.

4. The computer-implemented method for producing data quality rules for a data set from claim 1, wherein said predetermined expectation is set by a predetermined coverage estimate of a first portion of said data set that is covered by an individual one of said candidate conditional functional dependencies, and a predetermined error estimate of a second portion of said data set that will be erroneous.

5. The computer-implemented method for producing data quality rules for a data set from claim 4, wherein:

said set of candidate conditional functional dependencies has a predetermined number of said candidate conditional functional dependencies; and

said predetermined number of conditional functional dependencies can be adjusted by a user.

6. The computer-implemented method for producing data quality rules for a data set from claim 4, wherein said candidate conditional functional dependencies in said relevant set have a corresponding set of combined result signatures that have the best goodness of fit in terms of a maximum degree of coverage of said data set and a minimum proximity between a detected error rate and said predetermined error estimate.

7. The computer-implemented method for producing data quality rules for a data set from claim 6, further comprising:

ranking said candidate conditional functional dependencies in said relevant set according to an interestingness factor;

wherein

said interestingness factor increases for a particular one of said candidate conditional functional dependencies as a portion of said data set consisting of a data value on which said particular one of said candidate conditional functional dependencies is based on decreases.

8. The computer-implemented method for producing data quality rules for a data set from claim 4, wherein the size of a data segment of said data set that said candidate conditional functional dependencies are applied to during said applying is set by a predetermined scan period.

9. The computer-implemented method for producing data quality rules for a data set from claim 8, wherein said predetermined coverage estimate, said predetermined error estimate, and said predetermined scan period can be adjusted by a user.

10. The computer-implemented method for producing data quality rules for a data set from claim 4, wherein said quiescent state for a specific one of said candidate conditional functional dependencies when said specific one of said candidate conditional functional dependencies has been applied individually through said applying to a series of said data segments that, in total, contain an amount of data points equal in size to a predetermined window period without said refining altering said specific candidate conditional functional dependencies, and when said series of said data segments has contained stable data.

11. The computer-implemented method for producing data quality rules for a data set from claim 10, wherein said predetermined window period can be adjusted by a user.

12. A computer-implemented method for enhancing data quality, comprising the steps of:

generating a set of candidate conditional functional dependencies based on a set of candidate seeds by using an ontology of a data set, each of said candidate seeds being comprised of a subset of attributes drawn from a set of all the attributes of said data set that have a predetermined degree of separation in said ontology;

applying said candidate conditional functional dependencies individually to said data set to obtain a set of corresponding result values for each of said candidate conditional functional dependencies;

refining said candidate conditional functional dependencies individually and repeating said applying if said set of corresponding result values does not have a result signature that meets a pre-determined expectation;

terminating said applying and refining of said candidate conditional functional dependencies individually when said candidate conditional functional dependencies individually reach a quiescent state;

selecting a relevant set of said candidate conditional functional dependencies to be used as said data quality rules for said data set; and

enhancing the data quality of said data set by checking the data of said data set against said relevant set and screening said data if said data does not follow a rule contained in said relevant set.

13. The computer-implemented method for enhancing data quality from claim 12, further comprising continuing to apply said relevant set to enhance the data quality of a group of additional data sets that are related in content to said data set.

14. The computer-implemented method for enhancing data quality from claim 12, further comprising exporting said relevant set to one of a data quality product and an external data base management system.

15. The computer-implemented method for enhancing the data quality from claim 14, wherein said data quality product is one of TS Discovery, Informatica IDE/IDQ and Oracle Data Integrator.

16. A computer system for the development of data quality rules, comprising:

a rule repository for storing said data quality rules;

a user interface capable of receiving a data set, an ontology, and a set of rule generation parameters, and capable of outputting a set of of data quality rules;

a data quality rules discovery engine capable of receiving said data set, said ontology, and said set of rule generation parameters from said user interface, generating said set of data quality rules, and sending said set of data quality rules to said rule repository;

wherein said data quality rules discovery engine formulates a set of candidate conditional functional dependencies based on a set of candidate seeds by using said ontology, said candidate seeds being comprised of a subset of attributes that have a predetermined degree of separation in said ontology drawn from a set of all the attributes of said data set; and

wherein said data quality rules discovery engine refines said set of candidate conditional functional dependencies iteratively if they do not meet a predetermined expectation when applied to said data set, and terminates said refining when said set of conditional functional dependencies reach a quiescent state and become said data quality rules.

17. The computer system for the development of data quality rules from claim 16, said user interface further comprising:

a graphical user interface capable of displaying and receiving said rule generation parameters, an address of said data set, an address of a related data set, and an address of said ontology, and said set of data quality rules; and

wherein said rule generation parameters can be adjusted by the user through said graphical user interface.

18. The computer system for the development of data quality rules from claim 16, wherein said data quality rules discovery engine refines said candidate conditional functional dependencies by eliminating a high entropy attribute of said candidate conditional functional dependencies.

19. The computer system for the development of data quality rules from claim 16, further comprising a data exchanger plug-in capable of exporting a relevant set of said data quality rules to one of a data quality product and an external data base management system.

20. The computer system for the development of data quality rules from claim 19, wherein said data quality product is one of TS Discovery, Informatica IDE/IDQ and Oracle Data Integrator.