DATA QUALITY MEASUREMENT FOR ETL PROCESSES

• US 20080195430A1
• Filed: 02/12/2007
• Published: 08/14/2008
• Est. Priority Date: 02/12/2007
• Status: Abandoned Application

• Current Assignee: Oath Inc. (Verizon Communications Inc.)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Rustagi, Amit
• Application Number:
  US11/673,690
• Publication Number:
  US 20080195430A1
• Time in Patent Office: Days
• US Class Current: 705/7.41
• CPC Class Codes: G06Q 10/06395 Quality analysis or management
• CPC Class Codes: G06Q 10/10 Office automation; Time management

Abstract

Techniques for maintaining data quality of transformed data generated using an Extract-Transform-Load (ETL) process and stored in at least one data warehouse, the method comprising generating a quality metric for each of a plurality of units of the transformed data with reference to at least one data quality measurement rule, the quality metric for each unit of the transformed data representing a validity measure defined by the corresponding data quality measurement rule; and generating a report organizing the quality metrics for selected units of the transformed data.

Claims

1. A computer-implemented method for maintaining data quality of transformed data generated using an Extract-Transform-Load (ETL) process and stored in at least one data warehouse, the method comprising:

generating a quality metric for each of a plurality of units of the transformed data with reference to at least one data quality measurement rule, the quality metric for each unit of the transformed data representing a validity measure defined by the corresponding data quality measurement rule; and

generating a report organizing the quality metrics for selected units of the transformed data.

2. A computer-implemented method, as recited in claim 1, further comprising:

defining the at least one data quality measurement rule; and

storing the at least one data quality measurement rule in a repository.

3. A computer-implemented method, as recited in claim 2, further comprising:

updating the at least one data quality measurement rule stored in the repository.

4. A computer-implemented method, as recited in claim 2, further comprising:

after generating the quality metrics, storing the quality metrics in the repository; and

sending the report to a user.

5. A computer-implemented method, as recited in claim 1, further comprising:

alerting a user when at least one quality metric is not within a corresponding error tolerance defined by the corresponding data quality measurement rule.

6. A computer-implemented method, as recited in claim 1, further comprising:

generating the quality metrics for the units of the transformed data on a periodic basis; and

generating the report on the periodic basis,

wherein the periodic basis corresponds to one of a day, a week, or a month.

7. The computer-implemented method, as recited in claim 1, wherein the quality metric is one selected from a group consisting of a Boolean value indicating good and bad, and a rating number indicating a quality level.

8. The computer-implemented method, as recited in claim 1, further comprising:

performing business analysis on selected units of the transformed data for which the corresponding quality metrics indicate that those selected units of the transformed data contain no error.

9. A system for maintaining data quality of transformed data generated using an ETL process, comprising:

at least one data warehouse configured to store the transformed data;

a repository linked with each of the at least one data warehouse configured to store at least one data quality measurement rule and quality metrics, wherein each quality metric represents a validity measure defined by the corresponding data quality measurement rule for each of a plurality of units of the transformed data; and

a reporting mechanism linked with the repository configured to generate a report organizing the at least one quality metric for the transformed data.

10. A system, as recited in claim 9, further comprising:

an alerting mechanism linked with the repository configured to alert a user when at least one quality metric is not within a corresponding error tolerance defined by the corresponding data quality measurement rule,

wherein the at least one data quality measurement rule comprises at least one standard measure and at least one corresponding error tolerance for the transformed data.

11. A system, as recited in claim 9, wherein

there is one and only one repository in the system,

the repository is an SQL-based database, and

the at least one data quality measurement rule and the quality metrics are expressed in SQL format.

12. A system, as recited in claim 9, wherein the at least one data quality measurement rule comprises:

at least one daily data quality measurement rule used for determining daily quality metrics for a plurality of units of transformed daily data on a daily basis;

at least one weekly data quality measurement rule used for determining weekly quality for a plurality of units of transformed weekly data on a weekly basis; and

at least one monthly data quality measurement rule used for determining monthly quality for a plurality of units of transformed monthly data on a monthly basis.

13. A computer program product for maintaining data quality of transformed data generated using an Extract-Transform-Load (ETL) process and stored in at least one data warehouse, the computer program product comprising a computer-readable medium having a plurality of computer program instructions stored therein, which are operable to cause at least one computer device to:

generate a quality metric for each of a plurality of units of the transformed data with reference to at least one data quality measurement rule, the quality metric for each unit of the transformed data representing a validity measure defined by the corresponding data quality measurement rule; and

generate a report organizing the quality metrics for selected units of the transformed data.

14. A computer program product, as recited in claim 13, further comprising computer instructions to:

define the at least one data quality measurement rule; and

store the at least one data quality measurement rule in a repository.

15. A computer program product, as recited in claim 14, further comprising computer instructions to:

update the at least one data quality measurement rule stored in the repository.

16. A computer program product, as recited in claim 14, further comprising computer instructions to:

after generating the quality metrics, store the quality metrics in the repository; and

send the report to a user.

17. A computer program product, as recited in claim 13, further comprising computer instructions to:

alert a user when at least one quality metric is not within a corresponding error tolerance defined by the corresponding data quality measurement rule.

18. A computer program product, as recited in claim 13, further comprising computer instructions to:

generate the quality metrics for the units of the transformed data on a periodic basis; and

generate the report on the periodic basis,

wherein the periodic basis corresponds to one of a day, a week, or a month.

19. A computer program product, as recited in claim 13, wherein the quality metric is one selected from a group consisting of a Boolean value indicating good and bad, and a rating number indicating a quality level.

20. A computer program product, as recited in claim 13, further comprising computer instructions to:

performing business analysis on selected units of the transformed data for which the corresponding quality metrics indicate that those selected units of the transformed data contain no error.