Method and system of measuring data quality

• US 7,251,578 B1
• Filed: 03/10/2006
• Issued: 07/31/2007
• Est. Priority Date: 03/10/2006
• Status: Active Grant

• Current Assignee: R2 Solutions LLC (Acacia Research Corporation)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Chen, Peiji, Lin, Long-Ji, Narayanareddy, Jagannatha
• Primary Examiner(s): Tsai; Carol S. W.
• Application Number:
  US11/373,701
• Time in Patent Office: 508 Days
• Field of Search: 702/81, 702/82, 702/84, 702/182, 702/185, 702/186, 702/127, 709/230, 709/231, 709/223, 709/205, 709/232, 709/233
• US Class Current: 702/127
• CPC Class Codes: H04L 1/20 using signal quality detector

Abstract

Data quality measurement is provided for use in a data processing stream, which comprises at least one upstream data processing system and at least one downstream data processing system. An input alert component can be used to provide a measurement of data prior to its input to a data processing system (e.g., a downstream data processing system or an upstream data processing system). An output alert component can be used to provide a measurement on data output by a data processing system. A self-consistency component can be used to measure consistency between items of input, or output data. An end-to-end component can be used to measure data quality using data items from both input data and output data. These components can be used in some combination, or independent of the other, and in any order. In addition, the data quality measurements can be performed separate from that processing performed by either the upstream or downstream processing system.

Claims

1. A data quality measurement method for use in a data processing stream comprising at least one upstream data processing system, which outputs data for use by at least one downstream data processing system, the method comprising:

validating data processed by an upstream data processing system prior to processing of the data by a data processing system downstream of the upstream data processing system, such that at least one property of the upstream data is validated; and

performing validation on data processed by the downstream data processing system, comprising;

validating at least one data item from the processed data using a predefined relationship between the at least one data item from the processed data and at least one data item from the upstream data; and

validating two or more data items from the processed data using a predefined relationship between the two or more data items.

2. The method of claim 1, further comprising:

validating the data processed by the downstream data processing system prior to forwarding the processed data to another downstream data processing system, such that at least one property of the processed data is validated.

3. The method of anyone of claims 1 and 2, wherein at least one of the validating steps uses one or more rules selected from a rule set.

4. The method of claim 1, wherein the validating data processed by an upstream data processing system uses benchmark information which corresponds to previous upstream data, the step further comprising:

determining a deviation between a property of the upstream data and a corresponding property from the previous upstream data; and

determining whether the determined deviation is acceptable using a predefined threshold corresponding to the upstream data property.

5. The method of claim 4, wherein in a case that the determined deviation is unacceptable, the method further comprises the step of:

generating an alert as notification that the determined deviation exceeds the predefined threshold.

6. The method of claim 4, wherein the predefined threshold is selected from multiple predefined thresholds.

7. The method of claim 6, wherein the multiple predefined thresholds comprise a first and a second threshold, wherein the first threshold is selected in a case that either the upstream data or the previous upstream data is associated with a holiday, and wherein the second threshold is selected in a case that neither the upstream data nor the previous upstream data is associated with a holiday.

8. The method of claim 1, wherein the at least one property of the upstream data comprises a size property, and wherein the benchmark comprises a corresponding size property of the previous upstream data.

9. The method of claim 1, wherein the at least one property of the upstream data comprises a value property, and wherein the benchmark comprises a corresponding value of the previous upstream data.

11. The method of claim 1, wherein one or more of the validating steps use a rule set which comprises at least one measurement rule, which defines a measurement to be performed.

12. The method of claim 1, wherein the benchmark corresponds to the at least one property of previous upstream data, the step of performing the pre-processing data quality measurement comprises:

determining a deviation between the at least one property of the upstream data and the at least one property of previous upstream data; and

determining whether the determined deviation is acceptable using a predefined threshold corresponding to the at least one property of the upstream data.

13. The method of claim 3, wherein in a case that the determined deviation is unacceptable, the method further comprises the step of:

generating an alert as notification that the determined deviation exceeds the predefined threshold.

14. The method of claim 3, wherein the predefined threshold is selected from multiple predefined thresholds.

15. The method of claim 5, wherein the multiple predefined thresholds comprise a first and a second threshold, wherein the first threshold is selected in a case that either the upstream data or the previous upstream data are associated with a holiday, and wherein the second threshold is selected in a case that neither the upstream data nor the previous upstream data are associated with a holiday.

16. The method of claim 1, wherein the at least one property of the upstream data comprises data size, and wherein the benchmark comprises a data size of previous upstream data corresponding to the upstream data.

17. The method of claim 1, wherein the at least one property of upstream data comprises a value of the upstream data, and wherein the benchmark comprises a value of previous upstream data corresponding to the upstream data.

10. A data quality measurement method for use in a data processing stream comprising at least one upstream data processing system, which outputs data for use by at least one downstream data processing system, comprises the steps of:

performing a pre-processing data quality measurement on upstream data comprising comparing at least one property of the upstream data to a benchmark corresponding to the at least one property of the upstream data;

performing a processed data quality measurement on data resulting from processing the upstream data, the processed data quality measurement comprising;

comparing at least one property of the resulting data with a benchmark corresponding to the at least one property of the resulting data;

validating at least one resulting data item using a predefined relationship between the at least one resulting data item and at least one upstream data item; and

validating two or more resulting data items using a predefined relationship between the two or more resulting data items.

18. A data quality measurement device for use in a data processing stream, the data processing stream comprising at least one upstream data processing system, which outputs data for use by at least one downstream data processing system, the method comprising:

an alert module configured to validate data processed by an upstream data processing system prior to processing of the data by a data processing system downstream of the upstream data processing system, such that at least one property of the upstream data is validated;

an end-to-end module configured validate at least one data item from data processed by the downstream data processing system, the processed data using a predefined relationship between the at least one data item from the processed data and at least one data item from the upstream data; and

a self-consistency module configured to validate two or more data items from the processed data using a predefined relationship between the two or more data items.

19. The device of claim 18, further comprising:

a second alert module configured to validate the data processed by the downstream data processing system prior to forwarding the processed data to another downstream data processing system, such that at least one property of the processed data is validated.

20. The device of any one of claims 18 and 19, wherein at least one of the modules is configured to use one or more rules selected from a rule set.

21. The device of claim 18, wherein the alert module configured to validate data processed by an upstream data processing system uses benchmark information which corresponds to previous upstream data, the alert module further comprising:

a first comparator configured to determine a deviation between a property of the upstream data and a corresponding property from the previous upstream data; and

a second comparator configured to determine whether the determined deviation is acceptable using a predefined threshold corresponding to the upstream data property.

22. The device of claim 21, further comprising:

an alert generator configured to generate an alert as notification that the determined deviation exceeds the predefined threshold.

23. The device of claim 22, further comprising:

a threshold selector configured to select the predefined threshold from multiple predefined thresholds.

24. The device of claim 23, wherein the multiple predefined thresholds comprise a first and a second threshold, wherein the threshold selector is configured to select the first threshold in a case that either the upstream data or the previous upstream data is associated with a holiday, and the threshold selector is configured to select the second threshold in a case that neither the upstream data nor the previous upstream data is associated with a holiday.

25. The device of claim 18, wherein the at least one property of the upstream data comprises a size property, and wherein the benchmark comprises a corresponding size property of the previous upstream data.

26. The device of claim 18, wherein the at least one property of the upstream data comprises a value property, and wherein the benchmark comprises a corresponding value of the previous upstream data.