Method and system for including data quality in data streams

• US 20080263062A1
• Filed: 04/20/2007
• Published: 10/23/2008
• Est. Priority Date: 04/20/2007
• Status: Active Grant

• Current Assignee: SAP SE
• Original Assignee: SAP AG (SAP SE)
• Inventors: Hackenbroich, Gregor, Klein, Anja, Do, Hong-Hai
• Granted Patent: US 7,676,522 B2
• Time in Patent Office: Days
• US Class Current: 707/100
• CPC Class Codes: H04L 41/5003 Managing SLA; Interaction between SLA and QoS
• CPC Class Codes: H04L 41/5009 Determining service level performance parameters or violations of service level contracts, e.g. violations of agreed response time or mean time between failures [MTBF]
• CPC Class Codes: H04L 43/022 by sampling
• CPC Class Codes: H04L 43/106 using time related information in packets, e.g. by adding timestamps

Abstract

A method and system are described for including data quality in data streams. An example method may include obtaining a first group of data items, each data item including one or more data attribute values. A first group of data quality items may be determined, each data quality item including one or more data quality attribute values associated with one of the data items of the first group. A first aggregated data quality value may be determined based on the first group of data quality items. A first data stream interval including the first group of data items and the first aggregated data quality value may be output.

Claims

1. A system comprising:

a data stream manager including;

a data manager configured to obtain a first group of data items, each data item including one or more data attribute values;

a data quality manager configured to determine a first group of data quality items, each data quality item including one or more data quality attribute values associated with one of the data items of the first group,

an aggregation manager configured to determine a first aggregated data quality value based on the first group of data quality items, and

a stream interval manager configured to output a first data stream interval including the first group of data items and the first aggregated data quality value.

2. The system of claim 1, wherein the stream interval manager is configured to:

output a first jumping window including the first data stream interval including the first group of data items and the first aggregated data quality value.

3. The system of claim 1, wherein:

the data manager is configured to obtain a second group of data items, each data item including one or more data attribute values,

the data quality manager is configured to determine a second group of data quality items, each data quality item including one or more data quality attribute values associated with one of the data items of the second group,

the aggregation manager is configured to determine a second aggregated data quality value based on the second group of data quality items, and

the stream interval manager is configured to output a second data stream interval including the second group of data items and the second aggregated data quality value.

4. The system of claim 3, wherein:

the stream interval manager is configured to output a first jumping window including the first data stream interval including the first group of data items and the first aggregated data quality value,

output a second jumping window including the second data stream interval including the second group of data items and the second aggregated data quality value, and

wherein the first jumping window and the second jumping window are nonoverlapping, and

wherein the data manager is configured to obtain the second group of data items immediately after the first group of data items is obtained.

5. The system of claim 3, wherein:

the data manager is configured to obtain the second group of data items immediately after the first group of data items is obtained, and wherein the first group of data items and the second group of data items are nonoverlapping.

6. The system of claim 1, wherein the first group of data items includes an ordering of the data items of the first group.

7. The system of claim 6, wherein the ordering of the data items of the first group is based on an ordering of generation of the data attribute values included in the data items of the first group.

8. The system of claim 1, wherein the first group of data items includes attribute values associated with data sources.

9. The system of claim 1, wherein the first group of data items includes attribute values associated with sensor measurements.

10. The system of claim 1, wherein the first group of data items includes attribute values representing sensor measurement data, wherein sensor measurements include one or more of:

a timestamp value,

a pressure value,

a temperature value,

a light intensity value,

a motion detection value,

an air quality value,

a location detection value, or

a humidity detection value.

11. A method comprising:

obtaining a first group of data items, each data item including one or more data attribute values;

determining a first group of data quality items, each data quality item including one or more data quality attribute values associated with one of the data items of the first group;

determining a first aggregated data quality value based on the first group of data quality items; and

outputting a first data stream interval including the first group of data items and the first aggregated data quality value.

12. The method of claim 11, wherein outputting the first data stream interval comprises:

outputting a first jumping window including the first data stream interval including the first group of data items and the first aggregated data quality value.

13. The method of claim 11, further comprising:

obtaining a second group of data items, each data item including one or more data attribute values;

determining a second group of data quality items, each data quality item including one or more data quality attribute values associated with one of the data items of the second group;

determining a second aggregated data quality value based on the second group of data quality items; and

outputting a second data stream interval including the second group of data items and the second aggregated data quality value.

14. The method of claim 13, wherein:

outputting the first data stream interval comprises outputting a first jumping window including the first data stream interval including the first group of data items and the first aggregated data quality value,

outputting the second data stream interval comprises outputting a second jumping window including the second data stream interval including the second group of data items and the second aggregated data quality value, and

wherein the first jumping window and the second jumping window are nonoverlapping, and

wherein the second group of data items is obtained immediately after the first group of data items is obtained.

15. The method of claim 13, wherein the second group of data items is obtained immediately after the first group of data items is obtained, and wherein the first group of data items and the second group of data items are nonoverlapping.

16. The method of claim 11, wherein the first group of data items includes an ordering of the data items of the first group.

17. The method of claim 16, wherein the ordering of the data items of the first group is based on an ordering of generation of the data attribute values included in the data items of the first group.

18. The method of claim 11, wherein the first group of data items includes attribute values associated with data sources.

19. The method of claim 11, wherein the first group of data items includes attribute values associated with sensor measurements.

20. The method of claim 11, wherein the first group of data items includes attribute values representing sensor measurement data, wherein sensor measurements include one or more of:

a timestamp value,

a pressure value,

a temperature value,

a light intensity value,

a motion detection value,

an air quality value,

a location detection value, or

a humidity detection value.

21. The method of claim 11, wherein determining the first group of data quality items comprises:

determining the first group of data quality items, each data quality item including data quality attribute values including one or more of a completeness value or an accuracy value.

22. The method of claim 11, wherein determining the first aggregated data quality value comprises:

determining a first aggregated data quality value based on an application dependent aggregation function that includes at least one of;

determining a linear average value of data quality attribute values of the first group of data quality items,

determining a squared average value of data quality attribute values of the first group of data quality items,

determining a weighted squared average value of data quality attribute values of the first group of data quality items,

determining a minimum value of data quality attribute values of the first group of data quality items, or

determining a maximum value of data quality attribute values of the first group of data quality items.

23. A computer program product being tangibly embodied on a computer-readable medium and being configured to cause a data processing apparatus to:

obtain a first group of data items, each data item including one or more data attribute values;

determine a first group of data quality items, each data quality item including one or more data quality attribute values associated with one of the data items of the first group;

determine a first aggregated data quality value based on the first group of data quality items; and

output a first data stream interval including the first group of data items and the first aggregated data quality value.

24. The computer program product of claim 23, further configured to cause the data processing apparatus to:

output a first jumping window including the first data stream interval including the first group of data items and the first aggregated data quality value.

25. The computer program product of claim 24, further configured to cause the data processing apparatus to:

determine a first aggregated data quality value based on an application dependent aggregation function that includes at least one of;

determining a linear average value of data quality attribute values of the first group of data quality items,

determining a squared average value of data quality attribute values of the first group of data quality items,

determining a weighted squared average value of data quality attribute values of the first group of data quality items,

determining a minimum value of data quality attribute values of the first group of data quality items, or

determining a maximum value of data quality attribute values of the first group of data quality items.