Method and system for managing data quality

• US 20080263096A1
• Filed: 04/20/2007
• Published: 10/23/2008
• Est. Priority Date: 04/20/2007
• Status: Active Grant

• Current Assignee: SAP SE
• Original Assignee: SAP AG (SAP SE)
• Inventors: Klein, Anja, Hackenbroich, Gregor, Anke, Juergen, Do, Hong-Hai
• Granted Patent: US 7,676,523 B2
• Time in Patent Office: Days
• US Class Current: 707/104.1
• CPC Class Codes: G05B 23/0221 Preprocessing measurements, e.g. data collection rate adjustment; Standardization of measurements; Time series or signal analysis, e.g. frequency analysis or wavelets; Trustworthiness of measurements; Indexes therefor; Measurements using easily measured parameters to estimate parameters difficult to measure; Virtual sensor creation; De-noising; Sensor fusion; Unconventional preprocessing inherently present in specific fault detection methods like PCA-based methods

Abstract

A method and system are described for managing data quality. An example method may include obtaining a first data stream interval including a first group of data items and a first aggregated data quality value associated with a quality of obtaining the first group, each data item including data attribute values, each data quality item including data quality attribute values associated with one of the data items. The first aggregated data quality value, a first indicator associating the first aggregated data quality value with the first group, and the first group may be selected. The first group and the first indicator may be stored in a user table of a database. A data quality table associated with the user table may be determined based on an entry in a system table. The first aggregated data quality value and the first indicator may be stored in the data quality table.

Claims

1. A system comprising:

a data stream manager including;

a data receiver configured to obtain a first data stream interval including a first group of data items and a first aggregated data quality value associated with a quality of obtaining the first group of data items, each data item including one or more data attribute values, each data quality item including one or more data quality attribute values associated with one of the data items of the first group;

a data selection manager configured to select the first aggregated data quality value and a first indicator associating the first aggregated data quality value with the first group of data items, and to select the first group of data items;

a data storage manager configured to store the first group of data items and the first indicator in a user table associated with a relational database;

a system table manager configured to determine a data quality table associated with the user table based on an entry in a system table; and

a data quality storage manager configured to store the first aggregated data quality value and the first indicator in the determined data quality table.

2. The system of claim 1, wherein the first indicator includes a timestamp value indicating a time associated with a first data item collected for transmission in the first data stream interval.

3. The system of claim 1, wherein the first indicator includes a timestamp value indicating a beginning of a jumping window included in the first data stream interval.

4. The system of claim 1, wherein:

the data storage manager is configured to store the first group of data items and timestamps associated with each data item in the first group in the user table associated with a relational database, wherein the timestamps include the first indicator.

5. The system of claim 1, wherein the determined data quality table is associated with the user table based on a column identifier stored in a system table describing the determined data quality table and a column identifier stored in a system table describing tables including the user table and the determined data quality table associated with the relational database.

6. The system of claim 1, wherein the determined data quality table includes one or more columns configured to store attribute values associated with aggregated data quality values and a column identifier associated with an associated data item stored in the user table.

7. The system of claim 1, wherein:

the data receiver is configured to obtain a second data stream interval including a second group of data items and a second aggregated data quality value associated with a quality of obtaining the second group of data items, each data item including one or more data attribute values, each data quality item including one or more data quality attribute values associated with one of the data items of the second group;

the data selection manager is configured to select the second aggregated data quality value and a second indicator associating the second aggregated data quality value with the second group of data items and to select the second group of data items;

the data storage manager is configured to store the second group of data items and the second indicator in the user table;

the system table manager is configured to determine the data quality table associated with the user table based on an entry in a system table; and

the data quality storage manager is configured to store the second aggregated data quality value and the second indicator in the determined data quality table.

8. The system of claim 7, wherein:

the data receiver is configured to obtain a first jumping window including the first data stream interval including the first group of data items and the first aggregated data quality value, and a second jumping window including the second data stream interval including the second group of data items and the second aggregated data quality value, and

wherein the first jumping window and the second jumping window are nonoverlapping, and

wherein the second group of data items is obtained immediately after the first group of data items is obtained.

9. The system of claim 7, wherein the second group of data items is obtained immediately after the first group of data items is obtained, and wherein the first group of data items and the second group of data items are nonoverlapping.

10. The system of claim 1, wherein the first group of data items includes attribute values representing sensor measurement data, wherein sensor measurements include one or more of:

a timestamp value,

a pressure value,

a temperature value,

a light intensity value,

a motion detection value,

an air quality value,

a location detection value, or

a humidity detection value.

11. A method comprising:

obtaining a first data stream interval including a first group of data items and a first aggregated data quality value associated with a quality of obtaining the first group of data items, each data item including one or more data attribute values, each data quality item including one or more data quality attribute values associated with one of the data items of the first group;

selecting the first aggregated data quality value and a first indicator associating the first aggregated data quality value with the first group of data items;

selecting the first group of data items;

storing the first group of data items and the first indicator in a user table associated with a relational database;

determining a data quality table associated with the user table based on an entry in a system table; and

storing the first aggregated data quality value and the first indicator in the determined data quality table.

12. The method of claim 11, wherein the first indicator includes a timestamp value indicating a time associated with a first data item collected for transmission in the first data stream interval.

13. The method of claim 11, wherein the first indicator includes a timestamp value indicating a beginning of a jumping window included in the first data stream interval.

14. The method of claim 11, wherein storing the first group of data items and the first indicator in a user table associated with a relational database comprises storing the first group of data items and timestamps associated with each data item in the first group in the user table associated with a relational database, wherein the timestamps include the first indicator.

15. The method of claim 11, wherein the determined data quality table is associated with the user table based on a column identifier stored in a system table describing the determined data quality table and a column identifier stored in a system table describing tables including the user table and the determined data quality table associated with the relational database.

16. The method of claim 11, wherein the determined data quality table includes one or more columns configured to store attribute values associated with aggregated data quality values and a column identifier associated with an associated data item stored in the user table.

17. The method of claim 11, further comprising:

obtaining a second data stream interval including a second group of data items and a second aggregated data quality value associated with a quality of obtaining the second group of data items, each data item including one or more data attribute values, each data quality item including one or more data quality attribute values associated with one of the data items of the second group;

selecting the second aggregated data quality value and a second indicator associating the second aggregated data quality value with the second group of data items;

selecting the second group of data items;

storing the second group of data items and the second indicator in the user table;

determining the data quality table associated with the user table based on an entry in a system table; and

storing the second aggregated data quality value and the second indicator in the determined data quality table.

18. The method of claim 17, wherein:

obtaining the first data stream interval comprises obtaining a first jumping window including the first data stream interval including the first group of data items and the first aggregated data quality value,

obtaining the second data stream interval comprises obtaining a second jumping window including the second data stream interval including the second group of data items and the second aggregated data quality value, and

wherein the first jumping window and the second jumping window are nonoverlapping, and

wherein the second group of data items is obtained immediately after the first group of data items is obtained.

19. The method of claim 17, wherein the second group of data items is obtained immediately after the first group of data items is obtained, and wherein the first group of data items and the second group of data items are nonoverlapping.

20. The method of claim 11, wherein the first group of data items includes attribute values associated with data sources.

21. The method of claim 11, wherein the first group of data items includes attribute values associated with sensor measurements.

22. The method of claim 11, wherein the first group of data items includes attribute values representing sensor measurement data, wherein sensor measurements include one or more of:

a timestamp value,

a pressure value,

a temperature value,

a light intensity value,

a motion detection value,

an air quality value,

a location detection value, or

a humidity detection value.

23. A method comprising:

requesting a first group of data items from a relational database;

requesting a first aggregated data quality value associated with a quality of obtaining the first group of data items and a first indicator associating the first aggregated data quality value with the first group of data items;

retrieving the first group of data items from a user table associated with the relational database;

determining a data quality table associated with the user table based on an entry in a system table; and

retrieving the first aggregated data quality value and the first indicator from the determined data quality table.

24. A computer program product being tangibly embodied on a computer-readable medium and being configured to cause a data processing apparatus to:

obtain a first data stream interval including a first group of data items and a first aggregated data quality value associated with a quality of obtaining the first group of data items, each data item including one or more data attribute values, each data quality item including one or more data quality attribute values associated with one of the data items of the first group;

select the first aggregated data quality value and a first indicator associating the first aggregated data quality value with the first group of data items,

select the first group of data items;

store the first group of data items and the first indicator in a user table associated with a relational database;

determine a data quality table associated with the user table based on an entry in a system table; and

store the first aggregated data quality value and the first indicator in the determined data quality table.

25. The computer program product of claim 24, further configured to cause the data processing apparatus to:

store the first group of data items and timestamps associated with each data item in the first group in the user table associated with a relational database, wherein the timestamps include the first indicator.

26. The computer program product of claim 25, wherein the determined data quality table is associated with the user table based on a column identifier stored in a system table describing the determined data quality table and a column identifier stored in a system table describing tables including the user table and the determined data quality table associated with the relational database.