Method and system for validation of data extraction

• US 20080162999A1
• Filed: 12/27/2006
• Published: 07/03/2008
• Est. Priority Date: 12/27/2006
• Status: Active Grant

• Current Assignee: SAP SE
• Original Assignee: SAP AG (SAP SE)
• Inventors: Schlueter, Markus, Zimmerer, Peter Karlheinz, Fey, Harald
• Granted Patent: US 8,626,702 B2
• Time in Patent Office: Days
• US Class Current: 714/33
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses

Abstract

Validation of an extraction process from an operation system to an on-line analytics and processing (“OLAP”) system may be achieved utilizing a function module that reads data from a queue and outputs the data in a structured form. A second function module may be used to perform an existence and uniqueness check on the data to determine the existence and/or uniqueness of various data elements.

Claims

1. A system for validating an extraction process from an operational system to an OLAP (“

Online Analytics Processing”

) system, comprising;

a queue configured to store extracted data that the extraction process has extracted from the operational system, wherein the extracted data in the queue is stored based upon an associated data source; and

a processor, wherein the processor is adapted to;

determine at least one queue based upon a received data source;

read first data from the at least one queue as a function of at least a portion of the selected data source to generate a structured data element that is based upon the selected data source;

receive at least one identifier associated with the first data for validation of the structured data element;

determine at least one of an existence and uniqueness of each of the at least one identifier in the structured data element, and

report a validation result for the structured data element, based upon the existence and uniqueness determination.

2. The system according to claim 1, wherein the data source comprises a function module and an extraction structure.

3. The system according to claim 2, wherein the extraction structure comprises a plurality of structured data fields.

4. The system according to claim 2, wherein the function module is configured to process at least one field generated by a data generator in the operational system to generate at least one transformed field.

5. The system according to claim 4, wherein the function module is configured to cause the aggregation of a plurality of fields into a single field.

6. The system according to claim 4, wherein the function module is configured to execute an identity operation.

7. The system according to claim 4, wherein the function module is configured to causes the filtering of data from at least one field generated by a data generator.

8. A method for validating an extraction process from an operational system to an OLAP system, the method comprising:

determining at least one queue associated with a received data source;

reading first data from the at least one a queue as a function of at least a portion of the selected data source to generate a structured data element that is based upon the selected data source;

receiving at least one identifier associated with the first data for validation of the structured data element;

determining at least one of an existence and uniqueness of each of the at least one identifier in the structured data element, and

reporting a validation result for the structured data element, based upon the existence and uniqueness determination.

9. The method according to claim 8, wherein determining the selected queue comprises determining a function module and an extraction structure associated with the selected data source.

10. The method according to claim 9, wherein the extraction structure comprises a plurality of structured data fields.

11. The method according to claim 9, wherein the function module is configured to process at least one field generated by a data generator in the operational system to generate at least one transformed field.

12. The method according to claim 11, wherein the function module is configured to cause the aggregation of a plurality of fields into a single field.

13. The method according to claim 11, wherein the function module comprises an identity operation.

14. The method according to claim 11, wherein the function module is configured to cause the filtering of data from at least one field generated by a data generator.

15. A validation interface configured to validate an extraction process from an operational system to an OLAP system, the validation interface comprising:

a first function module, the first function module configured to receive a first input of a data source, and configured to read data from a queue to provide an output of data in the queue in a structured form based on the data source;

a second function module, the second function module configured to receive an input of the data source for the operational system and configured to provide an output of at least one queue associated with the data source; and

a third function module, the third function module configured to determine at least one of an existence and a uniqueness of selected data extracted by an extraction system performing the extraction process.

16. The validation interface according to claim 15, further comprising:

a first interface configured to invoke the first function module, the first interface characterized by a first input parameter specifying the data source;

a second interface configured to invoke the second function module, the second interface characterized by the first input parameter specifying the data source; and

,

a third interface configured to invoke the third function module, the third interface characterized by an input parameter specifying at least one UUID for validation.

17. The validation interfaces according to claim 15, wherein the data source includes at least one of a function module and an extraction structure.

18. The validation interface according to claim 15, wherein the third function checks for existence of at least one data item in a queue associated with the data source.

19. The validation interface according to claim 15, wherein the third function checks for uniqueness of at least one data item in a queue associated with the data source.

20. The validation interface according to claim 15, wherein the first function module is further configured to generate a structured table from the data read from the memory element, the structured table having a structure corresponding to the metadata for the data source.