Model-based data pipeline system optimization

• US 9,396,037 B2
• Filed: 02/27/2012
• Issued: 07/19/2016
• Est. Priority Date: 02/27/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Morsi, Magdi, Au, Wai Ho, Neematallah, Amgad, Xu, Will
• Primary Examiner(s): Spieler, William
• Application Number:
  US13/405,362
• Publication Number:
  US 20130227573A1
• Time in Patent Office: 1,604 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 2209/5022 Workload threshold
• CPC Class Codes: G06F 9/5083 Techniques for rebalancing the load in a distributed system

Abstract

A computer-implemented method for optimizing a data pipeline system includes processing a data pipeline configuration manifest to generate a framework of the data pipeline system and a data flow logic package of the data pipeline system. The data pipeline configuration manifest includes an object-oriented metadata model of the data pipeline system. The computer-implemented method further includes monitoring performance of the data pipeline system during execution of the data flow logic package to obtain a performance metric for the data pipeline system, and modifying, with a processor, the framework of the data pipeline system based on the data pipeline configuration manifest and the performance metric.

Claims

1. A computer-implemented method for optimizing a data pipeline system, the method comprising:

processing a data pipeline configuration manifest to generate a framework of the data pipeline system and a data flow logic package of the data pipeline system, the data pipeline configuration manifest comprising an object-oriented metadata model of the data pipeline system;

monitoring performance of the data pipeline system during execution of the data flow logic package to obtain a performance metric for the data pipeline system; and

modifying, with a processor coupled to the data pipeline system, the framework of the data pipeline system based on the data pipeline configuration manifest and the performance metric.

2. The computer-implemented method of claim 1, wherein the object-oriented metadata model comprises an entity-relationship schema of the data pipeline system.

3. The computer-implemented method of claim 1, wherein processing the data pipeline configuration manifest comprises parsing a list of object definitions in the data pipeline configuration manifest to generate a set of metadata entity objects of the framework of the data pipeline system and to further generate data source binding metadata reflective of relationships between respective metadata entity objects of the set of metadata entity objects.

4. The computer-implemented method of claim 1, wherein the framework of the data pipeline system comprises a physical mapping of the data pipeline system, generic data storage entities of the data pipeline system, and generic pipelines of the data pipeline system.

5. The computer-implemented method of claim 1, wherein the object-oriented metadata model comprises a container object definition specifying a model of a data store, the data store model comprising an entity object definition, the entity object definition specifying configuration parameters of a data table of the data store.

6. The computer-implemented method of claim 5, wherein the configuration parameters of the data table configuration establish a set of operational performance properties for an object entity instantiation.

7. The computer-implemented method of claim 5, wherein the container object definition comprises relationship metadata representative of an association between entity objects.

8. The computer-implemented method of claim 7, wherein the relationship metadata comprises pipeline property metadata.

9. The computer-implemented method of claim 1, wherein processing the data pipeline configuration manifest comprises:

generating generic pipeline metadata from the object-oriented metadata model; and

creating the data flow logic package based on the generic pipeline metadata.

10. The computer-implemented method of claim 1, wherein monitoring performance of the data pipeline system comprises updating usage mapping data reflective of use of the data pipeline system, wherein the performance metric is based on the updated usage mapping data.

11. The computer-implemented method of claim 10, wherein modifying the framework of the data pipeline system comprises:

generating the performance metric from the updated usage mapping data; and

generating, with the processor, the modified framework of the data pipeline system by re-processing the data pipeline configuration manifest based on the performance metric.

12. The computer-implemented method of claim 1, wherein modifying the framework of the data pipeline system is implemented during the execution of the data flow logic package.

13. The computer-implemented method of claim 1, wherein:

processing the data pipeline configuration manifest comprises parsing a list of object definitions in the data pipeline configuration manifest to generate a set of metadata objects for the framework of the data pipeline system;

each metadata object of the set of metadata objects specifies a version of the framework of the data pipeline system; and

modifying the framework of the data pipeline system comprises reverting the framework of the data pipeline system to a previous version if a validation of the data pipeline system fails.

14. A system for configuring a data pipeline system, the system comprising:

a processor coupled to the data pipeline system;

a memory coupled with the processor;

wherein the memory comprises computer-executable instructions configured for execution by the processor, the computer-executable instructions comprising code to;

process a data pipeline configuration manifest to generate a physical mapping and pipeline metadata for the data pipeline system, the data pipeline configuration manifest comprising an object-oriented metadata model of the data pipeline system;

generate a data flow logic package based on the pipeline metadata;

obtain a performance metric reflective of performance of the data pipeline system as configured in accordance with the physical mapping and during execution of the data flow logic package; and

modify the physical mapping based on the data pipeline configuration manifest and the performance metric.

15. The system of claim 14, wherein the object-oriented metadata model comprises an entity-relationship schema of the data pipeline system.

16. The system of claim 14, wherein the computer-executable instructions comprise further code to update usage mapping data reflective of use of the data pipeline system, wherein the performance metric is based on the updated usage mapping data.

17. The system of claim 14, wherein the computer-executable instructions comprise further code to generate the modified physical mapping of the data pipeline system by re-processing the data pipeline configuration manifest in accordance with the performance metric.

18. The system of claim 14, wherein the computer-executable instructions comprise further code to revert the physical mapping and the pipeline metadata to a previous version if a validation of the data pipeline system fails.

19. A system comprising:

a data pipeline system comprising computer storage media and a set of data pipelines, each data pipeline comprising a source data store, a destination data store, and data transformation logic coupling the source data store and the destination data store;

a configuration system comprising a processing unit, coupled with the data pipeline system and operative to configure the set of data pipelines based on a data pipeline configuration manifest and usage data indicative of use of the set of data pipelines, the data pipeline configuration manifest comprising an object-oriented metadata model of the data pipeline system;

wherein the configuration system comprises a manifest compiler configured to process the data pipeline configuration manifest to generate a physical mapping of, and pipeline metadata for, the data pipeline system, the physical mapping specifying a framework for the set of data pipelines;

wherein the configuration system further comprises a metadata transformer configured to generate a respective data flow logic package for each data pipeline of the set of data pipelines based on the pipeline metadata; and

wherein the configuration system is configured to update the usage data to reflect performance of the data pipeline system during execution of the respective data flow logic package for a respective one of the set of data pipelines and further configured to direct the manifest compiler to re-process the data pipeline configuration manifest to update the physical mapping based on the updated usage data to optimize the set of data pipelines based on the usage data.

20. The system of claim 19, wherein the configuration system is further configured to revert the physical mapping and the pipeline metadata to a previous version if a validation of the data pipeline system fails.