Model-based data pipeline system optimization
First Claim
1. A computer-implemented method for optimizing a data pipeline system, the method comprising:
- processing a data pipeline configuration manifest to generate a framework of the data pipeline system and a data flow logic package of the data pipeline system, the data pipeline configuration manifest comprising an object-oriented metadata model of the data pipeline system;
monitoring performance of the data pipeline system during execution of the data flow logic package to obtain a performance metric for the data pipeline system; and
modifying, with a processor coupled to the data pipeline system, the framework of the data pipeline system based on the data pipeline configuration manifest and the performance metric.
2 Assignments
0 Petitions
Accused Products
Abstract
A computer-implemented method for optimizing a data pipeline system includes processing a data pipeline configuration manifest to generate a framework of the data pipeline system and a data flow logic package of the data pipeline system. The data pipeline configuration manifest includes an object-oriented metadata model of the data pipeline system. The computer-implemented method further includes monitoring performance of the data pipeline system during execution of the data flow logic package to obtain a performance metric for the data pipeline system, and modifying, with a processor, the framework of the data pipeline system based on the data pipeline configuration manifest and the performance metric.
-
Citations
20 Claims
-
1. A computer-implemented method for optimizing a data pipeline system, the method comprising:
-
processing a data pipeline configuration manifest to generate a framework of the data pipeline system and a data flow logic package of the data pipeline system, the data pipeline configuration manifest comprising an object-oriented metadata model of the data pipeline system; monitoring performance of the data pipeline system during execution of the data flow logic package to obtain a performance metric for the data pipeline system; and modifying, with a processor coupled to the data pipeline system, the framework of the data pipeline system based on the data pipeline configuration manifest and the performance metric. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13)
-
-
14. A system for configuring a data pipeline system, the system comprising:
-
a processor coupled to the data pipeline system; a memory coupled with the processor; wherein the memory comprises computer-executable instructions configured for execution by the processor, the computer-executable instructions comprising code to; process a data pipeline configuration manifest to generate a physical mapping and pipeline metadata for the data pipeline system, the data pipeline configuration manifest comprising an object-oriented metadata model of the data pipeline system; generate a data flow logic package based on the pipeline metadata; obtain a performance metric reflective of performance of the data pipeline system as configured in accordance with the physical mapping and during execution of the data flow logic package; and modify the physical mapping based on the data pipeline configuration manifest and the performance metric. - View Dependent Claims (15, 16, 17, 18)
-
-
19. A system comprising:
-
a data pipeline system comprising computer storage media and a set of data pipelines, each data pipeline comprising a source data store, a destination data store, and data transformation logic coupling the source data store and the destination data store; a configuration system comprising a processing unit, coupled with the data pipeline system and operative to configure the set of data pipelines based on a data pipeline configuration manifest and usage data indicative of use of the set of data pipelines, the data pipeline configuration manifest comprising an object-oriented metadata model of the data pipeline system; wherein the configuration system comprises a manifest compiler configured to process the data pipeline configuration manifest to generate a physical mapping of, and pipeline metadata for, the data pipeline system, the physical mapping specifying a framework for the set of data pipelines; wherein the configuration system further comprises a metadata transformer configured to generate a respective data flow logic package for each data pipeline of the set of data pipelines based on the pipeline metadata; and wherein the configuration system is configured to update the usage data to reflect performance of the data pipeline system during execution of the respective data flow logic package for a respective one of the set of data pipelines and further configured to direct the manifest compiler to re-process the data pipeline configuration manifest to update the physical mapping based on the updated usage data to optimize the set of data pipelines based on the usage data. - View Dependent Claims (20)
-
Specification