Employing abstract pipeline component connections to maintain data flow
First Claim
1. A pipeline system comprising a processor and a plurality of components, each component comprising computer executable instructions encoded upon computer readable media and being executable upon the processor, the components comprising:
- a plurality of metadata components;
a reference component, the reference component identifying reference data which is used in resolving metadata differences; and
a connector component, the connector component comprising three subcomponents, the three subcomponents consisting of (i) a receiver component, (ii) an execution component, and (iii) a transmission component,the receiver component acquiring data and metadata from a first metadata component of the plurality of metadata components and providing the acquired data and metadata to the execution component for processing,the execution component being communicatively coupled to one or more heuristic components, the one or more heuristic components providing functionality for resolving metadata mismatches,the transmission component receiving arbitrated data from the execution component for transmission to a downstream component,the connector component analyzing metadata associated with at least two metadata components of the plurality of metadata components,the connector component arbitrating metadata differences between the at least two metadata components by manipulation of the metadata associated with a first metadata component based at least in part on the execution component resolving metadata mismatches such that the metadata associated with the first metadata component is compatible with a schema associated with the at least one other metadata component,the execution component providing arbitrated data to the transmission component, andthe transmission component transmitting the arbitrated data to a downstream metadata component.
2 Assignments
0 Petitions
Accused Products
Abstract
The subject disclosure pertains to systems and methods for connecting pipeline components such as data flow pipeline components. Pipeline components are wrapped in metadata components that include the component as well as abstracted metadata in the form of input and output metadata descriptors. A connector can arbitrate metadata differences between pipeline components based on the metadata descriptors, thereby preventing upstream modifications from invalidating or breaking components downstream. Native or plug-in heuristic and/or related components can provide the functionality necessary to resolve metadata differences.
-
Citations
9 Claims
-
1. A pipeline system comprising a processor and a plurality of components, each component comprising computer executable instructions encoded upon computer readable media and being executable upon the processor, the components comprising:
-
a plurality of metadata components; a reference component, the reference component identifying reference data which is used in resolving metadata differences; and a connector component, the connector component comprising three subcomponents, the three subcomponents consisting of (i) a receiver component, (ii) an execution component, and (iii) a transmission component, the receiver component acquiring data and metadata from a first metadata component of the plurality of metadata components and providing the acquired data and metadata to the execution component for processing, the execution component being communicatively coupled to one or more heuristic components, the one or more heuristic components providing functionality for resolving metadata mismatches, the transmission component receiving arbitrated data from the execution component for transmission to a downstream component, the connector component analyzing metadata associated with at least two metadata components of the plurality of metadata components, the connector component arbitrating metadata differences between the at least two metadata components by manipulation of the metadata associated with a first metadata component based at least in part on the execution component resolving metadata mismatches such that the metadata associated with the first metadata component is compatible with a schema associated with the at least one other metadata component, the execution component providing arbitrated data to the transmission component, and the transmission component transmitting the arbitrated data to a downstream metadata component. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9)
-
Specification