Employing abstract pipeline component connections to maintain data flow

• US 7,779,017 B2
• Filed: 01/12/2006
• Issued: 08/17/2010
• Est. Priority Date: 01/12/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: David, Matthew E., Haselden, J. Kirk
• Primary Examiner(s): Rones; Charles
• Assistant Examiner(s): MUELLER, KURT A
• Application Number:
  US11/330,842
• Publication Number:
  US 20070162444A1
• Time in Patent Office: 1,678 Days
• Field of Search: 707/101
• US Class Current: 707/756
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 9/4494 data driven

Abstract

The subject disclosure pertains to systems and methods for connecting pipeline components such as data flow pipeline components. Pipeline components are wrapped in metadata components that include the component as well as abstracted metadata in the form of input and output metadata descriptors. A connector can arbitrate metadata differences between pipeline components based on the metadata descriptors, thereby preventing upstream modifications from invalidating or breaking components downstream. Native or plug-in heuristic and/or related components can provide the functionality necessary to resolve metadata differences.

Claims

1. A pipeline system comprising a processor and a plurality of components, each component comprising computer executable instructions encoded upon computer readable media and being executable upon the processor, the components comprising:

a plurality of metadata components;

a reference component, the reference component identifying reference data which is used in resolving metadata differences; and

a connector component, the connector component comprising three subcomponents, the three subcomponents consisting of (i) a receiver component, (ii) an execution component, and (iii) a transmission component,

the receiver component acquiring data and metadata from a first metadata component of the plurality of metadata components and providing the acquired data and metadata to the execution component for processing,

the execution component being communicatively coupled to one or more heuristic components, the one or more heuristic components providing functionality for resolving metadata mismatches,

the transmission component receiving arbitrated data from the execution component for transmission to a downstream component,

the connector component analyzing metadata associated with at least two metadata components of the plurality of metadata components,

the connector component arbitrating metadata differences between the at least two metadata components by manipulation of the metadata associated with a first metadata component based at least in part on the execution component resolving metadata mismatches such that the metadata associated with the first metadata component is compatible with a schema associated with the at least one other metadata component,

the execution component providing arbitrated data to the transmission component, and

the transmission component transmitting the arbitrated data to a downstream metadata component.

2. The system of claim 1, wherein at least one of the plurality of metadata components comprises at least one input metadata descriptor and at least one output metadata descriptor.

3. The system of claim 1, wherein the execution component executes functionality provided by at least one of the one or more heuristic components.

4. The system of claim 1, wherein the receiver component acquires an input descriptor from the downstream metadata component.

5. The system of claim 3, further comprising an interface component to enable a user to modify a particular heuristic component.

6. The system of claim 5, further comprising a generation component communicatively coupled to the interface component that facilitates producing a new heuristic component.

7. The system of claim 5, further comprising one or more interface components to facilitate receipt of non-native or plug-in heuristic components.

8. The system of claim 3, the at least one of the one or more heuristic components employing a file that contains reference data to facilitate transforming data from a first format or first schema associated with a first domain to a disparate format or disparate schema associated with a disparate domain.

9. The system of claim 3, wherein the at least one of the one or more heuristic components:

analyzes at least one of data or metadata associated with the first metadata component and the downstream metadata component,

infers one or more types of data respectively associated with the first metadata component and the downstream metadata component based at least in part on the analysis of the at least one of data or metadata, and

automatically determines a mapping between the first metadata component and the downstream metadata component that is utilized to facilitate arbitration of metadata differences between the first metadata component and the downstream metadata component.