Abstract pipeline component connection

• US 20070162444A1
• Filed: 01/12/2006
• Published: 07/12/2007
• Est. Priority Date: 01/12/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: David, Matthew, Haselden, J.
• Granted Patent: US 7,779,017 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 9/4494 data driven

Abstract

The subject disclosure pertains to systems and methods for connecting pipeline components such as data flow pipeline components. Pipeline components are wrapped in metadata components that include the component as well as abstracted metadata in the form of input and output metadata descriptors. A connector can arbitrate metadata differences between pipeline components based on the metadata descriptors, thereby preventing upstream modifications from invalidating or breaking components downstream. Native or plug-in heuristic and/or related components can provide the functionality necessary to resolve metadata differences.

Claims

1. A pipeline system comprising the following computer-implemented components:

a plurality of metadata components; and

a connector component that arbitrates metadata differences between the components to eliminate breaks downstream.

2. The system of claim 1, the metadata components include a pipeline component and input and output metadata descriptors.

3. The system of claim 1, the connector component includes an execution component that executes functionality provided by a heuristic component.

4. The system of claim 3, the connector component includes a receiver component and a transmission component that enables data to be transmitted between the metadata components.

5. The system of claim 3, further comprising an interface component to enable a user to modify a heuristic component.

6. The system of claim 5, further comprising a generation component communicatively coupled to the interface component that facilitates producing a new heuristic component.

7. The system of claim 5, further comprising one or more interface components to facilitate receipt of non-native or plug-in heuristic components.

8. The system of claim 3, the heuristic component employs a domain mapping file.

9. The system of claim 2, the heuristic component automatically determines how to resolve metadata differences based on the metadata descriptors provided by the metadata components.

10. A data pipeline component connection system comprising:

a computer-implemented means for receiving data from a first pipeline component;

a computer-implemented means for arbitrating the differences between the first pipeline component and a second pipeline component; and

a computer-implemented means for transmitting arbitrated data to the second pipeline component.

11. The system of claim 10, the means for arbitrating the differences transforms data in a first format described by a first component descriptor to data in a second format described by a second component descriptor.

12. A pipeline methodology comprising the following computer-implemented acts:

receiving metadata from a first and second pipeline component; and

resolving mismatches in metadata between the components.

13. The method of claim 12, further comprising transmitting data from the first to the second pipeline components.

14. The method of claim 12, resolving mismatches comprises executing a native heuristic component.

15. The method of claim 12, resolving mismatches comprises executing a third party plug-in.

16. The method of claim 12, resolving mismatches comprises connecting like names.

17. The method of claim 12, resolving mismatches comprises connecting similar types.

18. The method of claim 12, resolving mismatches comprises connecting like names and types.

19. The method of claim 12, resolving mismatches comprises looking up differences in a mapping file that describes a particular domain.

20. The method of claim 12, further comprising wrapping pipeline components in a metadata component including metadata descriptors for input and output.