Replication-based propagation mechanism for pipelines

• US 20050149581A1
• Filed: 02/11/2005
• Published: 07/07/2005
• Est. Priority Date: 04/21/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Gupta, Amit
• Granted Patent: US 7,424,493 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2428 Query predicate definition using graphical user interfaces, including menus and forms G06F16/2423 takes precedence
• CPC Class Codes: Y10S 707/99952 Coherency, e.g. same view to multiple users
• CPC Class Codes: Y10S 707/99953 Recoverability

Abstract

Processing database objects via a database pipeline having a plurality of stages. The invention provides a replication mechanism for propagating selected data from a database in one stage to a database in another stage without accessing a file system. Each database stage may have a plurality of input and output pipes. The invention enables a user to select a subset of the database objects in each stage for propagation to a subsequent stage.

Claims

1. In a database pipeline having a plurality of stages, each of the plurality of stages having at least one database associated therewith, said database having one or more input pipes and a plurality of output pipes, a method for processing a plurality of database objects in the database via the database pipeline, said method comprising:

defining a plurality of output pipes for a database, said database having a plurality of database objects stored therein;

specifying a subset of the plurality of database objects in the database for each of the defined plurality of output pipes;

defining a subscription as a function of the specified subset of the plurality of database objects;

propagating, from a first one of the stages to another one of the stages, the specified subset of the plurality of database objects as a function of the defined subscription; and

creating a publication as a function of the propagated subset of the plurality of database objects.

2. The method of claim 1, further comprising generating the created publication.

3. The method of claim 1, further comprising defining one or more input pipes.

4. The method of claim 1, further comprising:

incrementing a counter representing a quantity of created publications;

determining if the incremented counter exceeds a pre-defined threshold; and

deleting a previously created publication as a function of said determining.

5. The method of claim 1, wherein the subset of the plurality of database objects comprises a plurality of data rows in the database, and further comprising filtering the subset of the plurality of database objects horizontally by eliminating one or more of the plurality of data rows from the subset of the plurality of database objects.

6. The method of claim 1, wherein the subset of the plurality of database objects comprises a plurality of data columns in the database, and further comprising filtering the subset of the plurality of database objects vertically by eliminating one or more of the plurality of data columns from the subset of the plurality of database objects.

7. The method of claim 1, further comprising adding a source identifier to each of the database objects in the specified subset of the plurality of database objects.

8. The method of claim 1, further comprising:

initializing one or more of the input pipes as a function of the created publication; and

initializing the defined plurality of output pipes.

9. The method of claim 1, wherein propagating the specified subset of the plurality of database objects comprises replicating the specified subset of the plurality of database objects without accessing a file system.

10. The method of claim 1, further comprising deleting a previously defined subscription.

11. The method of claim 1, wherein one or more computer-readable media have computer-executable instructions for performing the method recited in claim 1.

12. One or more computer-readable media having computer-executable components for processing a plurality of database objects in a database, said components comprising:

a configuration module for defining a plurality of output pipes for one database in a stage in a pipeline, said database having a plurality of database objects stored therein;

a selection module for specifying a subset of the plurality of database objects in the database for each of the plurality of output pipes defined by the configuration module; and

a replication module for propagating, from the stage to another stage, the subset of the plurality of database objects specified by the selection module.

13. The computer-readable media of claim 12, further comprising a subscription module for defining a subscription as a function of the subset of the plurality of database objects specified by the selection module.

14. The computer-readable media of claim 13, further comprising a publication module for generating a publication as a function of the subscription defined by the subscription module.

15. The computer-readable media of claim 14, further comprising a filter module for filtering the subset of the plurality of database objects specified by the selection module by removing a row or a column from the specified subset of the plurality of database objects.

16. The computer-readable media of claim 12, further comprising an initialization module for initializing the plurality of output pipes defined by the configuration module.

17. The computer-readable media of claim 12, further comprising a file system, and wherein the replication module propagates the subset of the plurality of database objects specified by the selection module without accessing the file system.

18. A system representing a pipeline, said system comprising:

a configuration service for selecting a subset of a plurality of database objects in a database;

a first stage for executing a first data processing service to output the subset of the plurality of database objects selected by the configuration service, wherein the database is associated with said first stage;

a second stage for executing a second data processing service, said second stage having another database associated therewith;

a data host, communicatively coupled to the first and second stages, for storing the selected subset of the plurality of database objects such that the executing of the first data processing service is asynchronous with respect to the executing of the second data processing service; and

a propagation mechanism for replicating the subset of the plurality of database objects stored in the data host to the other database associated with the second stage.

19. The system of claim 18, further comprising a data structure representing a configuration file, said data structure comprising:

an input field storing one or more values each representing an input pipe;

an output field storing a plurality of values each representing an output pipe; and

an object field associated with each of the plurality of values stored in the output field, said object field identifying a subset of the plurality of database objects.

20. The system of claim 18, wherein the configuration service comprises means for selecting the subset of the plurality of database objects in the database, wherein the propagation mechanism comprises means for replicating the subset of the plurality of database objects stored in the data host to the other database associated with the second stage, and further comprising means for defining a plurality of output pipes for the database in the first stage.