METHOD AND SYSTEM FOR EXECUTING A DATA INTEGRATION APPLICATION USING EXECUTABLE UNITS THAT OPERATE INDEPENDENTLY OF EACH OTHER

• US 20090282383A1
• Filed: 07/22/2008
• Published: 11/12/2009
• Est. Priority Date: 05/12/2008
• Status: Active Grant

• Current Assignee: QlikTech International AB (Qlik Technologies)
• Original Assignee: Expressor Software
• Inventors: RUSSELL, John
• Granted Patent: US 8,141,029 B2
• Time in Patent Office: Days
• US Class Current: 717/100
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 8/71 Version control security arrangements therefor G06F21/57; Configuration management
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling

Abstract

Methods and systems for executing a data integration application using executable units that operate independently of each other. Methods include receiving a specification for a data integration application comprising programmatic rules and specifying data-flow, creating executable units corresponding to the programmatic rules, creating data-flow channels corresponding to the specified data-flow, and executing the executable units such that data is passed between them in a non-blocking manner.

Claims

1. A method of executing a data integration application using executable units, wherein the executable units are able to operate independently of each other while preserving a non-blocking, uninterrupted data-flow, the method comprising:

a. receiving specification for a data integration application, wherein the received specification specifies;

a set of computer-readable data sources from which data are to be extracted;

a set of computer-writable data targets into which data are to be stored;

a set of programmatic rules comprising extraction rules, which operate to extract data from a data source and to produce the extracted data as output, transformation rules, which operate to receive data as input and to produce transformed data as output, and storage rules, which operate to receive data as input and to store the received data into a data target; and

a set of directional data-flow segments that represent the data-flow among the programmatic rules, wherein each data-flow segment represents the unidirectional flow of data from a source rule to a destination rule;

b. creating executable units, wherein each executable unit corresponds to one of the programmatic rules and is initialized with computer logic operable to implement the corresponding programmatic rule;

c. creating data-flow channels, wherein each data-flow channel corresponds to one of the specified data-flow segments such that the data-flow channel is operable to convey data through a non-blocking data pipeline from the source rule of the data-flow segment to the destination rule of the data-flow segment;

d. launching the executable units, such that the executable units operate independently of each other to process data and to convey said data through the data-flow channels according to the specified data-flow, and such that each of the executable units provides status information on a non-blocking, out-of-band information channel that is independent of the data-flow channels;

e. such that the flow of data among the executable rules is status-message independent, preserving a non-blocking, pipelined data-flow.

2. The method according to claim 1 further comprising reserving an exclusive portion of a shared memory structure for each of the executable units, wherein each of the executable units writes status information in a non-blocking manner to the exclusive portion of the shared memory structure reserved for said executable unit.

3. The method according to claim 2 further comprising launching a control unit that periodically reads, according to a configurable parameter that specifies a fixed schedule, the status information written by each of the executable units to the shared memory structure.

4. The method according to claim 1 wherein all of the executable units are launched on a single host machine.

5. The method according to claim 1 further comprising automatically validating the received specification in order to ensure that it is free of syntax and datatype errors.

6. The method according to claim 1 further comprising automatically validating the received specification in order to ensure that the appropriate data may be extracted from the specified data sources and stored into the specified data targets.

7. The method according to claim 1 further comprising providing user controls to create a specification for a data integration application, wherein said user controls provide sets of choices to the user, and wherein said sets of choices are limited to valid choices.

8. A system for executing a data integration application using executable units, wherein the executable units are able to operate independently of each other while preserving a non-blocking, uninterrupted data-flow, the system comprising:

a. logic for receiving a specification for a data integration application, wherein the received specification specifies;

a set of computer-readable data sources from which data are to be extracted;

a set of computer-writable data targets into which data are to be stored;

a set of programmatic rules comprising extraction rules, which operate to extract data from a data source and to produce the extracted data as output, transformation rules, which operate to receive data as input and to produce transformed data as output, and storage rules, which operate to receive data as input and to store the received data into a data target; and

a set of directional data-flow segments that represent the data-flow among the programmatic rules, wherein each data-flow segment represents the unidirectional flow of data from a source rule to a destination rule;

b. logic for creating executable units, wherein each executable unit corresponds to one of the programmatic rules and is initialized with computer logic operable to implement the corresponding programmatic rule;

c. logic for creating data-flow channels, wherein each data-flow channel corresponds to one of the specified data-flow segments such that the data-flow channel is operable to convey data through a non-blocking data pipeline from the source rule of the data-flow segment to the destination rule of the data-flow segment;

d. logic for launching the executable units, such that the executable units operate independently of each other to process data and to convey said data through the data-flow channels according to the specified data-flow, and such that each of the executable units provides status information on a non-blocking, out-of-band information channel that is independent of the data-flow channels;

e. such that the flow of data among the executable rules is status-message independent, preserving a non-blocking, pipelined data-flow.

9. The system according to claim 8 further comprising a shared memory structure, wherein an exclusive portion of the shared memory structure is reserved for each of the executable units, and wherein each of the executable units writes status information in a non-blocking manner to the portion of the shared memory structure reserved for said executable unit.

10. The system according to claim 9 further comprising logic for launching a control unit that periodically reads, according to a configurable parameter that specifies a fixed schedule, the status information written by each of the executable units to the shared memory structure.

11. The system according to claim 9 further comprising logic for automatically validating the received specification in order to ensure that it is free of syntax and datatype errors.

12. The system according to claim 9 further comprising logic for automatically validating the received specification in order to ensure that the appropriate data may be extracted from the specified data sources and stored into the specified data targets.

13. The system according to claim 9 further comprising logic for providing user controls to create a specification for a data integration application, wherein said user controls provide sets of choices to the user, and wherein said sets of choices are limited to valid choices.