SYSTEM AND METHOD OF DATA MOVEMENT BETWEEN A DATA SOURCE AND A DESTINATION

• US 20080140692A1
• Filed: 12/07/2007
• Published: 06/12/2008
• Est. Priority Date: 12/11/2006
• Status: Active Grant

• Current Assignee: Breakaway Technologies, Inc.
• Original Assignee: Breakaway Technologies, Inc.
• Inventors: Gehring, Scott
• Granted Patent: US 7,747,563 B2
• Time in Patent Office: Days
• US Class Current: 707/101
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses

Abstract

A reusable business performance management integration framework provides high performance and maximum efficiency for data transfer of general ledger data. The system issues commands to extract data from a named data source and then processes, tracks, prepares and delivers the data to a named system designation. The system identifies unique data intersections and synchronizes and/or reloads data against data intersection patterns without the use of primary keys. Integration templates may be applied to the basic framework. Each template contains the map for data transmission between a given general ledger system and the destination application type.

Claims

1. A framework of an automated, computer-implemented system, comprising:

a controller for executing a data migration process operable to issue commands to extract data from a named data source, process said data and load said data to a named destination;

a request broker module operable to control asynchronous operation of on-demand and scheduled requests issued by the controller;

an automation workflow module operable to log and record the operational data of the process;

a staging database module providing a repository of integration framework configurations, process logs, and a data transformation audit trail;

a data process control module operable to visually display the operation of the controller; and

a pipeline configuration module operable to provide configuration of data movement from the data source to the named destination.

2. The system of claim 1 wherein the staging database includes database tables and stored procedures.

3. The system of claim 2 wherein said tables comprise configuration settings to identify the server environment;

the transformational business rules, security and connection information, behavioral options, performance tuning switches and source query definitions,

4. The system of claim 2 wherein said tables further include audit trail tables operational to write the results of each of the process-dependent steps as inbound data is changed and processed through the data pipeline.

5. The system of claim 2 wherein said tables further include log tables to log operational data of the data pipeline controller including success/failure information, execution time, pipeline volume and errors.

6. The system of claim 2 wherein said stored procedures read, execute and apply sequential demands on the tables to drive the execution of the data process.

7. The system of claim 1 wherein said staging database further includes an intersection coordination module which coordinates keyless data synchronization between source and destination systems.

8. The system of claim 1 wherein said staging database further includes column-to-row transformation.

9. The system of claim 1 wherein said pipeline configuration module further includes an injection template.

15. The system of claim 9 further including an integration template containing a map for data movement between a given source system and a given destination type including transformational layouts, business rules, source schema and filters.

16. The system of claim 1 further including a collision control mechanism for queuing scheduled queries.

17. The system of claim 15 further including a plurality of interchangeable integration templates employed singly or alternatively.

18. The system of claim 1 wherein said staging database further includes a kickout handling subsystem where data that does not pass referential validation is stored.

19. The system of claim 17 wherein said integration templates are automatically injected into the data process controller during configuration of the framework.

20. The system of claim 1 wherein said automation workflow module is operable to sequentially execute commands in a progressive flow.

10. A method of data migration and integration in an automated, computer-implemented system comprising the steps of:

extracting the data from a named source by remote queries;

mapping the data from the source to a destination by mapping inbound source fields from the source to outbound fields of the destination;

providing supplemental transformations to inbound data for data renaming, null scrubbing and data polishing;

filtering extraneous data that has accumulated during prior steps or that has originated from the source;

compressing the data to consolidate any fragmented balances to provide a set table signature that provides a replica of the destination fact table format, data types and column order;

validating data line by line to provide referential integrity of the inbound data flow against the named destination system;

routing data that does not pass referential validation to a kickout area of the data process controller;

comparing the inbound data flow to the data in the named destination to determine how to synchronize the data to limit it to certain fields; and

deploying the data connection transformation process to the destination.

11. The method of claim 10 wherein said deploying step comprises a synch method which performs side-by-side, line-by-line data comparison of the final results of the data connection transformation process and the final production data in a named destination system.

12. The method of claim 10 wherein said deploying step comprises a reload method which deploys all inbound data to final production regardless of the differences from an optimized state.

13. The method of claim 10 further including a post-maintenance step which provides staging cleanup, index maintenance and cube processing.

14. The method of claim 10 wherein said mapping step further includes identifying unique data intersections and synchronizing and/or reloading data against data intersection patterns without the use of primary keys.