Modeless event-driven data transformation

• US 6,820,135 B1
• Filed: 10/10/2001
• Issued: 11/16/2004
• Est. Priority Date: 08/31/2000
• Status: Expired due to Term

• Current Assignee: Actian Corporation
• Original Assignee: Pervasive Software Incorporated (Actian Corporation)
• Inventors: Dingman, Paul C., Van Dyken, Kathryn E., Landrum, Thomas, Bunton, William G.
• Primary Examiner(s): EL HADY, NABIL M
• Application Number:
  US09/974,747
• Time in Patent Office: 1,133 Days
• Field of Search: 709/202, 709/217, 709/219, 709/232, 709/246, 707/101, 707/102, 707/203, 707/10
• US Class Current: 709/246
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99953 Recoverability
• CPC Class Codes: Y10S 707/99954 Version management

Abstract

A system and method is described for event-driven data transformation. Generally, the system and method is directed to a transformation engine that iterates through one or more data sources, transforms data received from the data sources, and stores the output to one or more data targets. More specifically, the transformation engine is driven by executing specified event actions upon occurrence of specified triggering events. Thus, flexible, adaptable, highly tailored transformations can be implemented without incurring the often substantial expense of developing customized point-to-point solutions from scratch. The present invention supports one-to-one mappings, many-to-one mappings, one-to-many mappings, and many-to-many mappings. In addition, the present invention supports both hierarchical and flat data sources and targets.

Claims

1. A system for data transformation, comprising:

one or more read spokes, each read spoke configured to connect to one or more data sources, wherein each data source has one or more data structures referred to collectively as source structures;

one or more modeless write spokes, each modeless write spoke configured to connect to one or more data targets, wherein each data target has one or more data structures referred to collectively as target structures; and

a transformation engine operatively coupled to the one or more read spokes for retrieving data from the one or more data sources, and coupled to the one or more modeless write spokes for storing data in the one or more data targets, comprising;

a data transformation map that comprises one or more mappings relates one or more source structures to one or more target structures; and

an event list, comprising one or more event actions, each with a corresponding triggering event, wherein the transformation engine is configured to;

iterate through the data sources and detect occurrence of triggering events; and

in response to the detection of triggering events, execute the respective one or more event actions from the event action list.

2. The system of claim 1, wherein the transformation engine further comprises a query language preprocessor operable to review the data transformation map and evaluate embedded expressions in the one or more mappings.

3. The system of claim 1, wherein at least one of the event actions, referred to as a transformation event action, comprises:

retrieving at least one source structure from the data source;

transforming said at least one source structure, referred to as transformed source data;

storing said transformed source data in one or more target structures; and

wherein said transformation engine is operable, in response to a transformation event action, to transform data specified by said transformation event action in a manner described in the data transformation map.

4. The system of claim 1, further comprising a user interface configured to allow a user to define the one or more data sources, and to define data structures in each of the one or more source databases.

5. The system of claim 1, further comprising a user interface configured to allow a user to define the one or more data targets, and to define data structures in each of the one or more target databases.

6. The system of claim 1, further comprising a user interface configured to allow a user to define the relationship between one or more data sources and one or more data targets.

7. The system of claim 6, wherein the user interface is further configured to allow the user to relate source structures to target data structures.

8. The system of claim 7, wherein the user interface is further configured to graphically depict the relation between the source structures and the target structures specified in the transformation map.

9. The system of claim 6, wherein the user interface is further configured to define the relationship between one or more data sources and one or more data targets as a logical expression.

10. The system of claim 6, wherein the user interface is further configured to define the relationship between one or more data sources and one or more data targets as a numeric expression.

11. The system of claim 1, wherein the transformation engine is further configured to show the contents of the data source and the contents of data structures in the data target on a display of the system.

12. The system of claim 1, wherein at least one of the triggering events is a generic source event.

13. The system of claim 1, wherein at least one of the triggering events is a generic target event.

14. The system of claim 1, wherein at least one of the triggering events is a generic transformation event.

15. The system of claim 1, wherein at least one of the triggering events is a specific source record event.

16. The system of claim 1, wherein the transformation engine is further configured to filter the data retrieved from the data source, the data passing the filter referred to as filtered source data, and is further configured to iterate through only the filtered source data.

17. The system of claim 16, wherein the transformation engine is further configured to filter the data using predetermined sampling parameters governing a range.

18. The system of claim 16, wherein the transformation engine is further configured to filter the data using a predetermined a logical extraction criteria.

19. The system of claim 1, wherein the read spokes may connect to the one or more data sources by utilizing a raw sequential mode wherein an intuitive visual parser reconstructs record layouts.

20. The system of claim 1, wherein the read spokes may connect to the one or more data sources by utilizing a compatible physical file format allowing the transformation engine to physically read from the one or more data sources using the native internal storage format.