Model-based design for transforming data

• US 10,083,215 B2
• Filed: 04/06/2015
• Issued: 09/25/2018
• Est. Priority Date: 04/06/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Davis Jones, Dale K., Dhondse, Amol A., Pikle, Anand, Sivakumar, Gandhi
• Primary Examiner(s): Bibbee, Jared M
• Application Number:
  US14/679,068
• Publication Number:
  US 20160292244A1
• Time in Patent Office: 1,268 Days
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database

Abstract

A model-based approach for transforming data. A modeling framework that includes a plurality of rules for performing transformations of data is developed. A data unit is configured. The data unit is assigned to a job that includes one or more rules of the plurality of rules. The job transforms the data unit from a data format of a source system into a data format of a destination system. Usage statistics for at least one rule of the job are monitored. An aggregate rule is determined based on a first atomic rule and a second atomic rule. The first atomic rule and the second atomic rule are identified by at least one, or in part by a common pattern. A first rule of the job is identified based on the usage statistics of a plurality of data units. The first rule is removed from memory.

Claims

1. A method for providing a model-based design for transforming data, the method comprising:

developing, by one or more processors, a modeling framework, wherein the modeling framework includes a plurality of rules for performing transformations of data;

configuring, by one or more processors, a data unit, wherein the data unit is assigned to a job that includes one or more rules of the plurality of rules, and wherein the job transforms the data unit from a data format of a source system into a data format of a destination system;

monitoring, by one or more processors, usage statistics for at least one rule of the job, wherein a plurality of data units are split into a first stream and a second stream for parallel processing, wherein different operations are performed on each stream in parallel, and wherein results of each stream are combined;

determining, by one or more processors, a pattern in at least one data unit of the first stream and at least a second data unit of the second stream;

determining, by one or more processors, an aggregate rule based on a first atomic rule of the plurality of rules and a second atomic rule of the plurality of rules, wherein the first atomic rule and the second atomic rule are identified, in part, by the pattern;

identifying, by one or more processors, a first rule of the job based on failing to meet a predetermined threshold of usage of data units assigned to the job that include the first rule;

removing, by one or more processors, the first rule from memory, based on the first rule of the job failing to meet the predetermined threshold; and

configuring, by one or more processors, a subsequent data unit, wherein;

the subsequent data unit is assigned to a subsequent job that includes the aggregate rule; and

the subsequent job transforms the subsequent data unit from the data format of the source system into the data format of the destination system.

2. The method of claim 1, wherein the usage statistics of the first rule indicate that a usage of the first rule is below a predetermined threshold.

3. The method of claim 1, wherein the usage statistics of the first rule indicate that the first rule and a second rule outputs are redundant.

4. The method of claim 1, wherein configuring the data unit is based on input received via a user interface.

5. The method of claim 4, wherein the user interface is adapted for dragging and dropping of data units to elements of the source system and the destination system.

6. The method of claim 1, wherein a defined monitoring policy retains a second rule of the plurality of rules, wherein the second rule is associated with the data unit that is most recently used.

7. The method of claim 1, wherein the first atomic rule identifies a first data unit of a first primitive data type, wherein the second atomic rule identifies a second data unit of a second primitive data type, and wherein the aggregate rule comprises a composite data type that is based on the first primitive data type and the second primitive data type.

8. A computer program product for providing a model-based design for transforming data, the computer program product comprising:

one or more computer readable storage media and program instructions stored on the one or more computer readable storage media, the program instructions comprising;

program instructions to develop a modeling framework, wherein the modeling framework includes a plurality of rules for performing transformations of data;

program instructions to configure a data unit, wherein the data unit is assigned to a job that includes one or more rules of the plurality of rules, and wherein the job transforms the data unit from a data format of a source system into a data format of a destination system;

program instructions to monitor usage statistics for at least one rule of the job, wherein a plurality of data units are split into a first stream and a second stream for parallel processing, wherein different operations are performed on each stream in parallel, and wherein results of each stream are combined;

program instructions to determine a pattern in at least one data unit of the first stream and at least a second data unit of the second stream;

program instructions to determine an aggregate rule based on a first atomic rule of the plurality of rules and a second atomic rule of the plurality of rules, wherein the first atomic rule and the second atomic rule are identified, in part by, the pattern;

program instructions to identify a first rule of the job based on failing to meet a predetermined threshold of usage of data units assigned to the job that include the first rule;

program instructions to remove the first rule from memory, based on the first rule of the job failing to meet the predetermined threshold; and

program instructions to configure a subsequent data unit, wherein;

the subsequent data unit is assigned to a subsequent job that includes the aggregate rule; and

the subsequent job transforms the subsequent data unit from the data format of the source system into the data format of the destination system.

9. The computer program product of claim 8, wherein the usage statistics of the first rule indicate that a usage of the first rule is below a predetermined threshold.

10. The computer program product of claim 8, wherein the usage statistics of the first rule indicate that the first rule and a second rule outputs are redundant.

11. The computer program product of claim 8, wherein configuring the data unit is based on input received via a user interface.

12. The computer program product of claim 11, wherein the user interface is adapted for dragging and dropping of data units to elements of the source system and the destination system.

13. The computer program product of claim 8, wherein a defined monitoring policy specifies to retain a second rule of the plurality of rules, wherein the second rule is associated with the data unit that is most recently used.

14. The computer program product of claim 8, wherein the first atomic rule identifies a first data unit of a first primitive data type, wherein the second atomic rule identifies a second data unit of a second primitive data type, and wherein the aggregate rule comprises a composite data type that is based on the first primitive data type and the second primitive data type.

15. A computer system for a model-based approach for transforming data, the computer system comprising:

one or more computer processors, one or more computer readable storage media, and program instructions stored on the computer readable storage media for execution by at least one of the one or more processors, the program instructions comprising;

program instructions to develop a modeling framework, wherein the modeling framework includes a plurality of rules for performing transformations of data;

program instructions to configure a data unit, wherein the data unit is assigned to a job that includes one or more rules of the plurality of rules, and wherein the job transforms the data unit from a data format of a source system into a data format of a destination system;

program instructions to monitor usage statistics for at least one rule of the job, wherein a plurality of data units are split into a first stream and a second stream for parallel processing, wherein different operations are performed on each stream in parallel, and wherein results of each stream are combined;

program instructions to determine a pattern in at least one data unit of the first stream and at least a second data unit of the second stream;

program instructions to determine an aggregate rule based on a first atomic rule of the plurality of rules and a second atomic rule of the plurality of rules, wherein the first atomic rule and the second atomic rule are identified, in part, by the pattern;

program instructions to identify a first rule of the job based on failing to meet a predetermined threshold of usage of data units assigned to the job that include the first rule;

program instructions to remove the first rule from memory, based on the first rule of the job failing to meet the predetermined threshold; and

program instructions to configure a subsequent data unit, wherein;

the subsequent data unit is assigned to a subsequent job that includes the aggregate rule; and

the subsequent job transforms the subsequent data unit from the data format of the source system into the data format of the destination system.

16. The computer system of claim 15, wherein the usage statistics of the first rule indicate that a usage of the first rule is below a predetermined threshold.

17. The computer system of claim 15, wherein the usage statistics of the first rule indicate that the first rule and a second rule outputs are redundant.

18. The computer system of claim 15, wherein configuring the data unit is based on input received via a user interface.

19. The computer system of claim 18, wherein the user interface is adapted for dragging and dropping of data units to elements of the source system and the destination system.

20. The computer system of claim 15, wherein a defined monitoring policy retains a second rule of the plurality of rules, wherein the second rule is associated with the data unit that is most recently used.