PERFORMANCE CHECKING COMPONENT FOR AN ETL JOB

• US 20150347261A1
• Filed: 05/30/2014
• Published: 12/03/2015
• Est. Priority Date: 05/30/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Li, Jeff J., Nusbickel, Wendi L., Tsimis, James G.
• Granted Patent: US 9,626,416 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/2268 Logging of test results
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/302 where the computing system component is a software system
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3414 Workload generation, e.g. scripts, playback
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 2201/80 Database-specific techniques

Abstract

Generation of a performance determination report for an Extract, Transform, Load (ETL) job includes decomposing the ETL job into two or more stage instances, and identifying one or more conditions for each of the stage instances. A set of tests for each of the identified conditions are generated. A first set of test results are generated by performing the set of tests. It is determined whether a test result from the first set of test results is outside of a first range. Conditions that can be identified include a non-volatile free memory condition, a network reliability condition, a network configuration condition, an application availability condition, a database availability condition, a database performance condition, a schema validity condition, an installed libraries condition, a configuration parameter condition, a volatile free memory condition, and a third party tool condition.

Claims

1. A computer-implemented method performed by a processor for generating a performance determination report for an Extract, Transform, Load (ETL) job, comprising:

decomposing an ETL job into two or more stage instances;

identifying one or more conditions for each of the stage instances;

generating a set of tests for each of the identified conditions;

generating a first set of test results by performing the sets of tests;

determining whether a test result from the first set of test results is outside of a first range; and

generating the performance determination report, the performance determination report including the test result.

2. The method of claim 1, wherein the one or more conditions are selected from a group consisting of a non-volatile free memory condition, a network reliability condition, a network configuration condition, an application availability condition, a database availability condition, a database performance condition, a schema validity condition, an installed libraries condition, a configuration parameter condition, a volatile free memory condition, and a third party tool condition.

3. The method of claim 1, further comprising saving the first set of test results to an archive.

4. The method of claim 1, wherein the determining of whether a test result from the first set of test results is outside a first range includes:

comparing the test result with another test result for the same test performed at a second time, the second time being prior to a first time, wherein the generating of the first set of test results is performed at the first time.

5. The method of claim 1, further comprising determining the first range for the test result, wherein determining the first range includes:

calculating one or more statistical metrics from two or more historical test results.

6. The method of claim 1, further comprising accepting a user input to modify a test of the sets of tests.

7. The method of claim 1, further comprising accepting a user input to add a test to the sets of tests.

8. The method of claim 1, further comprising determining:

whether a compute node hosting a stage instance of a first ETL job is simultaneously hosting a second ETL job at a first time; and

whether the compute node hosting the stage instance of the first ETL job is simultaneously hosting the second ETL job at a second time, the second time being prior to first time.

9. A computer system comprising:

a processor; and

a memory communicatively coupled with the processor, wherein the memory is encoded within instructions that when executed by the processor perform operations comprising;

decomposing an ETL job into two or more stage instances;

identifying one or more conditions for each of the stage instances;

generating a set of tests for each of the identified conditions;

generating a first set of test results by performing the sets of tests;

determining whether a test result from the first set of test results is outside a first range; and

generating the performance determination report, the performance determination report including the test result.

10. The computer system of claim 9, wherein the one or more conditions are selected from a group consisting of a non-volatile free memory condition, a network reliability condition, a network configuration condition, an application availability condition, a database availability condition, a database performance condition, a schema validity condition, an installed libraries condition, a configuration parameter condition, a volatile free memory condition, and a third party tool condition.

11. The computer system of claim 9, further comprising saving the first set of test results to an archive.

12. The computer system of claim 9, wherein the determining of whether a test result from the first set of test results is outside a first range includes:

comparing the test result with another test result for the same test performed at a second time, the second time being prior to a first time, wherein the generating of the first set of test results is performed at the first time.

13. The computer system of claim 9, further comprising determining the first range for the test result, wherein determining the first range includes:

calculating one or more statistical metrics from two or more historical test results.

14. The computer system of claim 9, further comprising accepting a user input to add a test to the sets of tests.

15. The computer system of claim 9, further comprising determining:

whether a compute node hosting a stage instance of a first ETL job is simultaneously hosting a second ETL job at a first time; and

whether the compute node hosting the stage instance of the first ETL job is simultaneously hosting the second ETL job at a second time, the second time being prior to first time.

16. A computer program product including a computer readable storage medium having computer readable program instructions stored thereon for causing a processor to perform the following operations to generate a performance determination report for an Extract, Transform, Load (ETL) job:

decomposing an ETL job into two or more stage instances;

identifying one or more conditions for each of the stage instances;

generating a set of tests for each of the identified conditions;

generating a first set of test results by performing the sets of tests;

determining whether a test result from the first set of test results is outside a first range; and

generating the performance determination report, the performance determination report including the test result.

17. The computer program product of claim 16, wherein the one or more conditions are selected from a group consisting of a non-volatile free memory condition, a network reliability condition, a network configuration condition, an application availability condition, a database availability condition, a database performance condition, a schema validity condition, an installed libraries condition, a configuration parameter condition, a volatile free memory condition, and a third party tool condition.

18. The computer program product of claim 16, wherein the determining of whether a test result from the first set of test results is outside a first range includes:

comparing the test result with another test result for the same test performed at a second time, the second time being prior to a first time, wherein the generating of the first set of test results is performed at a first time.

19. The computer program product of claim 16, further comprising determining the first range for the test result, wherein determining the first range includes:

calculating one or more statistical metrics from two or more historical test results.

20. The computer program product of claim 16, further comprising determining:

whether a compute node hosting a stage instance of a first ETL job is simultaneously hosting a second ETL job at a first time; and

whether the compute node hosting the stage instance of the first ETL job is simultaneously hosting the second ETL job at a second time, the second time being prior to first time.