Resource optimization for parallel data integration

• US 8,935,702 B2
• Filed: 09/04/2009
• Issued: 01/13/2015
• Est. Priority Date: 09/04/2009
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Harris, Simon David, Pu, Xiaoyan
• Primary Examiner(s): Bullock, Jr., Lewis A
• Assistant Examiner(s): Lu, Kevin X
• Application Number:
  US12/554,867
• Publication Number:
  US 20110061057A1
• Time in Patent Office: 1,957 Days
• Field of Search: None
• US Class Current: 718/104
• CPC Class Codes: G06F 2209/5019 Workload prediction
• CPC Class Codes: G06F 2209/508 Monitor
• CPC Class Codes: G06F 9/5072 Grid computing

Abstract

For optimizing resources for a parallel data integration job, a job request is received, which specifies a parallel data integration job to deploy in a grid. Grid resource utilizations are predicted for hypothetical runs of the specified job on respective hypothetical grid resource configurations. This includes automatically predicting grid resource utilizations by a resource optimizer module responsive to a model based on a plurality of actual runs of previous jobs. A grid resource configuration is selected for running the parallel data integration job, which includes the optimizer module automatically selecting a grid resource configuration responsive to the predicted grid resource utilizations and an optimization criterion.

Claims

1. A program product for optimizing a parallel data integration job, the program product comprising:

a nontransitory computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising;

computer readable code configured to receive a job request specifying a parallel data integration job to deploy in a grid, wherein the job request includes operators specifying parallel integration operations performed when the parallel data integration job is run;

computer readable code configured to predict grid resource utilizations for hypothetical runs of the specified job on respective hypothetical grid resource configurations responsive to a model based on performance data from a plurality of actual runs of previously deployed, parallel data jobs; and

computer readable code configured to select a grid resource configuration for running the parallel data integration job, including resource optimizer module computer readable code configured to automatically select a grid resource configuration responsive to the predicted grid resource utilizations and an optimization criterion based on at least one resource utilization index for the job; and

computer readable code configured to generate the at least one resource utilization index for the job, comprising;

computer readable code configured to generate resource utilization indices for each respective operator responsive to the predicted grid resource utilizations on resource portions;

computer readable code configured to generate a respective operator index maximum for each respective operator;

computer readable code configured to generate, for each of a respective group of the operators, a respective maximum of the operator index maxima among the operators of the respective group;

computer readable code configured to select a first maximum of resource utilization indices for a first and second subset of data source and sink operator groups;

computer readable code configured to select a second maximum of resource utilization indices for a first and second subset of processing and scratch operator groups; and

computer readable code configured to generate the at least one resource utilization index for the job responsive to a ratio of the first and second maxima.

2. The program product of claim 1, comprising:

computer readable code configured to generate resource utilization categories, including resource optimizer module computer readable code configured to automatically generate resource utilization categories responsive to the predicted grid resource utilizations, wherein automatically selecting the grid resource configuration responsive to the optimization criterion includes selecting the grid resource configuration responsive to the categories.

3. The program product of claim 1, comprising:

computer readable code configured to generate a resource utilization index for the job responsive to a sum of the predicted grid resource utilizations for all the operators.

4. The program product of claim 1, comprising:

computer readable code configured to generate correlation coefficients for the model responsive to performance data for a plurality of previous jobs actually run on respective configurations of the grid resources.

5. The program product of claim 1, wherein the computer readable code configured to select the grid resource configuration for running the parallel data integration job comprises:

computer readable code configured to adjust estimated resource utilizations for operators responsive to a ratio of user-specified job execution time to estimated job execution time; and

computer readable code configured to select a number of partitions for each operator responsive to a combined total of adjusted estimated resource utilizations on all the operator'"'"'s partitions and a minimum of the adjusted estimated resource utilizations among all the operator'"'"'s partitions.

6. The program product of claim 1, wherein the job request includes a data graph of linked operators specifying a sequence of parallel data integration operations performed when the parallel data integration job is run, such that each operator has one or more respective link mates, and wherein selecting the grid resource configuration for running the parallel data integration job comprises:

computer readable code configured to traverse the data graph and increase numbers of partitions for operators having throughputs less than their respective link mates.

7. A computer system comprising:

at least one storage system for storing a parallel data integration job resource optimization program; and

at least one processor for processing the parallel data integration job resource optimization program, the system being configured with the program and the processor to;

receive a job request specifying a parallel data integration job to deploy in a grid, wherein the job request includes operators specifying parallel integration operations performed when the parallel data integration job is run;

predict grid resource utilizations for hypothetical runs of the specified job on respective hypothetical grid resource configurations responsive to a model based on a performance data from plurality of actual runs of previously deployed, parallel data jobs;

select a grid resource configuration for running the parallel data integration job, including an optimizer module automatically selecting a grid resource configuration responsive to the predicted grid resource utilizations and an optimization criterion based on at least one resource utilization index for the job; and

generate the at least one resource utilization index for the job, comprising;

generate resource utilization indices for each respective operator responsive to the predicted grid resource utilizations on resource portions;

generate a respective operator index maximum for each respective operator;

generate, for each of a respective group of the operators, a respective maximum of the operator index maxima among the operators of the respective group;

select a first maximum of resource utilization indices for a first and second subset of data source and sink operator groups;

select a second maximum of resource utilization indices for a first and second subset of processing and scratch operator groups; and

generate the at least one resource utilization index for the job responsive to a ratio of the first and second maxima.

8. The computer system of claim 7, the system being configured with the program and the processor to generate resource utilization categories, including the optimizer module automatically generating resource utilization categories responsive to the predicted grid resource utilizations, wherein the optimizer module automatically selecting the grid resource configuration responsive to the optimization criterion includes selecting the grid resource configuration responsive to the categories.

9. The computer system of claim 7, the system being configured with the program and the processor to generate a resource utilization index for the job responsive to a sum of the predicted grid resource utilizations for all the operators.

10. The computer system of claim 7, wherein the system being configured with the program and the processor to select the grid resource configuration for running the parallel data integration job comprise the system being configured to adjust estimated resource utilizations for operators responsive to a ratio of user-specified job execution time to estimated job execution time, and select a number of partitions for each operator responsive to a combined total of adjusted estimated resource utilizations on all the operator'"'"'s partitions and a minimum of the adjusted estimated resource utilizations among all the operator'"'"'s partitions.

11. The computer system of claim 7, wherein the job request includes a data graph of linked operators specifying a sequence of parallel data integration operations performed when the parallel data integration job is run, such that each operator has one or more respective link mates, and wherein the system being configured with the program and the processor to select the grid resource configuration for running the parallel data integration job comprise the system being configured to traverse the data graph and increase numbers of partitions for operators having throughputs less than their respective link mates.