Resource Optimization for Parallel Data Integration

• US 20110061057A1
• Filed: 09/04/2009
• Published: 03/10/2011
• Est. Priority Date: 09/04/2009
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Pu, Xiaoyan, Harris, Simon David
• Granted Patent: US 8,935,702 B2
• Time in Patent Office: Days
• US Class Current: 718/104
• CPC Class Codes: G06F 2209/5019 Workload prediction
• CPC Class Codes: G06F 2209/508 Monitor
• CPC Class Codes: G06F 9/5072 Grid computing

Abstract

For optimizing resources for a parallel data integration job, a job request is received, which specifies a parallel data integration job to deploy in a grid. Grid resource utilizations are predicted for hypothetical runs of the specified job on respective hypothetical grid resource configurations. This includes automatically predicting grid resource utilizations by a resource optimizer module responsive to a model based on a plurality of actual runs of previous jobs. A grid resource configuration is selected for running the parallel data integration job, which includes the optimizer module automatically selecting a grid resource configuration responsive to the predicted grid resource utilizations and an optimization criterion.

Claims

1. A method of optimizing resources for a parallel data integration job, the method comprising:

receiving a job request specifying a parallel data integration job to deploy in a grid;

predicting grid resource utilizations for hypothetical runs of the specified job on respective hypothetical grid resource configurations, including automatically predicting grid resource utilizations by a resource optimizer module responsive to a model based on a plurality of actual runs of previous jobs; and

selecting a grid resource configuration for running the parallel data integration job, including the optimizer module automatically selecting a grid resource configuration responsive to the predicted grid resource utilizations and an optimization criterion.

2. The method of claim 1, comprising:

generating resource utilization categories, including the optimizer module automatically generating resource utilization categories responsive to the predicted grid resource utilizations, wherein the optimizer module automatically selecting the grid resource configuration responsive to the optimization criterion includes selecting the grid resource configuration responsive to the categories.

3. The method of claim 2, wherein the job request includes operators specifying parallel data integration operations performed when the parallel data integration job is run, and wherein the optimizer module automatically generating resource utilization categories responsive to the predicted grid resource utilizations comprises:

generating resource utilization indices for the respective operators responsive to the predicted grid resource utilizations;

generating resource utilization indices for respective groups of the operators responsive to the resource utilization indices of the respective operator groups; and

generating a first resource utilization index for the job responsive to the resource utilization indices of the respective operator groups

4. The method of claim 3, comprising:

generating a second resource utilization index for the job, wherein generating the second resource utilization index includes;

selecting a first maximum of the resource utilization indices for a first subset of the operator groups;

selecting a second maximum of the resource utilization indices for a second subset of the operator groups; and

computing a ratio of the first and second maxima.

5. The method of claim 3, comprising:

generating a third resource utilization index for the job responsive to a sum of the predicted grid resource utilizations for all the operators.

6. The method of claim 1, comprising:

generating correlation coefficients for the model responsive to performance data for a plurality of previous jobs actually run on respective configurations of the grid resources.

7. The method of claim 1, wherein selecting the grid resource configuration for running the parallel data integration job comprises:

selecting an optimal number of physical compute nodes.

8. The method of claim 1, wherein selecting the grid resource configuration for running the parallel data integration job comprises:

adjusting estimated resource utilizations for operators responsive to a ratio of user-specified job execution time to estimated job execution time; and

selecting a number of partitions for each operator responsive to a combined total of adjusted estimated resource utilizations on all the operator'"'"'s partitions and a minimum of the adjusted estimated resource utilizations among all the operator'"'"'s partitions.

9. The method of claim 1, wherein the job request includes a data graph of linked operators specifying a sequence of parallel data integration operations performed when the parallel data integration job is run, such that each operator has one or more respective link mates, and wherein selecting the grid resource configuration for running the parallel data integration job comprises:

traversing the data graph and increasing numbers of partitions for operators having throughputs less than their respective link mates.

10. The method of claim 2, wherein selecting the grid resource configuration for running the parallel data integration job comprises:

selecting a number of partitions for each physical node responsive to the categories.

11. A program product for parallel data integration job, the program product comprising:

a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising;

computer readable code configured to receive a job request specifying a parallel data integration job to deploy in a grid;

computer readable code configured to predict grid resource utilizations for hypothetical runs of the specified job on respective hypothetical grid resource configurations, including resource optimizer module computer readable code configured to automatically predict grid resource utilizations responsive to a model based on a plurality of actual runs of previous jobs; and

computer readable code configured to select a grid resource configuration for running the parallel data integration job, including resource optimizer module computer readable code configured to automatically select a grid resource configuration responsive to the predicted grid resource utilizations and an optimization criterion.

12. The program product of claim 11, comprising:

computer readable code configured to generate resource utilization categories, including resource optimizer module computer readable code configured to automatically generate resource utilization categories responsive to the predicted grid resource utilizations, wherein automatically selecting the grid resource configuration responsive to the optimization criterion includes selecting the grid resource configuration responsive to the categories.

13. The program product of claim 12, wherein the job request includes operators specifying parallel data integration operations performed when the parallel data integration job is run, and wherein automatically generating resource utilization categories responsive to the predicted grid resource utilizations comprises:

computer readable code configured to generate resource utilization indices for the respective operators responsive to the predicted grid resource utilizations;

computer readable code configured to generate resource utilization indices for respective groups of the operators responsive to the resource utilization indices of the respective operator groups; and

computer readable code configured to generate a first resource utilization index for the job responsive to the resource utilization indices of the respective operator groups

14. The program product of claim 13, comprising:

computer readable code configured to generate a second resource utilization index for the job, wherein generating the second resource utilization index includes;

selecting a first maximum of the resource utilization indices for a first subset of the operator groups;

selecting a second maximum of the resource utilization indices for a second subset of the operator groups; and

computing a ratio of the first and second maxima.

15. The program product of claim 13, comprising:

computer readable code configured to generate a third resource utilization index for the job responsive to a sum of the predicted grid resource utilizations for all the operators.

16. The program product of claim 11, comprising:

computer readable code configured to generate correlation coefficients for the model responsive to performance data for a plurality of previous jobs actually run on respective configurations of the grid resources.

17. The program product of claim 11, wherein the computer readable code configured to select the grid resource configuration for running the parallel data integration job comprises:

computer readable code configured to adjust estimated resource utilizations for operators responsive to a ratio of user-specified job execution time to estimated job execution time; and

computer readable code configured to select a number of partitions for each operator responsive to a combined total of adjusted estimated resource utilizations on all the operator'"'"'s partitions and a minimum of the adjusted estimated resource utilizations among all the operator'"'"'s partitions.

18. The program product of claim 11, wherein the job request includes a data graph of linked operators specifying a sequence of parallel data integration operations performed when the parallel data integration job is run, such that each operator has one or more respective link mates, and wherein selecting the grid resource configuration for running the parallel data integration job comprises:

computer readable code configured to traverse the data graph and increasing numbers of partitions for operators having throughputs less than their respective link mates.

19. A computer system comprising:

at least one storage system for storing a parallel data integration job resource optimization program; and

at least one processor for processing the parallel data integration job resource optimization program, the system being configured with the program and the processor to receive a job request specifying a parallel data integration job to deploy in a grid, predict grid resource utilizations for hypothetical runs of the specified job on respective hypothetical grid resource configurations, including automatically predicting grid resource utilizations by a resource optimizer module responsive to a model based on a plurality of actual runs of previous jobs, and select a grid resource configuration for running the parallel data integration job, including the optimizer module automatically selecting a grid resource configuration responsive to the predicted grid resource utilizations and an optimization criterion.

20. The computer system of claim 19, the system being configured with the program and the processor to generate resource utilization categories, including the optimizer module automatically generating resource utilization categories responsive to the predicted grid resource utilizations, wherein the optimizer module automatically selecting the grid resource configuration responsive to the optimization criterion includes selecting the grid resource configuration responsive to the categories.

21. The computer system of claim 20, wherein the job request includes operators specifying parallel data integration operations performed when the parallel data integration job is run, and wherein the program and the processor is configured such that the optimizer module automatically generating resource utilization categories responsive to the predicted grid resource utilizations includes generating resource utilization indices for the respective operators responsive to the predicted grid resource utilizations, generating resource utilization indices for respective groups of the operators responsive to the resource utilization indices of the respective operator groups, and generating a first resource utilization index for the job responsive to the resource utilization indices of the respective operator groups

22. The computer system of claim 21, the system being configured with the program and the processor to generate a second resource utilization index for the job, wherein generating the second resource utilization index includes:

selecting a first maximum of the resource utilization indices for a first subset of the operator groups;

selecting a second maximum of the resource utilization indices for a second subset of the operator groups; and

computing a ratio of the first and second maxima.

23. The computer system of claim 21, the system being configured with the program and the processor to generate a third resource utilization index for the job responsive to a sum of the predicted grid resource utilizations for all the operators.

24. The computer system of claim 19, wherein the system being configured with the program and the processor to select the grid resource configuration for running the parallel data integration job comprise the system being configured to adjust estimated resource utilizations for operators responsive to a ratio of user-specified job execution time to estimated job execution time, and select a number of partitions for each operator responsive to a combined total of adjusted estimated resource utilizations on all the operator'"'"'s partitions and a minimum of the adjusted estimated resource utilizations among all the operator'"'"'s partitions.

25. The computer system of claim 19, wherein the job request includes a data graph of linked operators specifying a sequence of parallel data integration operations performed when the parallel data integration job is run, such that each operator has one or more respective link mates, and wherein the system being configured with the program and the processor to select the grid resource configuration for running the parallel data integration job comprise the system being configured to traverse the data graph and increase numbers of partitions for operators having throughputs less than their respective link mates.