Method for resource optimization for parallel data integration

• US 8,954,981 B2
• Filed: 02/24/2012
• Issued: 02/10/2015
• Est. Priority Date: 09/04/2009
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Harris, Simon David, Pu, Xiaoyan
• Primary Examiner(s): Bullock, Jr., Lewis A
• Assistant Examiner(s): Lu, Kevin X
• Application Number:
  US13/405,160
• Publication Number:
  US 20120167112A1
• Time in Patent Office: 1,082 Days
• Field of Search: None
• US Class Current: 718/104
• CPC Class Codes: G06F 2209/5019 Workload prediction
• CPC Class Codes: G06F 2209/508 Monitor
• CPC Class Codes: G06F 9/5072 Grid computing

Abstract

For optimizing resources for a parallel data integration job, a job request is received, which specifies a parallel data integration job to deploy in a grid. Grid resource utilizations are predicted for hypothetical runs of the specified job on respective hypothetical grid resource configurations. This includes automatically predicting grid resource utilizations by a resource optimizer module responsive to a model based on a plurality of actual runs of previous jobs. A grid resource configuration is selected for running the parallel data integration job, which includes the optimizer module automatically selecting a grid resource configuration responsive to the predicted grid resource utilizations and an optimization criterion.

Claims

1. A method of optimizing resources for a parallel data integration job, the method comprising:

receiving a job request specifying a parallel data integration job to deploy in a grid wherein the job request includes operators specifying parallel integration operations performed when the parallel data integration job is run;

predicting grid resource utilizations for hypothetical runs of the specified job on respective hypothetical grid resource configurations responsive to a model based on performance data from a plurality of actual runs of previously deployed, parallel data jobs;

selecting a grid resource configuration for running the parallel data integration job, including automatically selecting a grid resource configuration responsive to the predicted grid resource utilizations and an optimization criterion based on at least one resource utilization index for the job; and

generating the at least one resource utilization index for the job, comprising;

generate resource utilization indices for each respective operator responsive to the predicted grid resource utilizations on resource portions;

generating a respective operator index maximum for each respective operator;

generating, for each of a respective group of the operators, a respective maximum of the operator index maxima among the operators of the respective group;

selecting a first maximum of resource utilization indices for a first and second subset of data source and sink operator groups;

selecting a second maximum of resource utilization indices for a first and second subset of processing and scratch operator groups; and

generating the at least one resource utilization index for the job responsive to a ratio of the first and second maxima.

2. The method of claim 1, comprising:

generating resource utilization categories, including automatically generating resource utilization categories responsive to the predicted grid resource utilizations, wherein automatically selecting the grid resource configuration responsive to the optimization criterion includes selecting the grid resource configuration responsive to the categories.

3. The method of claim 1, comprising:

generating a resource utilization index for the job responsive to a sum of the predicted grid resource utilizations for all the operators.

4. The method of claim 1, comprising:

generating correlation coefficients for the model responsive to performance data for a plurality of previous jobs actually run on respective configurations of the grid resources.

5. The method of claim 1, wherein selecting the grid resource configuration for running the parallel data integration job comprises:

selecting an optimal number of physical compute nodes.

6. The method of claim 1, wherein selecting the grid resource configuration for running the parallel data integration job comprises:

adjusting estimated resource utilizations for operators responsive to a ratio of user-specified job execution time to estimated job execution time; and

selecting a number of partitions for each operator responsive to a combined total of adjusted estimated resource utilizations on all the operator'"'"'s partitions and a minimum of the adjusted estimated resource utilizations among all the operator'"'"'s partitions.

7. The method of claim 1, wherein the job request includes a data graph of linked operators specifying a sequence of parallel data integration operations performed when the parallel data integration job is run, such that each operator has one or more respective link mates, and wherein selecting the grid resource configuration for running the parallel data integration job comprises:

traversing the data graph and increasing numbers of partitions for operators having throughputs less than their respective link mates.

8. The method of claim 1, wherein selecting the grid resource configuration for running the parallel data integration job comprises:

selecting a number of partitions for each physical node responsive to the categories.