Dynamic optimization of workload execution based on statistical data collection and updated job profiling

• US 9,983,906 B2
• Filed: 02/13/2015
• Issued: 05/29/2018
• Est. Priority Date: 03/11/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Caufield, Brian K., Greene, Lawrence A., Jacobson, Eric A., Li, Yong, Pu, Xiaoyan
• Primary Examiner(s): Wu, Benjamin C
• Application Number:
  US14/621,719
• Publication Number:
  US 20150261572A1
• Time in Patent Office: 1,201 Days
• CPC Class Codes: G06F 11/34 Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
• CPC Class Codes: G06F 9/4843 by program, e.g. task dispatcher, supervisor, operating system
• CPC Class Codes: G06F 9/50 Allocation of resources, e.g. of the central processing unit [CPU]

Abstract

Embodiments presented herein provide techniques for optimizing parallel data flows of a batch processing job using a profile of the processing job. An application retrieves a job profile for a processing job. The processing job has a plurality of processing stages specified in an execution profile. The job profile includes statistical data for at least one of the processing stages obtained during prior executions of the job. The application modifies properties of the execution profile based on the job profile to optimize the execution of the job. The application executes the processing job with the modified execution profile.

Claims

1. A method comprising:

retrieving, via a processor, a job profile for a processing job, wherein the processing job has a plurality of processing stages specified in an execution profile, wherein the execution profile specifies a respective set of one or more parameters for each of one or more of the processing stages of the processing job, and wherein the job profile includes statistical data for at least one of the processing stages obtained during prior executions of the processing job, wherein the statistical data includes an amount of data that was spilled to a disk during at least one of the one or more processing stages, and wherein the statistical data was obtained by, for each of the processing stages during at least a first of the prior executions;

identifying a plurality of operations in the processing stage, and

during performance of each of the operations;

determining whether a flag is set for the operation, wherein the flag indicates to gather statistical data relating to the operation, and

upon determining that a flag is set, collecting the statistical data relating to the operation;

identifying a plurality of optimizations to apply to the execution profile based on the statistical data and based on an indication that at least a first one of the processing stages of the processing job was selected for optimization during one of the prior executions;

receiving, via a user interface, a selection of at least a first optimization of the identified optimizations to apply to the execution profile, wherein the first optimization comprises a selection of a compression algorithm;

modifying the respective set of one or more parameters of the execution profile for the at least the first one of the processing stages of the processing job by applying the selected first optimization to the at least the first one of the processing stages to optimize an execution environment used to execute the processing job; and

executing the processing job in the optimized execution environment based on the modified execution profile.

2. The method of claim 1, wherein the statistical data includes at least one of a total data size processed in the processing stages, an average record size processed by data transfer blocks, and an amount of system resource usage for each processing stage.

3. The method of claim 1, wherein the job profile was generated during a prior execution of the processing job by monitoring the processing stages during the prior execution.

4. The method of claim 1, further comprising, receiving a specification of the parameters of the execution profile which are available to modify based on the job profile.

5. The method of claim 1, wherein the job profile comprises a plurality of job profiles generated during a corresponding plurality of prior executions of the processing job.

6. The method of claim 1, wherein the processing job is an extract, transform, and load job.

7. The method of claim 1, wherein the modified respective set of one or more parameters includes at least one of a transfer block size, table chaining, and a record layout specified in the execution profile.