OPTIMIZING PARALLEL DATA FLOWS WITH GENERATED JOB PROFILE DATA

• US 20150261572A1
• Filed: 02/13/2015
• Published: 09/17/2015
• Est. Priority Date: 03/11/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: CAUFIELD, Brian K., GREENE, Lawrence A., JACOBSON, Eric A., LI, Yong, PU, Xiaoyan
• Granted Patent: US 9,983,906 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/34 Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
• CPC Class Codes: G06F 9/4843 by program, e.g. task dispatcher, supervisor, operating system
• CPC Class Codes: G06F 9/50 Allocation of resources, e.g. of the central processing unit [CPU]

Abstract

Embodiments presented herein provide techniques for optimizing parallel data flows of a batch processing job using a profile of the processing job. An application retrieves a job profile for a processing job. The processing job has a plurality of processing stages specified in an execution profile. The job profile includes statistical data for at least one of the processing stages obtained during prior executions of the job. The application modifies properties of the execution profile based on the job profile to optimize the execution of the job. The application executes the processing job with the modified execution profile.

Claims

1. A method comprising:

retrieving, via a processor, a job profile for a processing job, wherein the processing job has a plurality of processing stages specified in an execution profile, wherein the job profile includes statistical data for at least one of the processing stages obtained during prior executions of the job;

modifying properties of the execution profile based on the job profile to optimize execution of the job; and

executing the processing job with the modified execution profile.

2. The method of claim 1, wherein the statistical data includes at least one of a total data size processed in the processing stages, an average record size processed by data transfer blocks, and an amount of system usage for each processing stage.

3. The method of claim 1, wherein the job profile was generated in a prior execution of the processing job by monitoring the processing stages during the prior execution.

4. The method of claim 1, further comprising, receiving a specification of properties to modify based on the job profile.

5. The method of claim 1, wherein the job profile comprises multiple job profiles from one or more prior executions of the processing job.

6. The method of claim 1, wherein the processing job is an extract, transform, and load job.

7. The method of claim 1, wherein the modified properties includes at least one of a transfer block size, a selection of a compression algorithm, table chaining, and a record layout.