DEPLOYING PARALLEL DATA INTEGRATION APPLICATIONS TO DISTRIBUTED COMPUTING ENVIRONMENTS

• US 20140281704A1
• Filed: 03/25/2014
• Published: 09/18/2014
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: LI, Yong, JACOBSON, Eric A., MUDAMBI, Shyam R., PU, Xiaoyan
• Granted Patent: US 9,594,637 B2
• Time in Patent Office: Days
• US Class Current: 714/15
• CPC Class Codes: G06F 11/1415 at system level
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: G06F 9/5027 the resource being a machine, e.g. CPUs, Servers, Terminals
• CPC Class Codes: G06F 9/5038 considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration scheduling strategies G06F9/4881 and subgroups
• CPC Class Codes: G06F 9/5066 Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs mappping at compile time, see G06F8/451

Abstract

System, method, and computer program product to process parallel computing tasks on a distributed computing system, by computing an execution plan for a parallel computing job to be executed on the distributed computing system, the distributed computing system comprising a plurality of compute nodes, generating, based on the execution plan, an ordered set of tasks, the ordered set of tasks comprising: (i) configuration tasks, and (ii) execution tasks for executing the parallel computing job on the distributed computing system, and launching a distributed computing application to assign the tasks of the ordered set of tasks to the plurality of compute nodes to execute the parallel computing job on the distributed computing system.

Claims

1. A method to process parallel computing tasks on a distributed computing system, comprising:

computing an execution plan for a parallel computing job to be executed on the distributed computing system, the distributed computing system comprising a plurality of compute nodes;

generating, based on the execution plan, an ordered set of tasks, wherein the ordered set of tasks comprises;

(i) configuration tasks, and (ii) execution tasks for executing the parallel computing job on the distributed computing system; and

launching a distributed computing application to assign the tasks of the ordered set of tasks to the plurality of compute nodes to execute the parallel computing job on the distributed computing system.

2. The method of claim 1, further comprising:

determining, based on a run time computing environment, that the parallel computing job is to be executed on the distributed computing system, and not a parallel computing system.

3. The method of claim 2, wherein each of the ordered set of tasks is a driver to at least one parallel operator in a parallel computing system.

4. The method of claim 3, wherein the at least one parallel operator is executed by the distributed computing system without modification.

5. The method of claim 4, wherein a set of functions of the configuration tasks comprise:

(i) loading a dynamic library of the at least one parallel operator, and (ii) creating at least one socket channel for communication between the configuration tasks and the at least one parallel operator.

6. The method of claim 4, wherein a set of functions of the execution tasks comprise:

(i) starting the at least one parallel operator, (ii) monitoring an execution of the at least one parallel operator, (iii) passing input data to the at least one parallel operator, and (iv) receiving output data from the at least one parallel operator.

7. The method of claim 1, further comprising:

responsive to detecting, at a point of failure, a failure in execution of the parallel computing job on the distributed computing system, resuming execution of the parallel computing job on the distributed computing system at the point of failure, without having to restart execution of the parallel computing job on the distributed computing system.