Managing parallel data processing jobs in grid environments

• US 8,281,012 B2
• Filed: 01/30/2008
• Issued: 10/02/2012
• Est. Priority Date: 01/30/2008
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Pu, Xiaoyan, Thornton, Christopher S.
• Primary Examiner(s): Patel, Chirag R
• Application Number:
  US12/022,602
• Publication Number:
  US 20090193427A1
• Time in Patent Office: 1,707 Days
• Field of Search: 709/226, 709/229, 709201-203, 718102-105
• US Class Current: 709/226
• CPC Class Codes: G06F 9/5072 Grid computing

Abstract

Method, system, and computer program product for managing parallel data processing jobs in grid environments are provided. A request to deploy a parallel data processing job in a grid environment is received. A plurality of resource nodes in the grid environment are dynamically allocated to the parallel data processing job. A configuration file is automatically generated for the parallel data processing job based on the allocated resource nodes. The parallel data processing job is then executed in the grid environment using the generated configuration file.

Claims

1. A method for managing parallel data processing jobs in grid environments, the method comprising:

receiving a request to deploy a parallel data processing job in a grid environment;

dynamically allocating at runtime a plurality of resource nodes in the grid environment to the parallel data processing job;

automatically generating a configuration file for the parallel data processing job based on the plurality of resource nodes dynamically allocated; and

executing the parallel data processing job in the grid environment using the configuration file, wherein dynamic allocation of the plurality of resource nodes to the parallel data processing job is based on resource requirements specified for the parallel data processing job and a configuration file template for the grid environment,

the resource requirements for the parallel data processing, job specifying a plurality of node pools to be used for the parallel data processing job,

the configuration file template mapping each of the plurality of resource nodes dynamically allocated to the parallel data processing job to one of the plurality of node pools specified in the resource requirements.

2. The method of claim 1, wherein the configuration file template for the grid environment is a job level template that only includes information on node pools useful for the parallel data processing job.

3. The method of claim 1, wherein the resource requirements for the parallel data processing job further specifies a number of nodes from each of the plurality of node pools to be used for the parallel data processing job.

4. The method of claim 1, wherein the parallel data processing job was designed in an environment different from the grid environment.

5. The method of claim 1, wherein the parallel data processing job is deployable in an environment different from the grid environment without having to be modified.

6. The method of claim 1, wherein the parallel data processing job is part of a job sequence and the configuration file generated for the parallel data processing job is used for executing another parallel data processing job in the job sequence.

7. A system for managing parallel data processing jobs in grid environments, the system comprising:

a processor; and

an application running on the processor, the application

receiving a request to deploy a parallel data processing job in a grid environment,

dynamically allocating at runtime a plurality of resource nodes in the grid environment to the parallel data processing job,

automatically generating a configuration file for the parallel data processing job based on the plurality of resource nodes dynamically allocated, and

executing the parallel data processing job in the grid environment using the configuration file, wherein dynamic allocation of the plurality of resource nodes to the parallel data processing job is based on resource requirements specified for the parallel data processing job and a configuration file template for the grid environment,

the resource requirements for the parallel data processing job specifying a plurality of node pools to be used for the parallel data processing job,

the configuration file template mapping each of the plurality of resource nodes dynamically allocated to the parallel data processing job to one of the plurality of node pools specified in the resource requirements.

8. The system of claim 7, wherein the configuration file template for the grid environment is a job level template that only includes information on node pools useful for the parallel data processing job.

9. The system of claim 7, wherein the resource requirements for the parallel data processing job further specifies a number of nodes from each of the plurality of node pools to be used for the parallel data processing job.

10. The system of claim 7, wherein the parallel data processing job was designed in an environment different from the grid environment.

11. The system of claim 7, wherein the parallel data processing job is deployable in an environment different from the grid environment without having to be modified.

12. The system of claim 7, wherein the parallel data processing job is part of a job sequence and the configuration file generated for the parallel data processing job is used for executing another parallel data processing job in the job sequence.

13. A computer program product comprising a non-transitory computer readable medium encoded with a computer program for managing parallel data processing jobs in grid environments, wherein the computer program, when executed on a computer, causes the computer to:

receive a request to deploy a parallel data processing job in a grid environment;

dynamically allocate at runtime a plurality of resource nodes in the grid environment to the parallel data processing job;

automatically generate a configuration file for the parallel data processing job based on the plurality of resource nodes dynamically allocated; and

execute the parallel data processing job in the grid environment using the configuration file, wherein dynamic allocation of the plurality of resource nodes to the parallel data processing job is based on resource requirements specified for the parallel data processing job and a configuration file template for the grid environment,

the resource requirements for the parallel data processing job specifying a plurality of node pools to be used for the parallel data processing job,

the configuration file template mapping each of the plurality of resource nodes dynamically allocated to the parallel data processing job to one of the plurality of node pools specified in the resource requirements.

14. The computer program product of claim 13, wherein the resource requirements for the parallel data processing job further specifies a number of nodes from each of the plurality of node pools to be used for the parallel data processing job.

15. The computer program product of claim 13, wherein the parallel data processing job was designed in an environment different from the grid environment.

16. The computer program product of claim 13, wherein the parallel data processing job is deployable in an environment different from the grid environment without having to be modified.

17. The computer program product of claim 13, wherein the parallel data processing job is part of a job sequence and the configuration file generated for the parallel data processing job is used for executing another parallel data processing job in the job sequence.