JOB DISTRIBUTION WITHIN A GRID ENVIRONMENT

• US 20130346994A1
• Filed: 06/20/2012
• Published: 12/26/2013
• Est. Priority Date: 06/20/2012
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: Platform Computing Corporation
• Inventors: Wang, Qi, Chen, Chong, Ding, Zhaohui, Yuan, Shutao
• Granted Patent: US 9,141,432 B2
• Time in Patent Office: Days
• US Class Current: 718/104
• CPC Class Codes: G06F 9/5044 considering hardware capabilities
• CPC Class Codes: G06F 9/505 considering the load
• CPC Class Codes: G06F 9/5083 Techniques for rebalancing the load in a distributed system

Abstract

According to one aspect of the present disclosure, a method and technique for job distribution within a grid environment is disclosed. The method includes: receiving jobs at a submission cluster for distribution of the jobs to at least one of a plurality of execution clusters, each execution cluster comprising one or more execution hosts; determining resource capacity corresponding to each execution cluster; determining resource requirements for the jobs; dynamically determining a pending job queue length for each execution cluster based on the resource capacity of the respective execution clusters and the resource requirements of the jobs; and forwarding jobs to the respective execution clusters according the determined pending job queue length for the respective execution cluster.

Claims

1. A method for job distribution within a grid environment, comprising:

receiving jobs at a submission cluster for distribution of the jobs to at least one of a plurality of execution clusters, each execution cluster comprising one or more execution hosts;

determining resource capacity corresponding to each execution cluster;

determining resource requirements for the jobs;

dynamically determining a pending job queue length for each execution cluster based on the resource capacity of the respective execution clusters and the resource requirements of the jobs; and

forwarding jobs to the respective execution clusters according the determined pending job queue length for the respective execution cluster.

2. The method of claim 1, wherein determining the pending job queue length for a respective execution cluster comprises:

determining a number of running jobs using a resource; and

determining a number of forwarded pending jobs requesting the resource.

3. The method of claim 2, wherein determining the pending job queue length for a respective execution cluster further comprises:

determining an amount of the resource allocated to a running job; and

determining an amount of the resource requested by a forwarded pending job.

4. The method of claim 1, further comprising:

responsive to a job requesting a plurality of resources, determining a weight factor assigned to each resource; and

selecting an execution cluster to receive the job based on an availability of resources based on the weight factor of the respective resource.

5. The method of claim 1, further comprising:

assigning a number of shares to each user submitting jobs to the submission cluster, each share representing a fraction of resources available from the execution clusters; and

dynamically determining a user priority for a user based on the shares assigned to the user and resource usage by the user.

6. The method of claim 1, further comprising:

determining if the submitted jobs include preference data indicating a preference of execution hosts for processing the jobs; and

responsive to the jobs including preference data, identifying preferred execution clusters for processing the jobs based on the preference of execution hosts.

7. The method of claim 6, further comprising:

determining if the preference data includes a preference level; and

responsive to determining that the preference data includes a preference level, sorting the preferred execution clusters based on the preference level.

8. A system, comprising:

a submission cluster for distributing jobs to at least one of a plurality of execution clusters, wherein each execution cluster comprises one or more execution hosts, and wherein the submission cluster comprises logic executable by a processor unit to;

determine resource capacity corresponding to each execution cluster;

determine resource requirements for the jobs;

dynamically determine a pending job queue length for each execution cluster based on the resource capacity of the respective execution clusters and the resource requirements of the jobs; and

forward jobs to the respective execution clusters according the determined pending job queue length for the respective execution cluster.

9. The system of claim 8, wherein the logic is executable to:

determine a number of running jobs using a resource; and

determine a number of forwarded pending jobs requesting the resource.

10. The system of claim 9, wherein the logic is executable to:

determine an amount of the resource allocated to a running job; and

determine an amount of the resource requested by a forwarded pending job.

11. The system of claim 8, wherein the logic is executable to:

responsive to a job requesting a plurality of resources, determine a weight factor assigned to each resource; and

select an execution cluster to receive the job based on an availability of resources based on the weight factor of the respective resource.

12. The system of claim 8, wherein the logic is executable to:

assign a number of shares to each user submitting jobs to the submission cluster, each share representing a fraction of resources available from the execution clusters; and

dynamically determine a user priority for a user based on the shares assigned to the user and resource usage by the user.

13. The system of claim 8, wherein the logic is executable to:

determine if the submitted jobs include preference data indicating a preference of execution hosts for processing the jobs; and

responsive to the jobs including preference data, identify preferred execution clusters for processing the jobs based on the preference of execution hosts.

14. The system of claim 8, wherein the logic is executable to:

determine if the preference data includes a preference level; and

responsive to determining that the preference data includes a preference level, sort the preferred execution clusters based on the preference level.

15. A computer program product for job distribution within a grid environment, the computer program product comprising:

a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising computer readable program code configured to;

determine resource capacity corresponding to each execution cluster;

determine resource requirements for the jobs;

dynamically determine a pending job queue length for each execution cluster based on the resource capacity of the respective execution clusters and the resource requirements of the jobs; and

forward jobs to the respective execution clusters according the determined pending job queue length for the respective execution cluster.

16. The computer program product of claim 15, wherein the computer readable program code is configured to:

determine a number of running jobs using a resource; and

determine a number of forwarded pending jobs requesting the resource.

17. The computer program product of claim 16, wherein the computer readable program code is configured to:

determine an amount of the resource allocated to a running job; and

determine an amount of the resource requested by a forwarded pending job.

18. The computer program product of claim 15, wherein the computer readable program code is configured to:

responsive to a job requesting a plurality of resources, determine a weight factor assigned to each resource; and

select an execution cluster to receive the job based on an availability of resources based on the weight factor of the respective resource.

19. The computer program product of claim 15, wherein the computer readable program code is configured to:

assign a number of shares to each user submitting jobs to the submission cluster, each share representing a fraction of resources available from the execution clusters; and

dynamically determine a user priority for a user based on the shares assigned to the user and resource usage by the user.

20. The computer program product of claim 15, wherein the computer readable program code is configured to:

determine if the submitted jobs include preference data indicating a preference of execution hosts for processing the jobs; and

responsive to the jobs including preference data, identify preferred execution hosts for processing the jobs.