System and method for topology-aware job scheduling and backfilling in an HPC environment

• US 20050235286A1
• Filed: 04/15/2004
• Published: 10/20/2005
• Est. Priority Date: 04/15/2004
• Status: Active Grant

• Current Assignee: Raytheon Company (Rtx Corporation)
• Original Assignee: Raytheon Company (Rtx Corporation)
• Inventors: Davidson, Shannon V., Ballew, James D., Early, Gary R.
• Granted Patent: US 8,336,040 B2
• Time in Patent Office: Days
• US Class Current: 718/100
• CPC Class Codes: G06F 11/1482 by means of middleware or OS functionality
• CPC Class Codes: G06F 11/2025 using centralised failover control functionality
• CPC Class Codes: G06F 2201/815 Virtual
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: G06F 9/50 Allocation of resources, e.g. of the central processing unit [CPU]
• CPC Class Codes: G06F 9/5038 considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration scheduling strategies G06F9/4881 and subgroups
• CPC Class Codes: G06F 9/5066 Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs mappping at compile time, see G06F8/451
• CPC Class Codes: G06F 9/5072 Grid computing
• CPC Class Codes: G06F 9/5077 Logical partitioning of resources; Management or configuration of virtualized resources specific details on emulation or internal functioning of virtual machines G06F9/455
• CPC Class Codes: G06F 9/5083 Techniques for rebalancing the load in a distributed system

Abstract

A method for job management in an HPC environment includes determining an unallocated subset from a plurality of HPC nodes, with each of the unallocated HPC nodes comprising an integrated fabric. An HPC job is selected from a job queue and executed using at least a portion of the unallocated subset of nodes.

Claims

1. A method for job management in an HPC environment comprising:

determining an unallocated subset from a plurality of HPC nodes, each of the unallocated HPC nodes comprising an integrated fabric;

selecting an HPC job from a job queue; and

executing the selected job using at least a portion of the unallocated subset of nodes.

2. The method of claim 1, wherein selecting the HPC job comprises selecting the HPC job from the job queue based on priority, the selected job comprising dimensions not greater than a topology of the unallocated subset.

3. The method of claim 2, wherein selecting the HPC job from the job queue based on priority comprises:

sorting the job queue based on job priority;

selecting a first HPC job from the sorted job queue;

determining dimensions of the first HPC job with the topology of the unallocated subset; and

in response to the dimensions of the first HPC job being greater than the topology of the unallocated subset, selecting a second HPC job from the sorted job queue.

4. The method of claim 2, wherein the dimensions of the first HPC job are based, at least in part, on one or more job parameters and an associated policy.

5. The method of claim 2, further comprising:

dynamically allocating a job spare from the unallocated subset based, at least in part, on the dimensions of the HPC job; and

wherein executing the selected job comprises executing the selected job using the dynamically allocated job spare.

6. The method of claim 1, the plurality of HPC nodes comprising a first plurality and the method further comprising:

determining that dimensions of the selected job are greater than a topology of the first plurality;

selecting one or more HPC nodes from a second plurality, each of the second HPC nodes comprising an integrated fabric; and

adding the selected second HPC nodes to the unallocated subset to satisfy the dimensions of the selected job.

7. The method of claim 6, further comprising returning the second HPC nodes to the second plurality.

8. The method of claim 1, further comprising;

determining that a second HPC job that was executing on a second subset in the plurality of HPC nodes has failed;

adding the second subset to the unallocated subset; and

adding the failed job to the job queue.

9. Software for job management in an HPC environment operable to:

determine an unallocated subset from a plurality of HPC nodes, each of the unallocated HPC nodes comprising an integrated fabric;

select an HPC job from a job queue; and

execute the selected job using at least a portion of the unallocated subset of nodes.

10. The software of claim 9, wherein the software operable to select the HPC job comprises software operable to select the HPC job from the job queue based on priority, the selected job comprising dimensions not greater than a topology of the unallocated subset.

11. The software of claim 10, wherein the software operable to select the HPC job from the job queue based on priority comprises software operable to:

sort the job queue based on job priority;

select a first HPC job from the sorted job queue;

determine dimensions of the first HPC job with the topology of the unallocated subset; and

in response to the dimensions of the first HPC job being greater than the topology of the unallocated subset, select a second HPC job from the sorted job queue.

12. The software of claim 10, wherein the dimensions of the first HPC job are based, at least in part, on one or more job parameters and an associated policy.

13. The software of claim 10, further operable to:

dynamically allocate a job spare from the unallocated subset based, at least in part, on the dimensions of the HPC job; and

wherein the software operable to execute the selected job comprises software operable to execute the selected job using the dynamically allocated job spare.

14. The software of claim 9, the plurality of HPC nodes comprising a first plurality and the software further operable to:

determine that dimensions of the selected job are greater than a topology of the first plurality;

select one or more HPC nodes from a second plurality, each of the second HPC nodes comprising an integrated fabric; and

add the selected second HPC nodes to the unallocated subset to satisfy the dimensions of the selected job.

15. The software of claim 14, further comprising returning the second HPC nodes to the second plurality.

16. The software of claim 9, further operable to:

determine that a second HPC job that was executing on a second subset in the plurality of HPC nodes has failed;

add the second subset to the unallocated subset; and

add the failed job to the job queue.

17. A system for job management in an HPC environment comprising:

a plurality of HPC nodes, each node including an integrated fabric; and

a management node operable to;

determine an unallocated subset from the plurality of HPC nodes;

select an HPC job from a job queue; and

execute the selected job using at least a portion of the unallocated subset of nodes.

18. The system of claim 17, wherein the management node operable to select the HPC job comprises the management node operable to select the HPC job from the job queue based on priority, the selected job comprising dimensions not greater than a topology of the unallocated subset.

19. The system of claim 18, wherein the management node operable to select the HPC job from the job queue based on priority comprises the management node operable to:

sort the job queue based on job priority;

select a first HPC job from the sorted job queue;

determine dimensions of the first HPC job with the topology of the unallocated subset; and

in response to the dimensions of the first HPC job being greater than the topology of the unallocated subset, select a second HPC job from the sorted job queue.

20. The system of claim 18, wherein the dimensions of the first HPC job are based, at least in part, on one or more job parameters and an associated policy.

21. The system of claim 18, further operable to:

dynamically allocate a job spare from the unallocated subset based, at least in part, on the dimensions of the HPC job; and

wherein the management node operable to execute the selected job comprises the management node operable to execute the selected job using the dynamically allocated job spare.

22. The system of claim 17, the plurality of HPC nodes comprising a first plurality and the management node further operable to:

determine that dimensions of the selected job are greater than a topology of the first plurality;

select one or more HPC nodes from a second plurality, each of the second HPC nodes comprising an integrated fabric; and

add the selected second HPC nodes to the unallocated subset to satisfy the dimensions of the selected job.

23. The system of claim 22, the management node further operable to return the second HPC nodes to the second plurality.

24. The system of claim 17, the management node further operable to:

determine that a second HPC job that was executing on a second subset in the plurality of HPC nodes has failed;

add the second subset to the unallocated subset; and

add the failed job to the job queue.