System and method for topology-aware job scheduling and backfilling in an HPC environment

• US 8,984,525 B2
• Filed: 10/11/2013
• Issued: 03/17/2015
• Est. Priority Date: 04/15/2004
• Status: Active Grant

• Current Assignee: Raytheon Company (Rtx Corporation)
• Original Assignee: Raytheon Company (Rtx Corporation)
• Inventors: Davidson, Shannon V., Richoux, Anthony N.
• Primary Examiner(s): Vo, Ted T
• Application Number:
  US14/052,093
• Publication Number:
  US 20140040912A1
• Time in Patent Office: 522 Days
• Field of Search: 717159-166, 718/104
• US Class Current: 718/104
• CPC Class Codes: G06F 11/1482 by means of middleware or OS functionality
• CPC Class Codes: G06F 11/2025 using centralised failover control functionality
• CPC Class Codes: G06F 2201/815 Virtual
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: G06F 9/50 Allocation of resources, e.g. of the central processing unit [CPU]
• CPC Class Codes: G06F 9/5038 considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration scheduling strategies G06F9/4881 and subgroups
• CPC Class Codes: G06F 9/5066 Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs mappping at compile time, see G06F8/451
• CPC Class Codes: G06F 9/5072 Grid computing
• CPC Class Codes: G06F 9/5077 Logical partitioning of resources; Management or configuration of virtualized resources specific details on emulation or internal functioning of virtual machines G06F9/455
• CPC Class Codes: G06F 9/5083 Techniques for rebalancing the load in a distributed system

Abstract

A method for job management in an HPC environment includes determining an unallocated subset from a plurality of HPC nodes, with each of the unallocated HPC nodes comprising an integrated fabric. An HPC job is selected from a job queue and executed using at least a portion of the unallocated subset of nodes.

Claims

1. A method, comprising:

receiving, by one or more computers, submission of a job from a user;

selecting a virtual cluster of a plurality of communicatively coupled nodes included in a computing environment, the virtual cluster associated with a group of users that submit similar jobs, and comprising a logical grouping of nodes configured to process related jobs, wherein the computing environment is configured to accommodate multiple virtual clusters therein;

retrieving a policy with one or more of the job and the user, and determining dimensions of the job to determine a job space within the selected virtual cluster, the job space comprising a set of nodes within the virtual cluster dynamically allocated to complete the job, wherein the virtual cluster is configured to accommodate multiple job spaces, with each job space being configured to concurrently execute a separate job;

determining whether there are a sufficient number of nodes available within the virtual cluster to allocate to the job space, and in the event sufficient nodes are not available within the virtual cluster, determining an earliest available subset of nodes in the virtual cluster on which to execute the job and adding the job to a job queue until the earliest available subset is available within the virtual cluster; and

upon a determination that a sufficient number of nodes are available within the virtual cluster, dynamically determining an optimum subset of nodes of the virtual cluster, allocating the subset for the job, and executing the job.

2. The method of claim 1, wherein the virtual cluster of a plurality of communicatively coupled nodes is selected responsive to verification of the user.

3. The method of claim 1, wherein the optimum subset of nodes is determined based on one or more of fastest processing time, most reliable nodes, physical locations, virtual locations, and first available nodes.

4. The method of claim 3, further comprising comparing the user to a group access control list (ACL) prior to selecting the virtual cluster.

5. The method of claim 4, wherein determining dimensions of the job comprises determining one or more of length, width and height.

6. The method of claim 1, wherein the plurality of communicatively coupled nodes included in the computing environment is arranged in a three dimensional node structure.

7. A system, comprising:

a plurality of communicatively coupled nodes of a computing environment; and

a management node configured to;

receive, by one or more computers, submission of a job from a user;

select a virtual cluster of the plurality of communicatively coupled nodes, the virtual cluster associated with a group of users that submit similar jobs, and comprising a logical grouping of nodes configured to process related jobs, wherein the computing environment is configured to accommodate multiple virtual clusters therein;

retrieve a policy with one or more of the job and the user, and determine dimensions of the job to determine a job space within the selected virtual cluster, the job space comprising a set of nodes within the virtual cluster dynamically allocated to complete the job, wherein the virtual cluster is configured to accommodate multiple job spaces, with each job space being configured to concurrently execute a separate job;

determine whether there are a sufficient number of nodes available within the virtual cluster to allocate to the job space, and in the event sufficient nodes are not available within the virtual cluster, determining an earliest available subset of nodes in the virtual cluster on which to execute the job and adding the job to a job queue until the earliest available subset is available within the virtual cluster; and

upon a determination that a sufficient number of nodes are available within the virtual cluster, dynamically determine an optimum subset of nodes of the virtual cluster, allocating the subset for the job, and executing the job.

8. The system of claim 7, wherein the virtual cluster of a plurality of communicatively coupled nodes is selected responsive to verification of the user.

9. The system of claim 7, wherein the optimum subset of nodes is determined based on one or more of fastest processing time, most reliable nodes, physical locations, virtual locations, and first available nodes.

10. The system of claim 9, wherein the management node is further configured to compare the user to a group access control list (ACL) prior to selecting the virtual cluster.

11. The system of claim 10, wherein determining dimensions of the job comprises determining one or more of length, width and height.

12. The system of claim 7, wherein the plurality of communicatively coupled nodes included in the computing environment is arranged in a three dimensional node structure.

13. A non-transitory, computer readable storage medium having computer readable instructions stored thereon that, when executed by a computer, implement a method, the method comprising:

receiving submission of a job from a user;

selecting a virtual cluster of a plurality of communicatively coupled nodes included in a computing environment, the virtual cluster associated with a group of users that submit similar jobs, and comprising a logical grouping of nodes configured to process related jobs, wherein the computing environment is configured to accommodate multiple virtual clusters therein;

retrieving a policy with one or more of the job and the user, and determining dimensions of the job to determine a job space within the selected virtual cluster, the job space comprising a set of nodes within the virtual cluster dynamically allocated to complete the job, wherein the virtual cluster is configured to accommodate multiple job spaces, with each job space being configured to concurrently execute a separate job;

determining whether there are a sufficient number of nodes available within the virtual cluster to allocate to the job space, and in the event sufficient nodes are not available within the virtual cluster, determining an earliest available subset of nodes in the virtual cluster on which to execute the job and adding the job to a job queue until the earliest available subset is available within the virtual cluster; and

upon a determination that a sufficient number of nodes are available within the virtual cluster, dynamically determining an optimum subset of nodes of the virtual cluster, allocating the subset for the job, and executing the job.

14. The computer readable storage medium of claim 13, wherein the virtual cluster of a plurality of communicatively coupled nodes is selected responsive to verification of the user.

15. The computer readable storage medium of claim 13, wherein the optimum subset of nodes is determined based on one or more of fastest processing time, most reliable nodes, physical locations, virtual locations, and first available nodes.

16. The computer readable storage medium of claim 15, further comprising comparing the user to a group access control list (ACL) prior to selecting the virtual cluster.

17. The computer readable storage medium of claim 16, wherein determining dimensions of the job comprises determining one or more of length, width and height.

18. The computer readable storage medium of claim 13, wherein the plurality of communicatively coupled nodes included in the computing environment is arranged in a three dimensional node structure.