System and method for topology-aware job scheduling and backfilling in an HPC environment

• US 10,621,009 B2
• Filed: 12/27/2017
• Issued: 04/14/2020
• Est. Priority Date: 04/15/2004
• Status: Active Grant

• Current Assignee: Raytheon Company (Rtx Corporation)
• Original Assignee: Raytheon Company (Rtx Corporation)
• Inventors: Davidson, Shannon V., Richoux, Anthony N.
• Primary Examiner(s): Vo, Ted T.
• Application Number:
  US15/855,554
• Publication Number:
  US 20180121253A1
• Time in Patent Office: 839 Days
• Field of Search: 717159-165, 717176-177, 718103, 718104
• CPC Class Codes: G06F 11/1482 by means of middleware or OS functionality
• CPC Class Codes: G06F 11/2025 using centralised failover control functionality
• CPC Class Codes: G06F 2201/815 Virtual middleware or OS functionality using virtual machines to implement generic software techniques for error detection or fault masking G06F11/1484
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: G06F 9/50 Allocation of resources, e.g. of the central processing unit [CPU]
• CPC Class Codes: G06F 9/5038 considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration scheduling strategies G06F9/4881 and subgroups
• CPC Class Codes: G06F 9/5066 Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs mappping at compile time, see G06F8/451
• CPC Class Codes: G06F 9/5072 Grid computing
• CPC Class Codes: G06F 9/5077 Logical partitioning of resources; Management or configuration of virtualized resources specific details on emulation or internal functioning of virtual machines G06F9/455
• CPC Class Codes: G06F 9/5083 Techniques for rebalancing the load in a distributed system

Abstract

A method for job management in an HPC environment includes determining an unallocated subset from a plurality of HPC nodes, with each of the unallocated HPC nodes comprising an integrated fabric. An HPC job is selected from a job queue and executed using at least a portion of the unallocated subset of nodes.

Claims

1. A method comprising:

determining, using one or more processors, a node of a first job space of a plurality of communicatively coupled nodes has failed, the first job space including a logical grouping of nodes configured to process a job and communicatively coupled to accommodate a logical shape specified by at least one of a job policy and a job parameter;

determining whether the failed node is allocated to execute the job;

responsive to a determination that the failed node is allocated to execute the job, identifying operational nodes, other than the failed node, in the first job space;

terminating a task of the job on each of the operational nodes executing the job;

deallocating the operational nodes from the job;

identifying a subset of nodes of the plurality of communicatively coupled nodes, other than the failed node, communicatively coupled in the logical shape;

allocating the identified subset of nodes to execute the job in a second job space; and

executing the job on the allocated second job space.

2. The method of claim 1, wherein identifying the subset of nodes further comprises:

retrieving the job policy for the job, the job policy detailing processing and management information including one or more of;

job size, job run time, timeslots, and users'"'"' allocation of nodes of the communicatively coupled nodes; and

identifying the subset of nodes based on the retrieved job policy.

3. The method of claim 1, wherein the identified subset of nodes is a completely different subset of nodes than the plurality of communicatively coupled nodes of the first job space.

4. The method of claim 1, wherein the plurality of communicatively coupled nodes are arranged in a three-dimensional node structure.

5. The method of claim 1, wherein each node comprises at least two first processors operable to communicate with each other, and a first switch, the first processors communicably coupled to the first switch, the first switch operable to communicably couple the first processors to at least six second nodes.

6. The method of claim 1, wherein determining the failure of the node comprises one of:

determining that a repeating communication is not received from the failed node; and

polling of the failed node.

7. The method of claim 1, wherein the job parameter identifies a shape of the job space.

8. A system comprising:

a plurality of communicatively coupled nodes; and

a management node, comprising at least one processing device, configured to;

determine, using one or more processors, a node of a first job space of a plurality of communicatively coupled nodes has failed, the first job space including a logical grouping of nodes configured to process a job and communicatively coupled to accommodate a logical shape specified by at least one of a job policy and a job parameter;

determine whether the failed node is allocated to execute the job;

responsive to a determination that the failed node is allocated to execute the job, identify operational nodes, other than the failed node, in the first job space;

terminate a task of the job on each of the operational nodes executing the job;

deallocate the operational nodes from the job;

identify a subset of nodes of the plurality of communicatively coupled nodes, other than the failed node, communicatively coupled in the logical shape;

allocate the identified subset of nodes to execute the job in a second job space; and

execute the job on the allocated second job space.

9. The system of claim 8, wherein the management node is further configured to:

retrieve the job policy, the job policy detailing processing and management information including one or more of;

problem size, problem run time, timeslots, and users'"'"' allocation of nodes; and

identify the subset of nodes based on the job policy.

10. The system of claim 8, wherein the identified subset of nodes is a completely different subset of nodes than the plurality of communicatively coupled nodes of the first job space.

11. The system of claim 8, wherein the plurality of communicatively coupled nodes are arranged in a three-dimensional node structure.

12. The system of claim 8, wherein each node comprises at least two processors operable to communicate with each other, and a switch, the processors communicably coupled to the switch, the switch operable to communicably couple the processors to at least six other nodes.

13. The system of claim 8, wherein determining the failure of the node comprises one of:

determining that a repeating communication is not received from the failed node; and

polling of the failed node.

14. A non-transitory computer readable storage medium having computer readable instructions stored thereon that, when executed by a computer, implement a method, the method comprising:

determining a node of a first job space of a plurality of communicatively coupled nodes has failed, the first job space including a logical grouping of nodes configured to process a job and communicatively coupled to accommodate a logical shape specified by at least one of a job policy and a job parameter;

determining whether the failed node is allocated to execute the job;

responsive to a determination that the failed node is allocated to execute the job, identifying operational nodes, other than the failed node, in the first job space;

terminating a task of the job on each of the operational nodes executing the job;

deallocating the operational nodes from the job;

identifying a subset of nodes of the plurality of communicatively coupled nodes, other than the failed node, communicatively coupled in the logical shape;

allocating the identified subset of nodes to execute the job in a second job space; and

executing the job on the allocated second job space.

15. The storage medium of claim 14, wherein determining the optimum subset of nodes further comprises:

retrieving the job policy, the job policy detailing processing and management information including one or more of;

problem size, problem run time, timeslots, and users'"'"' allocation of nodes; and

identify the subset of nodes based on the retrieved job policy.

16. The storage medium of claim 14, wherein the identified subset of nodes is a completely different subset of nodes than nodes in the plurality of communicatively coupled nodes of the first job space.

17. The storage medium of claim 14, wherein the plurality of communicatively coupled nodes are arranged in a three-dimensional node structure.

18. The storage medium of claim 14, wherein each node comprises at least two processors operable to communicate with each other, and a switch, the processors communicably coupled to the switch, the switch operable to communicably couple the processors to at least six other nodes.

19. The storage medium of claim 14, wherein determining the failure of the node comprises one of:

determining that a repeating communication is not received from the failed node; and

polling of the failed node.

20. The storage medium of claim 14, wherein the job parameter identifies a shape of the job space.