System and method for cluster management based on HPC architecture

• US 9,832,077 B2
• Filed: 11/02/2015
• Issued: 11/28/2017
• Est. Priority Date: 04/15/2004
• Status: Active Grant

• Current Assignee: Raytheon Company (Rtx Corporation)
• Original Assignee: Raytheon Company (Rtx Corporation)
• Inventors: Ballew, James D., Davidson, Shannon V., Richoux, Anthony N.
• Primary Examiner(s): GREENE, JOSEPH L
• Application Number:
  US14/930,050
• Publication Number:
  US 20160057016A1
• Time in Patent Office: 757 Days
• Field of Search: 709223
• CPC Class Codes: G06F 2209/505 Clust
• CPC Class Codes: G06F 9/5077 Logical partitioning of resources; Management or configuration of virtualized resources specific details on emulation or internal functioning of virtual machines G06F9/455
• CPC Class Codes: H04L 41/046 comprising network management agents or mobile agents therefor
• CPC Class Codes: H04L 41/0893 Assignment of logical groups to network elements
• CPC Class Codes: H04L 41/0894 Policy-based network configuration management
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 43/0817 by checking functioning
• CPC Class Codes: H04L 63/101 Access control lists [ACL]
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06

Abstract

Cluster management software comprises a plurality of cluster agents, with each cluster agent associated with an HPC node including an integrated fabric and the cluster agent operable to determine a status of the associated HPC node. The software further includes a cluster management engine communicably coupled with the plurality of the HPC nodes and operable to execute an HPC job using a dynamically allocated subset of the plurality of HPC nodes based on the determined status of the plurality of HPC nodes.

Claims

1. A method, comprising:

identifying, by one or more first hardware processors of a cluster management engine, which nodes in a virtual cluster are available, the virtual cluster including a plurality of communicatively coupled nodes, each node including a cluster agent in communication with the cluster management engine, and the virtual cluster comprising a logical grouping of nodes configured to process jobs;

identifying, by one or more of the one or more first hardware processors or one or more second hardware processors, a job of the jobs that is compatible with the identified available nodes including;

selecting the job from a job queue;

identifying a shape that sufficiently matches a shape of the job, the shape indicating a specific topology of a sub-cluster of nodes from the cluster of nodes suitable to execute the selected job;

identifying one or more shapes of the available nodes including one or more of (1) a cube in which nodes of the available nodes are allocated in a logical cubic volume so as to allow tasks of the job to exchange data with other tasks of the job so as to minimize a distance between nodes that exchange data, and (2) a sphere in which nodes of the available nodes are allocated in a logical spherical volume so as to allow a first task of the job to be placed in a center node of the sphere with remaining tasks of the job placed on nodes surrounding the center node so as to minimize a distance between the first task and the remaining tasks; and

determining whether the available nodes are sufficient to execute the job based on the identified shape that sufficiently matches the shape of the job and the one or more identified shapes of the available nodes;

in response to determining whether the available nodes are sufficient, allocating, by the one or more of the one or more first hardware processors, the one or more second hardware processors, or one or more third hardware processors, a plurality of the available nodes for the job that sufficiently fit the identified shape of the job; and

executing the job using the allocated nodes.

2. The method of claim 1, further comprising:

in response to allocating one or more of the available nodes for the job, re-identifying which nodes in the virtual cluster are available.

3. The method of claim 2, further comprising:

in response to determining the available nodes are not sufficient to execute the job, selecting a next job in the job queue and identifying a shape of the next job, determining whether the available nodes are sufficient to execute the next job based on the determined shape, and in response to determining the available nodes are sufficient to execute the next job, allocating one or more of the available nodes for the next job.

4. The method of claim 3, further comprising:

in response to executing the job, selecting a next job in the job queue and identifying a shape of the next job, determining whether the number of available nodes are sufficient to execute the next job based on the determined shape, and in response to determining the available nodes are sufficient to execute the next job, allocating one or more of the available nodes for the next job.

5. The method of claim 1, wherein the job queue is initially sorted based on a job priority.

6. The method of claim 1, wherein the shape is based on dimensions of the job, including one or more of length, width and height.

7. The method of claim 1, wherein the nodes are logically arranged in a three dimensional node structure.

8. A system comprising:

a cluster management engine;

a plurality of communicatively coupled nodes, each node comprising a processing device and a cluster agent, each of the nodes in communication with a cluster management engine; and

the cluster management engine configured to;

identify which nodes in a virtual cluster are available, the virtual cluster comprising a logical grouping of nodes configured to process related jobs;

identify a job that is compatible with the available nodes including the cluster management engine configured to;

select the job from a job queue;

identify a shape that sufficiently matches a shape of the job, the shape indicating a specific topology of a sub-cluster of nodes from the cluster of nodes suitable to execute the selected job;

identify one or more shapes of the available nodes, the shape including one or more of (1) a cube in which nodes of the available nodes are allocated in a cubic volume so as to allow tasks of the job to exchange data with other tasks of the job so as to minimize the distance between nodes that exchange data, and (2) a sphere in which nodes of the available nodes are allocated in a spherical volume so as to allow a first task of the job to be placed in a center node of the sphere with remaining tasks of the job placed on nodes surrounding the center node so as to minimize the distance between the first task and the remaining tasks; and

determine whether the available nodes are sufficient to execute the job based on the identified shape that sufficiently matches the shape of the job and the one or more identified shapes of the available nodes;

the cluster management engine further configured to, in response to determining the available nodes are sufficient, allocate a plurality of the available nodes for the job that sufficiently fit the identified shape of the job; and

the allocated nodes of the virtual cluster are configured to execute the job.

9. The system of claim 8, wherein the cluster management engine is further configured to:

in response to allocating one or more of the available nodes for the job, re-identify which nodes in the virtual cluster are available.

10. The system of claim 9, wherein the cluster management engine is further configured to:

in response to determining the available nodes are not sufficient to execute the job, select a next job in the job queue, identify a shape of the next job, determine whether the available nodes are sufficient to execute the next job based on the shape, and in response to determining the available nodes are sufficient, allocate one or more of the available nodes for the next job.

11. The system of claim 9, wherein the cluster management engine is further configured to:

in response to executing the job, select a next job in the job queue, identify a shape of the next job, determine whether the available nodes are sufficient to execute the next job based on the shape, and in response to determining the number of available nodes are sufficient, allocate one or more of the available nodes for the next job.

12. The system of claim 8, wherein the job queue is initially sorted based on a job priority.

13. The system of claim 8, wherein the shape is based on dimensions of the job including one or more of length, width and height.

14. The system of claim 8, wherein the nodes are logically arranged in a three dimensional node structure.

15. A non-transitory computer readable storage device including instructions stored thereon which, when executed by a machine, configure the machine to:

identify which nodes in a virtual cluster are available, the virtual cluster including a plurality of communicatively coupled nodes, each node including a cluster agent in communication with the cluster management engine, and the virtual cluster comprising a logical grouping of nodes configured to process jobs;

identify a job of the jobs that is compatible with the available nodes in the virtual cluster including;

select the job from a job queue;

select the job from a job queue;

identify a shape that sufficiently matches a shape of the job, the shape indicating a specific topology of a sub-cluster of nodes from the cluster of nodes suitable to execute the selected job;

identify one or more shapes of the available nodes, the shape including one or more of (1) a cube in which nodes of the available nodes are allocated in a cubic volume so as to allow tasks of the job to exchange data with other tasks of the job so as to minimize the distance between nodes that exchange data, and (2) a sphere in which nodes of the available nodes are allocated in a spherical volume so as to allow a first task of the job to be placed in a center node of the sphere with remaining tasks of the job placed on nodes surrounding the center node so as to minimize the distance between the first task and the remaining tasks; and

determine whether the available nodes are sufficient to execute the job based on the identified shape that sufficiently matches the shape of the job and one or more identified shapes of the available nodes;

in response to determining the available nodes are sufficient, allocate a plurality of he available nodes for the job that sufficiently fit the identified shape of the job; and

execute the job using the allocated nodes.

16. The storage device of claim 15, further comprising instructions which, when executed by the machine, configure the machine to:

in response to allocating the available nodes for the job, re-identify which nodes in the virtual cluster are available.

17. The storage device of claim 16, further comprising instructions which, when executed by the machine, configure the machine to:

in response to determining the available nodes are not sufficient to execute the job, select a next job in the job queue and identify a shape of the next job, determine whether the available nodes are sufficient to execute the next job based on the identified shape, and in response to determining the one or more available nodes are sufficient to execute the next job, allocating one or more of the available nodes for the next job.

18. The storage device of claim 16, further comprising instructions which, when executed by the machine, configure the machine to:

in response to executing the job, selecting a next job in the job queue and identifying a shape of the next job, determining whether the available nodes are sufficient to execute the next job based on the identified shape, and in response to determining the available nodes are sufficient to execute the next job, allocating one or more of the available nodes for the next job.

19. The storage device of claim 15, wherein the job queue is initially sorted based on a job priority.

20. The storage device of claim 15, wherein the shape is based on dimensions of the job, including one or more of length, width and height.