System and method for detecting and managing HPC node failure

• US 20050246569A1
• Filed: 04/15/2004
• Published: 11/03/2005
• Est. Priority Date: 04/15/2004
• Status: Active Grant

• Current Assignee: Raytheon Company (Rtx Corporation)
• Original Assignee: Raytheon Company (Rtx Corporation)
• Inventors: Davidson, Shannon V., Ballew, James D., Early, Gary R.
• Granted Patent: US 7,711,977 B2
• Time in Patent Office: Days
• US Class Current: 714/4
• CPC Class Codes: G06F 11/202 where processing functionality is redundant redundant communication control functionality G06F11/2005, redundant storage control functionality G06F11/2089
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: H04L 49/358 Infiniband Switches

Abstract

A method for managing HPC node failure includes determining that one of a plurality of HPC nodes has failed, with each HPC node comprising an integrated fabric. The failed node is then removed from a virtual list of HPC nodes, with the virtual list comprising one logical entry for each of the plurality of HPC nodes.

Claims

1. A method for managing HPC node failure comprising:

determining that one of a plurality of HPC nodes has failed, each HPC node comprising an integrated fabric; and

removing the failed node from a virtual list of HPC nodes, the virtual list comprising one logical entry for each of the plurality of HPC nodes.

2. The method of claim 1, further comprising:

determining that at least a portion of an HPC job was being executed on the failed node; and

terminating the HPC job.

3. The method of claim 2, further comprising:

determining that the HPC job was associated with a subset of the plurality of HPC nodes; and

deallocating the subset of HPC nodes.

4. The method of claim 3, each entry of the virtual list comprising a node status and the method further comprising changing the status of each of the subset of HPC nodes to “

available”

.

5. The method of claim 3, further comprising:

determining dimensions of the terminated job based on one or more job parameters and an associated policy;

dynamically allocating a second subset of the plurality of HPC nodes based on the determined dimensions; and

executing the terminated job on the allocated second subset.

6. The method of claim 5, the second subset comprising a substantially similar set of nodes to the first subset.

7. The method of claim 5, wherein dynamically allocating the second subset comprises:

determining an optimum subset of nodes from a topology of unallocated HPC nodes; and

allocating the optimum subset.

8. The method of claim 1, further comprising:

locating a replacement HPC node for the failed HPC node; and

updating the logical entry of the failed HPC node with information on the replacement HPC node.

9. The method of claim 1, wherein determining one of the plurality of HPC nodes has failed comprises determining that a repeating communication has not been received from the failed node.

10. The method of claim 1, wherein determining one of the plurality of HPC nodes has failed is accomplished through polling.

11. Software for managing HPC node failure operable to:

determine that one of a plurality of HPC nodes has failed, each node comprising an integrated fabric; and

remove the failed node from a virtual list of HPC nodes, the virtual list comprising one logical entry for each of the plurality of HPC nodes.

12. The software of claim 11, further operable to:

determine that at least a portion of an HPC job was being executed on the failed node; and

terminate the HPC job.

13. The software of claim 12, further operable to:

determine that the HPC job was associated with a subset of the plurality of HPC nodes; and

deallocate the subset of HPC nodes.

14. The software of claim 13, each entry of the virtual list comprising a node status and the software further operable to change the status of each of the subset of HPC nodes to “

available”

.

15. The software of claim 13, further operable to:

determine dimensions of the terminated job based on one or more job parameters and an associated policy;

dynamically allocate a second subset of the plurality of HPC nodes based on the determined dimensions; and

execute the terminated job on the allocated second subset.

16. The software of claim 15, the second subset comprising a substantially similar set of nodes to the first subset.

17. The software of claim 15, wherein the software operable to dynamically allocate the second subset comprises software operable to:

determine an optimum subset of nodes from a topology of unallocated HPC nodes; and

allocate the optimum subset.

18. The software of claim 11, further operable to:

locate a replacement HPC node for the failed HPC node; and

update the logical entry of the failed HPC node with information on the replacement HPC node.

19. The software of claim 11, wherein the software operable to determine one of the plurality of HPC nodes has failed comprises software operable to determine that a repeating communication has not been received from the failed node.

20. The software of claim 11, wherein the software operable to determine one of the plurality of HPC nodes has failed is accomplished through polling.

21. A system for managing HPC node failure comprising:

a plurality of HPC nodes, each node including an integrated fabric; and

a management node operable to;

determine that one of the plurality of HPC nodes has failed, each node comprising an integrated fabric; and

remove the failed node from a virtual list of HPC nodes, the virtual list comprising one logical entry for each of the plurality of HPC nodes.

22. The system of claim 21, the management node further operable to:

determine that at least a portion of an HPC job was being executed on the failed node; and

terminate the HPC job.

23. The system of claim 22, the management node further operable to:

determine that the HPC job was associated with a subset of the plurality of HPC nodes; and

deallocate the subset of HPC nodes.

24. The system of claim 23, each entry of the virtual list comprising a node status and the management node further operable to change the status of each of the subset of HPC nodes to “

available”

.

25. The system of claim 23, the management node further operable to:

determine dimensions of the terminated job based on one or more job parameters and an associated policy;

dynamically allocate a second subset of the plurality of HPC nodes based on the determined dimensions; and

execute the terminated job on the allocated second subset.

26. The system of claim 25, the second subset comprising a substantially similar set of nodes to the first subset.

27. The system of claim 25, wherein the management node operable to dynamically allocate the second subset comprises the management node operable to:

determine an optimum subset of nodes from a topology of unallocated HPC nodes; and

allocate the optimum subset.

28. The system of claim 21, the management node further operable to:

locate a replacement HPC node for the failed HPC node; and

update the logical entry of the failed HPC node with information on the replacement HPC node.

29. The system of claim 21, wherein the management node operable to determine one of the plurality of HPC nodes has failed comprises the management node operable to determine that a repeating communication has not been received from the failed node.

30. The system of claim 21, wherein the management node operable to determine one of the plurality of HPC nodes has failed is accomplished through polling.