Reliable Fault Resolution In A Cluster

• US 20100115338A1
• Filed: 07/05/2007
• Published: 05/06/2010
• Est. Priority Date: 08/27/2003
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Jackson, Bruce M., Rao, Sudhir G., Sridhara, Srikanth N., Davis, Mark C.
• Granted Patent: US 7,941,690 B2
• Time in Patent Office: Days
• US Class Current: 714/37
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0757 by exceeding a time limit, i.e. time-out, e.g. watchdogs
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/2007 using redundant communication media
• CPC Class Codes: G06F 11/2017 where memory access, memory control or I/O control functionality is redundant redundant communication control functionality G06F11/2005; redundant storage control functionality G06F11/2089
• CPC Class Codes: H04L 41/0677 Localisation of faults
• CPC Class Codes: H04L 41/069 using logs of notifications; Post-processing of notifications
• CPC Class Codes: H04L 43/10 Active monitoring, e.g. heartbeat, ping or trace-route

Abstract

A method and system for localizing and resolving a fault in a cluster environment. The cluster is configured with at least one multi-homed node, and at least one gateway for each network interface. Heartbeat messages are sent between peer nodes and the gateway in predefined periodic intervals. In the event of loss of a heartbeat message by any node or gateway, an ICMP echo is issued to each node and gateway in the cluster for each network interface. If neither a node loss not a network loss is validated in response to the ICMP echo, an application level ping is issued to determine if the fault associated with the absence of the heartbeat message is a transient error condition or an application software fault.

Claims

1. A method for resolving a fault in a computer system comprising:

determining a heartbeat loss in a cluster configured with a gateway for a network interface;

validating said heartbeat loss; and

localizing said loss by analyzing a response echo.

2. The method of claim 1, wherein the step of validating said heartbeat loss includes sending an ICMP echo to peer nodes and said gateway through a network interface.

3. The method of claim 1, further comprising determining a best path of connectivity through a comparison of echo response.

4. An article comprising:

a computer-readable medium having computer-readable instructions stored thereon executable by a processor, said computer-readable instructions comprising;

instructions to determine a heartbeat loss in a cluster configured with a gateway for a network interface;

instructions to validate said heartbeat loss; and

instructions to localize said loss through analysis of a response echo.

5. The article of claim 4, wherein the instructions to validate said heartbeat loss includes instructions to send an ICMP echo to peer nodes and said gateway through said network interface.

6. The article of claim 4, further comprising instructions to determine a best path of connectivity through a comparison of said echo response.

7. A computer system, comprising:

a cluster of multi-homed nodes;

a gateway in communication with said cluster and configured for a network interface;

an operating system ICMP echo adapted to be issued to peer nodes in said cluster and to said gateway through said network interface in response to a heartbeat loss detection; and

a response from said echo adapted to be analyzed for location of a fault in said cluster.

8. The system of claim 7, wherein analysis of said response from said echo includes determination of an intended recipient of said echo.

9. The system of claim 8, wherein receipt of a return of said echo from said peer nodes for said network interface within a predefined time interval is indicative of operation of said network interface.

10. The system of claim 8, wherein receipt of a return of said echo from said gateway for said network interface within a predefined time interval is indicative of operation of said network interface.

11. The system of claim 9, wherein absence of receipt of a return of said echo from peer nodes for said network interface within a predefined time interval is indicative of a fault selected from a group consisting of:

a peer node fault, a network fault local to the peer node, and combinations thereof.

12. The system of claim 7, further comprising an application level ping adapted to be issued to a peer node in response to both receipt of said echo within a predefined time interval and a heartbeat beat loss detection.

13. The system of claim 7, further comprising a comparison tool adapted to compare an echo response from a target node for each network interface.

14. The system of claim 13, wherein said comparison tool determines a network interface path based upon criteria selected from the group consisting of:

maximum availability of nodes in said network, availability of said gateway in said network, and combinations thereof.