Reliable fault resolution in a cluster

• US 7,284,147 B2
• Filed: 08/27/2003
• Issued: 10/16/2007
• Est. Priority Date: 08/27/2003
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Jackson, Bruce M., Rao, Sudhir G., Sridhara, Srikanth N., Davis, Mark C.
• Primary Examiner(s): Wilson; Yolanda L.
• Application Number:
  US10/649,269
• Publication Number:
  US 20050050398A1
• Time in Patent Office: 1,511 Days
• Field of Search: None
• US Class Current: 714/4.3
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0757 by exceeding a time limit, i.e. time-out, e.g. watchdogs
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/2007 using redundant communication media
• CPC Class Codes: G06F 11/2017 where memory access, memory control or I/O control functionality is redundant redundant communication control functionality G06F11/2005; redundant storage control functionality G06F11/2089
• CPC Class Codes: H04L 41/0677 Localisation of faults
• CPC Class Codes: H04L 41/069 using logs of notifications; Post-processing of notifications
• CPC Class Codes: H04L 43/10 Active monitoring, e.g. heartbeat, ping or trace-route

Abstract

A method and system for localizing and resolving a fault in a cluster environment. The cluster is configured with at least one multi-homed node, and at least one gateway for each network interface. Heartbeat messages are sent between peer nodes and the gateway in predefined periodic intervals. In the event of loss of a heartbeat message by any node or gateway, an ICMP echo is issued to each node and gateway in the cluster for each network interface. If neither a node loss nor a network loss is validated in response to the ICMP echo, an application level ping is issued to determine if the fault associated with the absence of the heartbeat message is a transient error condition or an application software fault.

Claims

1. A method for fault resolution in a computer system, comprising:

configuring a cluster with a gateway for a network interface;

issuing an operating system ICMP echo to peer nodes in said cluster and to said gateway through said network interface in response to a heartbeat loss detection; and

analyzing a response from said echo to determine location of a fault in said cluster.

2. The method of claim 1, wherein the step of analyzing a response from said echo includes receiving said response and determining an intended recipient of said echo.

3. The method of claim 2, wherein receipt of a return of said echo from said peer nodes for said network interface within a predefined time interval is indicative of operation of said network interface.

4. The method of claim 2, wherein receipt of a return of said echo from said gateway for said network interface within a predefined time interval is indicative of operation of said network interface.

5. The method of claim 2, wherein absence of return of said echo from said peer nodes for said network interface within a predefined time interval is indicative of a fault selected from a group consisting of:

a peer node fault, a network fault local to the peer node, and combinations thereof.

6. The method of claim 1, further comprising the step of issuing an application level ping to a peer node in response to both receipt of said echo response within a predefined time interval and said heartbeat beat loss detection.

7. The method of claim 1, further comprising comparing an echo response from a target node set for each network interface.

8. The method of claim 7, wherein the step of comparing an echo response from a target node set for each network interface includes criteria selected from the group consisting of:

maximum availability of nodes in said network, availability of said gateway in said network, and combinations thereof.

9. A multi-node computer system, comprising:

a cluster with a gateway configured for a network interface;

an operating system ICMP echo adapted to be issued to peer nodes in a cluster and to said gateway through said network interface in response to a heartbeat loss detection; and

a response from said echo adapted to be analyzed for location of a fault in said cluster.

10. The system of claim 9, wherein analysis of said response from said echo includes determination of an intended recipient of said echo.

11. The system of claim 10, wherein receipt of a return of said echo from said peer nodes for said network interface within a predefined time interval is indicative of operation of said network interface.

12. The system of claim 10, wherein receipt of a return of said echo from said gateway for said network interface within a predefined time interval is indicative of operation of said network interface.

13. The system of claim 11, wherein absence of receipt of a return of said echo from peer nodes for said network interface within a predefined time interval is indicative of a fault selected from a group consisting of:

a peer node fault, a network fault local to the peer node, and combinations thereof.

14. The system of claim 9, further comprising an application level ping adapted to be issued to a peer node in response to both receipt of said echo within a predefined time interval and a heartbeat beat loss detection.

15. The system of claim 9, further comprising a comparison tool adapted to compare an echo response from a target node for each network interface.

16. The system of claim 15, wherein said comparison tool determines a network interface path based upon criteria selected from the group consisting of:

maximum availability of nodes in said network, availability of said gateway in said network, and combinations thereof.

17. An article comprising:

a computer-readable medium having computer-readable instructions stored thereon executable by a processor, said computer-readable instructions comprising;

computer-readable instructions for issuing an operating system ICMP echo to a peer node in a cluster and to a configured cluster gateway through said network interface in response to heartbeat loss detection; and

computer-readable instructions for analyzing a response message from said echo to determine location of a fault in said cluster.

18. The article of claim 17, wherein the medium is a recordable data storage medium.

19. The article of claim 17, wherein said computer-readable instructions for analyzing a response message from said echo includes receiving said response and determining an intended recipient of said echo.

20. The article of claim 17, further comprising computer-readable instructions for issuing an application level ping to a peer node in response to both receipt of an echo response within a predetermined time interval and a heartbeat loss detection.

21. The article of claim 17, further comprising computer-readable instructions for comparing an echo response from a target node set for each network interface.

22. The article of claim 21, wherein the instructions for comparing an echo response from a target node set for each network interfaces includes criteria selected from the group consisting of:

maximum availability of nodes in said network, availability of said gateway in said network, and combinations thereof.

23. A method for localizing a fault in a computer system, comprising:

sending periodic heartbeat messages to peer nodes in a network;

issuing an operating system ICMP echo to said peers nodes and a gateway through a network interface in response to a heartbeat loss; and

determining a location of a fault in said cluster through a response echo.

24. The method of claim 23, wherein the step of issuing an operating system ICMP echo includes sending said echo on a first network interface and a second network interface for multi-homed nodes.

25. The method of claim 23, wherein said loss is selected from a group consisting of:

a node loss, and network path loss.

26. The method of claim 23, further comprising comparing echo responses to determine a best path of connectivity.

27. The method of claim 23, further comprising localizing a network connectivity problem in response to return of at least one echo.