Maintaining process group membership for node clusters in high availability computing systems

• US 7,627,694 B2
• Filed: 03/16/2001
• Issued: 12/01/2009
• Est. Priority Date: 03/16/2000
• Status: Expired due to Fees

• Current Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Original Assignee: Silicon Graphics Incorporated (Graphics Properties Holdings, Inc.)
• Inventors: Mallena, Raghu, Srivastava, Sharad, Castellano, Luca, Bradshaw, Robert David, Verma, Manish, Pramanick, Ira, Nishimoto, Michael, Dandapani, Ajit, Sreenivasan, Padmanabhan
• Primary Examiner(s): Refai, Ramsey
• Application Number:
  US09/811,158
• Publication Number:
  US 20020049845A1
• Time in Patent Office: 3,182 Days
• Field of Search: 709/226, 709/239, 709/228, 709/251, 709/252, 709/249, 395/200, 395/500, 714/4
• US Class Current: 709/251
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06

Abstract

A high availability computing system includes a plurality of computer nodes (for example, a server system) connected by a first and a second network, wherein the computer nodes communicate with each other to detect server failure and transfer applications to other computer nodes on detecting server failure. The system incorporates methods of maintaining high availability in a server cluster having a plurality of nodes. A group communications service, a membership service and a system resource manager are instantiated on each node and the group communications service, the membership service and the system resource manager on each node communicate with other nodes to detect node failures and to transfer applications to other nodes on detecting node failure.

Claims

1. A computing system comprising:

a plurality of nodes connected by a network;

a cluster membership service operating on the plurality of nodes, the cluster membership service operable to determine membership in a cluster by exchanging messages, wherein a message originating from a node includes a node data area defining the node'"'"'s view of the cluster relationships and wherein the message includes a checkmark data structure in which each node receiving the message sets the checkmark data structure according to whether the receiving node confirms the relationship defined in the node data area;

a group membership service operable to determine membership in a group of nodes formed by a subset of nodes in the cluster of a process executing on a node in the plurality of nodes the group of nodes for an application distributed across two or more of the nodes in the group, said membership communicated between the two or more nodes in the group utilizing a proposal message sent by a coordinator node for receipt by each node in the group and a commit message sent by the coordinator node to each node in the group after receiving acknowledgement that the proposal message has reached each node of the group, and further wherein the plurality of nodes in the group communicate with each other to detect a failure of an application in the group on a first node of the cluster and to transfer applications from the first node to other nodes of the plurality of nodes in the group on detecting the failure.

4. The computing system of claim 1, wherein the plurality of nodes includes an initiator node to send the proposal message to the coordinator node.

5. The computing system of claim 4, wherein the coordinator node comprises an oldest node in the plurality of nodes.

6. The computing system of claim 4, wherein the plurality of nodes are arranged in a network ring, the order of the plurality of nodes in the network ring being defined by a cluster membership age of each node in the plurality of nodes and wherein the coordinator node forwards the proposal message to a first node of the plurality of nodes, and wherein the proposal message is forwarded by a receiving node in the network ring to a successor node of the receiving node.

7. The computing system of claim 6, wherein the coordinator node issues the commit message upon receiving the proposal message from a non-initiator node in the network ring.

2. A method of maintaining high availability in a server cluster having a plurality of nodes, the method comprising:

determining membership by a cluster membership service in a cluster by exchanging messages, wherein a message originating from a node includes a node data area defining the node'"'"'s view of the cluster relationships and wherein the message includes a checkmark data structure in which each node receiving the message sets the checkmark data structure according to whether the receiving node confirms the relationship defined in the node data area;

instantiating a group communications service, a group membership service and a system resource manager on each node of the plurality of nodes, the plurality of nodes forming a group;

communicating process membership in the group utilizing a proposal message sent by a coordinator node for receipt by each node in the plurality of nodes and a commit message sent by the coordinator node to each node in the plurality of nodes after receiving acknowledgement that the proposal message has reached each node of the plurality of nodes;

communicating between the group communications service, the group membership service and the system resource manager on each node of the group to detect process failures and node failures within the group;

upon detecting a failure in a process on a first node of the group, transferring applications to other nodes of the group; and

updating, by the group membership service, process membership in a distributed application upon detecting a process failure on a node of the group.

8. The method of claim 2, wherein communicating the proposal message includes sending by an initiator node the proposal message to the coordinator node.

9. The method of claim 8, wherein the coordinator node comprises an oldest node in the plurality of nodes.

10. The method of claim 8, further comprising:

arranging the plurality of nodes in a network ring;

forwarding by the coordinator node the proposal message to a first node of the plurality of nodes; and

forwarding by the first node to a next node in the network ring.

11. The method of claim 10, wherein the coordinator node issues the commit message upon receiving the proposal message from a non-initiator node in the ring.

12. The method of claim 10, wherein upon receiving the commit message a node of the plurality of nodes in the network ring performs the tasks of:

caching the commit message;

forwarding the commit message to a next node in the network ring;

upon completing forwarding the commit message delivering the commit message to each process of a process group on the node.

3. A computer-readable medium having instructions stored thereon, wherein the instructions, when executed in a computer, perform operations comprising:

determining membership by a cluster membership service in a cluster by exchanging messages, wherein a message originating from a node includes a node data area defining the node'"'"'s view of the cluster relationships and wherein the message includes a checkmark data structure in which each node receiving the message sets the checkmark data structure according to whether the receiving node confirms the relationship defined in the node data area;

instantiating a group communications service, a group membership service and a system resource manager on each node of a plurality of nodes, the plurality of nodes forming a group;

communicating process membership in the group utilizing a proposal message including data defining one or more relationships between the plurality of nodes sent by a coordinator node for receipt by each node in the plurality of nodes and a commit message sent by the coordinator node to each node in the plurality of nodes after receiving acknowledgement that the proposal message has reached each node of the plurality of nodes;

communicating between the group communications service, the group membership service and the system resource manager on each node of the group to detect process failures and node failures within the group;

upon detecting a failure in a process on a first node of the group, transferring applications to other nodes of the group; and

updating, by the group membership service, process membership in a distributed application upon detecting a process failure on a node of the group.

13. The computer readable medium of claim 3, wherein communicating the proposal message includes sending by an initiator node the proposal message to the coordinator node.

14. The computer readable medium of claim 13, wherein the coordinator node comprises an oldest node in the plurality of nodes.

15. The computer readable medium of claim 13, wherein the method further comprises:

arranging the plurality of nodes in a network ring;

forwarding by the coordinator node the proposal message to a first node of the plurality of nodes; and

forwarding by the first node to a next node in the network ring.

16. The computer readable medium of claim 15, wherein the coordinator node issues the commit message upon receiving the proposal message from a non-initiator node in the ring.

17. The computer readable medium of claim 15, wherein upon receiving the commit message a node of the plurality of nodes in the network ring performs the tasks of:

caching the commit message;

forwarding the commit message to a next node in the network ring;

upon forwarding the commit message delivering the commit message to each process of a process group on the node.