Inter-node communication scheme for node status sharing

• US 8,433,760 B2
• Filed: 12/03/2010
• Issued: 04/30/2013
• Est. Priority Date: 12/03/2010
• Status: Expired due to Fees

• Current Assignee: Maplebear Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Ganapathy, Arunachalam, Mishra, Rajeev, Russell, Lance W., Vaddagiri, Murali
• Primary Examiner(s): Pham, Chi
• Assistant Examiner(s): CHOWDHURY, FAHMIDA S
• Application Number:
  US12/959,581
• Publication Number:
  US 20120143957A1
• Time in Patent Office: 879 Days
• Field of Search: 709/205, 709/465, 709/224, 709/202, 370/216, 370/248, 370/236, 370/338
• US Class Current: 709/205
• CPC Class Codes: H04L 43/0817 by checking functioning
• CPC Class Codes: H04L 43/10 Active monitoring, e.g. heartbeat, ping or trace-route

Abstract

A gossiping scheme for sharing node status in a cluster of nodes provides a robust mechanism for determining node status within the cluster. Nodes transmit gossip messages to each other nodes, the gossip messages listing other nodes in the cluster that are operational. When a node does not receive a gossip message from a particular node within a predetermined time period, then the node transmits messages to the other nodes indicating that the particular node is down. However, if another node has received a packet from the particular node within the predetermined time period and receives the node down message, then the other node responds with a node alive message.

Claims

1. A computer system comprising a processing cluster including a plurality of physical or virtual processing nodes, the computer system comprising at least one processor for executing program instructions and at least one memory coupled to the processor for executing the program instructions, wherein the program instructions are program instructions for determining node operating status among a cluster of the physical or virtual processing nodes, the program instructions comprising program instructions for:

first transmitting gossip messages directly between node pairs in the cluster of nodes, wherein the gossip messages contain an indication of operational status of other nodes in the cluster of nodes, wherein the other nodes are nodes other than the nodes in the node pairs;

receiving the gossip messages at the node pairs;

responsive to the receiving, at the individual nodes, locally updating a local operating status of the other nodes according to the received gossip messages, wherein the local status kept by the individual nodes indicates the status of a particular one of the other nodes as a non-operating status if the receiving by the individual nodes has not received a gossip message from the particular one of the other nodes during a predetermined time period;

responsive to setting the local status of the particular one of the other nodes as kept by the individual nodes to a non-operating status, second transmitting a node down message indicating the non-operating status of the particular node to the other nodes in the cluster, and

repeating the first transmitting, receiving, updating and second transmitting at each of the nodes in the node pairs, so that the local status kept by each of the nodes reflects the status of each of the other nodes in the cluster.

2. The computer system of claim 1, wherein the program instructions further comprise program instructions for:

at a first node other than the particular node, receiving the node down message;

responsive to receiving the node down message, determining whether or not the first node has received a gossip message from the particular node during the predetermined time period; and

responsive to determining that the first node has received the gossip message from the particular node during the predetermined time period, transmitting a node alive message from the first node indicating that the particular node is operating.

3. The computer system of claim 2, wherein the program instructions for transmitting the node alive message transmit the node alive message to multiple nodes in the cluster.

4. The computer system of claim 2, further comprising program instructions for, at a second node, receiving the node alive message from the first node, wherein the program instructions for updating, responsive to the receiving the node alive message from the first node, set the status of the particular node at the second node to an operating status.

5. The computer system of claim 1, wherein the gossip messages contain sequence numbers of the most recent transmission received from the other nodes, and wherein the program instructions for updating update the sequence numbers in a local storage.

6. The computer system of claim 5, further comprising program instructions for:

responsive to receiving at a first node a particular gossip message containing a first sequence number associated with a second node, determining whether the first sequence number is greater than a second sequence number of the most recently received message from the second node; and

responsive to determining that the first sequence number is greater than the second sequence number, transmitting a retransmit command to the second node.

7. The computer system of claim 1, wherein the program instructions for first transmitting transmit gossip messages containing indications of operational status for nodes having an operational status set as operating at the nodes originating the gossip messages.

8. A computer program product comprising a computer-readable storage media that is not a signal, the storage media storing program instructions for execution within a computer system, the computer system comprising a processing cluster including a plurality of physical or virtual processing modes, wherein the program instructions are program instructions for determining node operating status among a cluster of the physical or virtual processing nodes, the program instructions comprising program instructions for:

first transmitting gossip messages directly between node pairs in the cluster of nodes, wherein the gossip messages contain an indication of operational status of other nodes in the cluster of nodes, wherein the other nodes are nodes other than the nodes in the node pairs;

receiving the gossip messages at the node pairs;

responsive to the receiving, at the individual nodes, locally updating a local operating status of the other nodes according to the received gossip messages, wherein the local status kept by the individual node indicates the status of a particular one of the other nodes as a non-operating status if the receiving by the individual nodes has not received a gossip message from the particular one of the other nodes during a predetermined time period;

responsive to setting the local status of the particular one of the other nodes as kept by the individual nodes to a non-operating status, second transmitting a node down message indicating the non-operating status of the particular node to the other nodes in the cluster; and

repeating the first transmitting, receiving, updating and second transmitting at each of the nodes in the node pairs, so that the local status kept by each of the nodes reflects the status of each of the other nodes in the cluster.

9. The computer program product of claim 8, wherein the program instructions further comprise program instructions for:

at a first node other than the particular node, receiving the node down message;

responsive to receiving the node down message, determining whether or not the first node has received a gossip message from the particular node during the predetermined time period; and

responsive to determining that the first node has received the gossip message from the particular node during the predetermined time period, transmitting a node alive message from the first node indicating that the particular node is operating.

10. The computer program product of claim 9, wherein the program instructions for transmitting the node alive message transmit the node alive message to multiple nodes in the cluster.

11. The computer program product of claim 9, further comprising program instructions for, at a second node, receiving the node alive message from the first node, wherein the program instructions for updating, responsive to the receiving the node alive message from the first node, set the status of the particular node at the second node to an operating status.

12. The computer program product of claim 8, wherein the gossip messages contain sequence numbers of the most recent transmission received from the other nodes, and wherein the program instructions for updating update the sequence numbers in a local storage.

13. The computer program product of claim 12, further comprising program instructions for:

responsive to receiving at a first node a particular gossip message containing a first sequence number associated with a second node, determining whether the first sequence number is greater than a second sequence number of the most recently received message from the second node; and

responsive to determining that the first sequence number is greater than the second sequence number, transmitting a retransmit command to the second node.

14. The computer program product of claim 8, wherein the program instructions for first transmitting transmit gossip messages containing indications of operational status for nodes having an operational status set as operating at the nodes originating the gossip messages.