Inter-node communication scheme for sharing node operating status

• US 9,553,789 B2
• Filed: 06/25/2014
• Issued: 01/24/2017
• Est. Priority Date: 12/03/2010
• Status: Active Grant

• Current Assignee: Maplebear Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Ganapathy, Arunachalam, Mishra, Rajeev, Russell, Lance W., Vaddagiri, Murali
• Primary Examiner(s): Barry, Lance L
• Application Number:
  US14/314,146
• Publication Number:
  US 20140310410A1
• Time in Patent Office: 944 Days
• Field of Search: 709/224
• US Class Current: 1/1
• CPC Class Codes: H04L 43/0817 by checking functioning
• CPC Class Codes: H04L 43/10 Active monitoring, e.g. heartbeat, ping or trace-route

Abstract

A gossiping scheme for sharing node status in a cluster of nodes provides a robust mechanism for determining node status within the cluster. Nodes transmit gossip messages to each other nodes, the gossip messages listing other nodes in the cluster that are operational. When a node does not receive a gossip message from a particular node within a predetermined time period, then the node transmits messages to the other nodes indicating that the particular node is down. However, if another node has received a packet from the particular node within the predetermined time period and receives the node down message, then the other node responds with a node alive message.

Claims

1. A method for determining node operating status among a cluster of nodes of a computer system, the method comprising:

first transmitting gossip messages directly between node pairs in the cluster of nodes, wherein the gossip messages contain an indication of operating status of other nodes in the cluster of nodes, wherein the other nodes are nodes other than the nodes in the node pairs;

receiving the gossip messages at individual nodes of the node pairs;

responsive to the receiving the gossip messages at the individual nodes, at the other nodes, locally updating a local database of operating status according to the received gossip messages, wherein the updating sets a value of a local operating status kept by the individual nodes for a particular one of the other nodes to a non-operational status if the receiving by the individual nodes has not received a gossip message from the particular one of the other nodes during a predetermined time period;

responsive to the locally updating setting the local operating status of the particular one of the other nodes to a non-operational status, second transmitting a node down message separate from the gossip messages that indicates the non-operational status of the particular node to the other nodes in the cluster;

at a first node other than the particular node, receiving the node down message;

responsive to receiving the node down message, determining whether or not the first node has received a gossip message from the particular node during the predetermined time period;

responsive to determining that the first node has received the gossip message from the particular node during the predetermined time period, transmitting a node alive message from the first node indicating that the status of the particular node is operational and setting the local operating status of the particular node at the first node to an operational status; and

repeating the first transmitting, receiving, updating and second transmitting at each of the nodes in the node pairs, so that the local status kept by each of the nodes reflects the status of each of the other nodes in the cluster, wherein the first transmitting selectively transmits gossip messages containing an indication of operating status of other nodes depending on whether the operating status of the other nodes in the local database is set to non-operational, whereby the first transmitting halts gossip messaging for nodes marked non-operational.

2. The method of claim 1, wherein the node alive message is transmitted to multiple nodes in the cluster.

3. The method of claim 1, further comprising, at a second node, receiving the node alive message from the first node, wherein the updating, responsive to the receiving the node alive message from the first node, sets the status of the particular node at the second node to an operational status.

4. The method of claim 1, wherein the gossip messages contain sequence numbers of the most recent transmission received from the other nodes, and wherein the updating updates the sequence numbers in a local storage.

5. The method of claim 4, further comprising:

responsive to receiving at a first node a particular gossip message containing a first sequence number associated with a second node, determining whether the first sequence number is greater than a second sequence number of the most recently received message from the second node; and

responsive to determining that the first sequence number is greater than the second sequence number, transmitting a retransmit command to the second node.

6. The method of claim 1, wherein the first transmitting transmits gossip messages containing indications of operating status for nodes having an operating status set as operational at the nodes originating the gossip messages.