Dynamic Rate Heartbeating for Inter-Node Status Updating

• US 20120144018A1
• Filed: 12/03/2010
• Published: 06/07/2012
• Est. Priority Date: 12/03/2010
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Fried, Eric P., Russell, Lance W.
• Granted Patent: US 8,667,126 B2
• Time in Patent Office: Days
• US Class Current: 709/224
• CPC Class Codes: H04L 12/6418 Hybrid transport

Abstract

A scheme for monitoring node operational status according to communications transmits messages periodically according to a heartbeat rate among the nodes. The messages may be gossip messages containing the status of the other nodes in the pairs, are received at the nodes and indications of the communications delays of the received messages are stored, which are used to compute statistics of the stored communications delays. Parameters of the node status monitoring, which are used for determining operational status of the nodes, are adjusted according to the statistics, which may include adjusting the heartbeat rate, the maximum wait time before a message is considered missed, and/or the maximum number of missed messages, e.g., the sequence number deviation, before the node is considered non-operational (down).

Claims

1-7. -7. (canceled)

8. A computer system comprising a processing cluster including a plurality of physical or virtual processing nodes, the computer system comprising at least one processor for executing program instructions and at least one memory coupled to the processor for executing the program instructions, wherein the program instructions are program instructions for determining node operating status among a cluster of the physical or virtual processing nodes, the program instructions comprising program instructions for:

transmitting messages periodically according to a heartbeat rate among nodes in the cluster of nodes;

receiving the messages at the nodes and storing indications of communications delays for the messages;

computing statistics of the communications delay for the messages;

adjusting parameters for node status monitoring according to the computed statistics; and

monitoring the operational status of the nodes according to the indications of communications delay and the parameters.

9. The computer system of claim 8, wherein the program instructions for transmitting transmit the messages directly between pairs of nodes of the cluster, whereby the program instructions for storing store indications of endpoint-to-endpoint communication delays.

10. The computer system of claim 8, wherein the program instructions for transmitting transmit gossip messages containing an indication of operational status of other nodes in the cluster.

11. The computer system of claim 8, wherein the program instructions for computing compute a mean and variance of the communications delays.

12. The computer system of claim 8, wherein the program instructions for adjusting parameters of the node status monitoring adjust a rate of the transmitting in conformity with the computed statistics.

13. The computer system of claim 8, wherein the program instructions for adjusting parameters of the node status monitoring adjust a maximum wait time used to determine whether a node is operational when one or more of the messages transmitted by the transmitting has not been received.

14. The computer system of claim 8, wherein the program instructions for adjusting parameters of the node status monitoring adjust a maximum number of missed receptions of the receiving used to determine whether a node is operational.

15. A computer program product comprising a computer-readable storage media storing program instructions for execution within a computer system, the computer system comprising a processing cluster including a plurality of physical or virtual processing modes, wherein the program instructions are program instructions for determining node operating status among a cluster of the physical or virtual processing nodes, the program instructions comprising program instructions for:

transmitting messages periodically according to a heartbeat rate among nodes in the cluster of nodes;

receiving the messages at the nodes and storing indications of communications delays for the messages;

computing statistics of the communications delay for the messages;

adjusting parameters for node status monitoring according to the computed statistics; and

monitoring the operational status of the nodes according to the indications of communications delay and the parameters.

16. The computer program product of claim 15, wherein the program instructions for transmitting transmit the messages directly between pairs of nodes of the cluster, whereby the program instructions for storing store indications of endpoint-to-endpoint communication delays.

17. The computer program product of claim 15, wherein the program instructions for transmitting transmit gossip messages containing an indication of operational status of other nodes in the cluster.

18. The computer program product of claim 15, wherein the program instructions for computing compute a mean and variance of the communications delays.

19. The computer program product of claim 15, wherein the program instructions for adjusting parameters of the node status monitoring adjust a rate of the transmitting in conformity with the computed statistics.

20. The computer program product of claim 15, wherein the program instructions for adjusting parameters of the node status monitoring adjust a maximum wait time used to determine whether a node is operational when one or more of the messages transmitted by the transmitting has not been received.

21. The computer program product of claim 15, wherein the program instructions for adjusting parameters of the node status monitoring adjust a maximum number of missed receptions of the receiving used to determine whether a node is operational.