Dynamic modification of cluster communication parameters in clustered computer system

• US 6,983,324 B1
• Filed: 10/23/2000
• Issued: 01/03/2006
• Est. Priority Date: 10/23/2000
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Block, Timothy Roy, Thayib, Kiswanto
• Primary Examiner(s): Najjar, Saleh
• Assistant Examiner(s): BRUCKART, BENJAMIN R
• Application Number:
  US09/694,586
• Time in Patent Office: 1,898 Days
• Field of Search: 709/221, 709/227, 709/228, 709/204, 709/205, 709/232, 709/248, 709/249, 714/4
• US Class Current: 709/228
• CPC Class Codes: H04L 67/1001 for accessing one among a plurality of replicated servers
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06

Abstract

An apparatus, program product and method support the dynamic modification of cluster communication parameters through a distributed protocol whereby individual nodes locally confirm initiation and status information for every node participating in a parameter modification operation. By doing so, individual nodes are also able to locally determine the need to undo locally-performed parameter modifications should any other node be incapable of performing a parameter modification. Moreover, specifically with respect to cluster communication parameters such as heartbeat parameters, such parameters may be dynamically modified by configuring a sending node to send a heartbeat message to a receiving node, with the heartbeat message indicating that a heartbeat parameter is to be modified. In response to the heartbeat message, the receiving node may then send an acknowledgment message to the sending node that indicates whether the heartbeat parameter has been modified in the receiving node. Further, modification of the heartbeat parameter in the sending node may be deferred until the acknowledgment message from the receiving node indicates that the heartbeat parameter has been modified in the receiving node.

Claims

1. A method of dynamically modifying a cluster communication parameter in a clustered computer system, the method comprising:

(a) initiating a cluster communication parameter modification by transmitting a message to a plurality of nodes in the clustered computer system;

(b) locally confirming, within each node, receipt of the message by each of the plurality of nodes;

(c) in response to confirming receipt of the message by each of the plurality of nodes, invoking a local cluster communication parameter modification operation on each node;

(d) transmitting from each node a status of the local cluster communication parameter modification invoked on that node;

(e) locally detecting, within each node, an unsuccessful status for the local cluster communication parameter modification on any node; and

(f) in response to detecting an unsuccessful status for any node, locally undoing, in each node for which the local cluster communication operation was performed, the local cluster communication parameter modification operation performed on that node;

wherein the cluster communication parameter is selected from the group consisting of heartbeat message time out, heartbeat acknowledgment message time out, heartbeat frequency or interval, heartbeat failure threshold, heartbeat acknowledgment failure threshold, receive/send timer ratio, maximum fragment size, message retry timer value, maximum message retry time, send queue overflow threshold, message send window size, and combinations thereof.

2. The method of claim 1, wherein the cluster communication parameter comprises a heartbeat parameter.

3. The method of claim 1, wherein locally confirming receipt of the message by each of the plurality of nodes includes participating in an ACK round responsive to receipt of the message.

4. The method of claim 1, wherein transmitting from each node a status of the local cluster communication parameter modification invoked on that node is performed during an ACK round performed subsequent to invoking the local cluster communication parameter modification operation.

5. The method of claim 1, wherein transmitting the message, confirming receipt of the message, and transmitting the status are performed via multicast messages.

6. An apparatus, comprising:

(a) a memory; and

(b) a program resident in the memory, the program configured to dynamically modify a cluster communication parameter on a local node among a plurality of nodes in a clustered computer system, the program configured to locally confirm, for the local node, successful receipt of an initiation message by each of the plurality of nodes, and a status for a local cluster communication parameter modification operation performed by each of the plurality of nodes, the program further configured to undo a local cluster communication parameter modification operation performed on the local node in response to detection of an unsuccessful status for a local cluster communication parameter modification on any node;

wherein the cluster communication parameter is selected from the group consisting of heartbeat message time out, heartbeat acknowledgment message time out, heartbeat frequency or interval, heartbeat failure threshold, heartbeat acknowledgment failure threshold, receive/send timer ratio, maximum fragment size, message retry timer value, maximum message retry time, send queue overflow threshold, message send window size, and combinations thereof.

7. The apparatus of claim 6, wherein the program is further configured to locally confirm receipt of an initiating message by each of the plurality of nodes.

8. The apparatus of claim 7, wherein the program is configured to locally confirm receipt of the initiating message by each of the plurality of nodes by participating in an ACK round responsive to receipt of the message.

9. The apparatus of claim 6, wherein the program is further configured to transmit from the local node a status of the local cluster communication parameter modification operation.

10. The apparatus of claim 9, wherein the program is configured to transmit the status during an ACK round performed subsequent to invocation of the local cluster communication parameter modification operation.

11. A clustered computer system, comprising:

(a) a plurality of nodes coupled to one another over a network; and

(b) a plurality of programs, each local to a node among the plurality of nodes, each program configured to dynamically modify a cluster communication parameter on its respective local node, each program further configured to locally confirm, for its respective local node, successful receipt of an initiation message by each of the plurality of nodes, and a status for a local cluster communication parameter modification operation performed by each of the plurality of nodes, and each program further configured to undo a local cluster communication parameter modification operation performed on its respective local node in response to detection of an unsuccessful status for a local cluster communication parameter modification on any node;

wherein the cluster communication parameter is selected from the group consisting of heartbeat message time out, heartbeat acknowledgment message time out, heartbeat frequency or interval, heartbeat failure threshold, heartbeat acknowledgment failure threshold, receive/send timer ratio, maximum fragment size, message retry timer value, maximum message retry time, send queue overflow threshold, message send window size, and combinations thereof.

12. A program product, comprising:

(a) a program configured to dynamically modify a cluster communication parameter on a local node among a plurality of nodes in a clustered computer system, the program configured to locally confirm, for the local node, successful receipt of an initiation message by each of the plurality of nodes, and a status for a local cluster communication parameter modification operation performed by each of the plurality of nodes, the program further configured to undo a local cluster communication parameter modification operation performed on the local node in response to detection of an unsuccessful status for a local cluster communication parameter modification on any node; and

(b) a tangible computer readable medium bearing the program;

wherein the cluster communication parameter is selected from the group consisting of heartbeat message time out, heartbeat acknowledgment message time out, heartbeat frequency or interval, heartbeat failure threshold, heartbeat acknowledgment failure threshold, receive/send timer ratio, maximum fragment size, message retry timer value, maximum message retry time, send queue overflow threshold, message send window size, and combinations thereof.

13. The program product of claim 12, wherein the program is further configured to locally confirm receipt of an initiating message by each of the plurality of nodes.

14. The program product of claim 13, wherein the program is configured to locally confirm receipt of the initiating message by each of the plurality of nodes by participating in an ACK round responsive to receipt of the message.

15. The program product of claim 12, wherein the program is further configured to transmit from the local node a status of the local cluster communication parameter modification operation.

16. The program product of claim 15, wherein the program is configured to transmit the status during an ACK round performed subsequent to invocation of the local cluster communication parameter modification operation.

17. A method of dynamically modifying a heartbeat parameter in a node among a plurality of nodes in a clustered computer system, the plurality of nodes including first and second nodes, the first node configured to send a heartbeat message to the second node, and the second node configured to send an acknowledgment message to the first node in response to receiving the heartbeat message, the method comprising:

(a) sending a heartbeat message from the first node to the second node, the heartbeat message indicating that a heartbeat parameter is to be modified; and

(b) deferring modification of the heartbeat parameter in the first node until receipt of an acknowledgment message sent from the second node to the first node that indicates that the heartbeat parameter has been modified in the second node;

wherein the heartbeat parameter is selected from the group consisting of heartbeat message time out, heartbeat acknowledgment message time out, heartbeat frequency or interval, heartbeat failure threshold, heartbeat acknowledgment failure threshold, receive/send timer ratio, and combinations thereof.

18. The method of claim 17, further comprising determining whether modifying the heartbeat parameter on the first node requires synchronization with the second node.

19. The method of claim 18, wherein determining whether modifying the heartbeat parameter on the first node requires synchronization with the second node further comprises determining whether the heartbeat parameter is local or global in nature.

20. The method of claim 17, further comprising, in response to receiving the heartbeat message with the second node, sending an acknowledgment message from the second node to the first node, the acknowledgment message indicating whether the heartbeat parameter has been modified in the second node.

21. The method of claim 20, wherein each of sending the heartbeat message and sending the heartbeat acknowledgment message includes accessing a heartbeat message record that includes a change request indicator, the method further comprising:

(a) prior to sending the heartbeat message that indicates that the heartbeat parameter is to be modified, setting the change request indicator in the heartbeat message record; and

(b) prior to sending the heartbeat acknowledgment message that indicates whether the heartbeat parameter has been modified in the second node, selectively setting or clearing the change request indicator in the heartbeat message record.

22. The method of claim 21, wherein deferring modification of the heartbeat parameter in the first node until the acknowledgment message indicates that the heartbeat parameter has been modified in the second node includes modifying the heartbeat parameter in the first node only after receiving a heartbeat acknowledgment message with a set change request indicator.

23. The method of claim 17, further comprising:

(a) modifying the heartbeat parameter in the second node; and

(b) modifying the heartbeat parameter in the first node after receipt of an acknowledgment message sent from the second node to the first node that indicates that the heartbeat parameter has been modified in the second node.

24. An apparatus, comprising:

(a) a memory; and

(b) a program resident in the memory and configured to dynamically modify a heartbeat parameter in a first node among a plurality of nodes in a clustered computer system by sending a heartbeat message to a second node among the plurality of nodes that indicates that the heartbeat parameter is to be modified and thereafter deferring modification of the heartbeat parameter in the first node only after receiving an acknowledgment message from the second node indicating that the heartbeat parameter has been modified in the second node;

wherein the heartbeat parameter is selected from the group consisting of heartbeat message time out, heartbeat acknowledgment message time out, heartbeat frequency or interval, heartbeat failure threshold, heartbeat acknowledgment failure threshold, receive/send timer ratio, and combinations thereof.

25. The apparatus of claim 24, wherein the program is further configured to determine whether modifying the heartbeat parameter on the first node requires synchronization with the second node.

26. The apparatus of claim 25, wherein the program is configured to determine whether modifying the heartbeat parameter on the first node requires synchronization with the second node by determining whether the heartbeat parameter is local or global in nature.

27. The apparatus of claim 24, wherein the program is configured to send the heartbeat message by accessing a heartbeat message record that includes a change request indicator, and wherein the program is further configured to set the change request indicator in the heartbeat message record prior to sending the heartbeat message that indicates that the heartbeat parameter is to be modified.

28. The apparatus of claim 27, wherein the program is configured to defer modification of the heartbeat parameter in the first node until the acknowledgment message indicates that the heartbeat parameter has been modified in the second node by modifying the heartbeat parameter in the first node only after receiving a heartbeat acknowledgment message with a set change request indicator.

29. A program product, comprising:

(a) a program configured to dynamically modify a heartbeat parameter in a first node among a plurality of nodes in a clustered computer system by sending a heartbeat message to a second node among the plurality of nodes that indicates that the heartbeat parameter is to be modified and thereafter deferring modification of the heartbeat parameter in the first node only after receiving an acknowledgment message from the second node indicating that the heartbeat parameter has been modified in the second node; and

(b) a tangible computer readable medium bearing the program;

wherein the heartbeat parameter is selected from the group consisting of heartbeat message time out, heartbeat acknowledgment message time out, heartbeat frequency or interval, heartbeat failure threshold, heartbeat acknowledgment failure threshold, receive/send timer ratio, and combinations thereof.