Method and apparatus for exchanging heartbeat messages and configuration information between nodes operating in a master-slave configuration

• US 7,421,478 B1
• Filed: 03/07/2002
• Issued: 09/02/2008
• Est. Priority Date: 03/07/2002
• Status: Expired due to Term

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Muchow, James D.
• Primary Examiner(s): Thomson; William
• Assistant Examiner(s): ANYA, CHARLES E
• Application Number:
  US10/094,552
• Time in Patent Office: 2,371 Days
• Field of Search: 709201-250, 714 1- 57, 719313-318, 719/311
• US Class Current: 709/209
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0724 in a multiprocessor or a multi-core unit multiprocessors per se G06F15/80
• CPC Class Codes: G06F 11/0757 by exceeding a time limit, i.e. time-out, e.g. watchdogs
• CPC Class Codes: G06F 11/1482 by means of middleware or OS functionality
• CPC Class Codes: G06F 11/202 where processing functionality is redundant redundant communication control functionality G06F11/2005, redundant storage control functionality G06F11/2089
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: G06F 2209/505 Clust
• CPC Class Codes: G06F 9/44505 Configuring for program initiating, e.g. using registry, configuration files
• CPC Class Codes: G06F 9/5061 Partitioning or combining of resources
• CPC Class Codes: H04L 41/0266 using meta-data, objects or commands for formatting management information, e.g. using eXtensible markup language [XML]
• CPC Class Codes: H04L 41/0654 using network fault recovery ring fault isolation or reconfiguration in loop networks without recovery actions by a network management system H04L12/437
• CPC Class Codes: H04L 41/082 the condition being updates or upgrades of network functionality
• CPC Class Codes: H04L 41/0853 by actively collecting configuration information or by backing up configuration information
• CPC Class Codes: H04L 41/0869 Validating the configuration within one network element
• CPC Class Codes: H04L 43/0811 by checking connectivity
• CPC Class Codes: H04L 43/10 Active monitoring, e.g. heartbeat, ping or trace-route
• CPC Class Codes: H04L 67/025 for remote control or remote monitoring of applications
• CPC Class Codes: H04L 67/34 involving the movement of software or configuration parameters network booting or remote initial program loading [RIPL] G06F9/4416
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06

Abstract

A node (101, FIG. 1), within a networked computer system (100), is capable of supporting communications with other nodes relating to operating multiple application instances in a master-slave configuration. Each node periodically generates and sends (520, 526, FIG. 5) a Heartbeat message (FIG. 8) that indicates the operational status for one or more application instances being managed by the node. When a node receives a Heartbeat message from a remote node, it evaluates (FIG. 10) the Heartbeat information for each application instance reported in the message, and takes any appropriate actions. The node also determines (1206, FIG. 12) whether new configuration information should be obtained for each of the application instances the node is managing, and requests (1210, FIG. 12) that new configuration information, when necessary.

Claims

1. A method, performed by a node, for supporting communications between nodes that are connected by one or more networks and which operate in a master-slave configuration, the method comprising:

generating, by the node, a heartbeat message, which has a format that allows multiple heartbeat segments to be included within the heartbeat message, wherein each heartbeat segment of the multiple heartbeat segments is respectively associated with an application instance being managed by the node and includes one or more fields including a field identifying the associated application instance; and

sending the heartbeat message to one or more other nodes over the one or more networks to be processed by the one or more other nodes.

2. The method as claimed in claim 1, wherein generating the heartbeat message comprises generating the heartbeat message with a heartbeat segment that identifies an application instance in a master state.

3. The method as claimed in claim 1, wherein generating the heartbeat message comprises generating the heartbeat message with a heartbeat segment that identifies an application instance in a slave state.

4. The method as claimed in claim 1, wherein generating the heartbeat message comprises including a field, in a heartbeat segment, that identifies a configuration associated with the application instance.

5. The method as claimed in claim 1, wherein generating the heartbeat message comprises generating the heartbeat message with a single heartbeat segment after a master application instance has been promoted to a master state, wherein the single heartbeat segment identifies the master application instance.

6. The method as claimed in claim 1, further comprising:

determining whether each heartbeat segment within the heartbeat message would be the same as previously sent heartbeat segments; and

if each heartbeat segment would be substantially the same, sending the previously sent heartbeat segments.

7. The method as claimed in claim 1, further comprising:

determining whether a problem has occurred with a master application instance in the master state; and

if a problem has occurred, not including a heartbeat segment for the master application instance in the heartbeat message.

8. The method as claimed in claim 1, wherein the one or more networks include multiple networks, and wherein sending the heartbeat message comprises alternating which of the one or more networks the node sends the heartbeat message out over.

9. A method, performed by a local node, for supporting communications between nodes that are connected by one or more networks and which operate in a master-slave configuration, the method comprising:

receiving, by the local node, a heartbeat message from a remote node, the heartbeat message has a format that allows multiple heartbeat segments to be included within the heartbeat message, wherein each heartbeat segment of the multiple heartbeat segments is respectively associated with an application instance being managed by the node and includes one or more fields including a field identifying the associated application instance; and

processing a heartbeat segment of the heartbeat message based on the field identifying the application instance being managed by the remote node.

10. The method as claimed in claim 9, wherein processing the heartbeat segment comprises:

sending a quit message to the remote node when the remote node considers itself to be a master for the application instance, and the local node considers itself to be the master for the application instance.

11. The method as claimed in claim 9, wherein processing the heartbeat segment comprises:

restarting a timer associated with the application instance when the local node considers itself to be a slave for the application instance.

12. The method as claimed in claim 9, wherein processing the heartbeat segment comprises:

determining, from the heartbeat segment, whether the local node has current configuration information for the application instance; and

if the local node does not have the current configuration information, indicating that new configuration information is needed.

13. The method as claimed in claim 9, wherein processing the heartbeat segment comprises:

storing a node identifier for the remote node when the local node considers itself to be a master for the application instance associated with the heartbeat segment.

14. A method, performed by a first node, for supporting communications between nodes that are connected by one or more networks and which operate in a master-slave configuration, the method comprising:

generating, by the first node, a heartbeat message, which has a format that allows multiple heartbeat segments to be included within the heartbeat message, wherein a heartbeat segment includes a field identifying a current configuration associated with an application instance being managed by the first node; and

sending the heartbeat message to one or more other nodes over the one or more networks to be processed by the one or more other nodes.

15. The method as claimed in claim 14, wherein generating the heartbeat message comprises generating the heartbeat message with an indication of the current configuration for some or all application instances.

16. A method, performed by a local node, for supporting communications between nodes that are connected by one or more networks and which operate in a master-slave configuration, the method comprising:

generating, by the local node, a heartbeat message, which has a format that allows multiple heartbeat segments to be included within the heartbeat message, wherein the multiple heartbeat segments correspond to multiple application instances being managed by the local node and wherein each heartbeat segment includes one or more fields including a field identifying a corresponding application instance and a field identifying a current configuration associated with the corresponding application instance;

sending the heartbeat message to one or more remote nodes over the one or more networks;

receiving, by the remote node, the heartbeat message; and

processing, by the remote node, each heartbeat segment of the heartbeat message.

17. The method as claimed in claim 16, wherein generating the heartbeat message comprises generating the heartbeat message with a heartbeat segment that identifies an application instance in a slave state.

18. The method as claimed in claim 16, wherein generating the heartbeat message comprises generating the heartbeat message with a heartbeat segment that identifies an application instance in a master state.

19. The method as claimed in claim 16, wherein processing each heartbeat segment comprises:

determining, from a heartbeat segment, whether the remote node has current configuration information for the application instance; and

if the remote node does not have the current configuration information, indicating that new configuration information is needed.

20. A node within a computer system which operates in a master-slave configuration, the node comprising:

at least one processor that generates a heartbeat message, which has a format that allows multiple heartbeat segments to be included within the heartbeat message, wherein each heartbeat segment of the multiple heartbeat segments is respectively associated with an application instance being managed by the node and includes one or more fields including a field identifying the associated application instance; and

at least one interface to at least one external network, which sends the heartbeat message to one or more other nodes over at least one external network to be processed by the one or more other nodes.

21. The node as claimed in claim 20, wherein the at least one processor generates the heartbeat message with a single heartbeat segment after a master application instance has been promoted to a master state, wherein the single heartbeat segment identifies the master application instance.

22. The node as claimed in claim 20, wherein the at least one processor generates the heartbeat message with a field, in each heartbeat segment, that identifies a configuration associated with the application instance.

23. A local node within a computer system which operates in a master-slave configuration, the local node comprising:

at least one interface to at least one external network, wherein the at least one external network interconnects the local node and one or more remote nodes, and the at least one interface receives a heartbeat message from a remote node, which has a format that allows multiple heartbeat segments to be included within the heartbeat message, wherein each heartbeat segment of the multiple heartbeat segments is respectively associated with an application instance being managed by the node and includes one or more fields including a field identifying the associated application instance; and

at least one processor, which processes a heartbeat segment of the heartbeat message.

24. The local node as claimed in claim 23, wherein the at least one processor processes the heartbeat segment by sending a quit message to the remote node when the remote node considers itself to be a master for the application instance, and the local node considers itself to be the master for the application instance.

25. The local node as claimed in claim 23, wherein the at least one processor processes the heartbeat segment by restarting a timer associated with the application instance when the local node considers itself to be a slave for the application instance.

26. The local node as claimed in claim 23, wherein the at least one processor processes the heartbeat segment by:

determining, from the heartbeat segment, whether the local node has current configuration information for the application instance; and

if the local node does not have the current configuration information, indicating that new configuration information is needed.

27. A computer readable medium having computer executable instructions stored thereon for performing a method of supporting communications between a local node and one or more other nodes operating in a master-slave configuration, the method comprising:

generating, by the local node, a first heartbeat message, which has a format that allows multiple heartbeat segments to be included within the heartbeat message, wherein each heartbeat segment includes a field identifying an application instance being managed by the local node;

sending the first heartbeat message to one or more remote nodes over the one or more networks;

receiving, by the local node, a second heartbeat message from a remote node; and

processing a heartbeat segment of the second heartbeat message, wherein the heartbeat segment includes a field identifying an application instance being managed by the remote node.

28. The method as claimed in claim 27, wherein generating the first heartbeat message comprises generating the first heartbeat message with a heartbeat segment that identifies an application instance in a slave state.

29. The method as claimed in claim 27, wherein generating the first heartbeat message comprises generating the first heartbeat message with a heartbeat segment that identifies an application instance in a master state.

30. The method as claimed in claim 27, wherein processing the heartbeat segment comprises:

determining, from the heartbeat segment, whether the local node has current configuration information for the application instance being managed by the remote node; and

if the local node does not have the current configuration information, indicating that new configuration information is needed.

31. A node within a computer system which operates in a master-slave configuration, the node comprising:

means for processing, which generates a local heartbeat message that has a format that allows multiple heartbeat segments to be included within the local heartbeat message, wherein a heartbeat segment includes one or more fields to identify and to identify a current configuration associated with the application instance, and the means for processing also processes a remote heartbeat message from a remote node; and

means for interfacing with at least one external network, wherein the at least one external network interconnects the node and one or more other nodes, and wherein the means for interfacing sends the local heartbeat message to the one or more other nodes over the one or more networks, and receives the remote heartbeat message from the remote node.

32. The node as claimed in claim 31, wherein the means for processing generates the local heartbeat message by generating the local heartbeat message with a heartbeat segment that identifies an application instance in a slave state.

33. The node as claimed in claim 31, wherein the means for processing generates the local heartbeat message by generating the local heartbeat message with a heartbeat segment that identifies an application instance in a master state.

34. The node as claimed in claim 31, wherein the means for processing processes the remote heartbeat message by:

determining, from a remote heartbeat segment, whether the local node has current configuration information for an application instance identified by the remote heartbeat segment; and

if the local node does not have the current configuration information, indicating that new configuration information is needed.