Method and apparatus for exchanging heartbeat messages and configuration information between nodes operating in a master-slave configuration

• US 7,856,480 B2
• Filed: 05/15/2007
• Issued: 12/21/2010
• Est. Priority Date: 03/07/2002
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Muchow, James D.
• Primary Examiner(s): Hu; Jinsong
• Application Number:
  US11/748,966
• Publication Number:
  US 20080288607A1
• Time in Patent Office: 1,316 Days
• Field of Search: 709/208, 709/209, 709/223, 709/224
• US Class Current: 709/208
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0724 in a multiprocessor or a multi-core unit multiprocessors per se G06F15/80
• CPC Class Codes: G06F 11/0757 by exceeding a time limit, i.e. time-out, e.g. watchdogs
• CPC Class Codes: G06F 11/1482 by means of middleware or OS functionality
• CPC Class Codes: G06F 11/202 where processing functionality is redundant redundant communication control functionality G06F11/2005, redundant storage control functionality G06F11/2089
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: G06F 2209/505 Clust
• CPC Class Codes: G06F 9/44505 Configuring for program initiating, e.g. using registry, configuration files
• CPC Class Codes: G06F 9/5061 Partitioning or combining of resources
• CPC Class Codes: H04L 41/0266 using meta-data, objects or commands for formatting management information, e.g. using eXtensible markup language [XML]
• CPC Class Codes: H04L 41/0654 using network fault recovery ring fault isolation or reconfiguration in loop networks without recovery actions by a network management system H04L12/437
• CPC Class Codes: H04L 41/082 the condition being updates or upgrades of network functionality
• CPC Class Codes: H04L 41/0853 by actively collecting configuration information or by backing up configuration information
• CPC Class Codes: H04L 41/0869 Validating the configuration within one network element
• CPC Class Codes: H04L 43/0811 by checking connectivity
• CPC Class Codes: H04L 43/10 Active monitoring, e.g. heartbeat, ping or trace-route
• CPC Class Codes: H04L 67/025 for remote control or remote monitoring of applications
• CPC Class Codes: H04L 67/34 involving the movement of software or configuration parameters network booting or remote initial program loading [RIPL] G06F9/4416
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06

Abstract

A node (101, FIG. 1), within a networked computer system (100), is capable of supporting communications with other nodes relating to operating multiple application instances in a master-slave configuration. Each node periodically generates and sends (520, 526, FIG. 5) a Heartbeat message (FIG. 8) that indicates the operational status for one or more application instances being managed by the node. When a node receives a Heartbeat message from a remote node, it evaluates (FIG. 10) the Heartbeat information for each application instance reported in the message, and takes any appropriate actions. The node also determines (1206, FIG. 12) whether new configuration information should be obtained for each of the application instances the node is managing, and requests (1210, FIG. 12) that new configuration information, when necessary.

Claims

1. A non-transitory machine-readable storage medium comprising instructions, which when implemented by one or more machines that cause the one or more machines to perform the following operations:

generating, by a node, a heartbeat message, which has a format that allows multiple heartbeat segments to be included within the heartbeat message, wherein each heartbeat segment of the multiple heartbeat segments is respectively associated with an application instance being managed by the node and includes one or more fields including a field identifying the associated application instance; and

sending the heartbeat message to one or more other nodes over one or more networks to be processed by the one or more other nodes.

2. The non-transitory machine-readable storage medium comprising instructions of claim 1, wherein generating the heartbeat message comprises generating the heartbeat message with a heartbeat segment that identifies an application instance in a master state.

3. The non-transitory machine-readable storage medium comprising instructions of claim 1, wherein generating the heartbeat message comprises generating the heartbeat message with a heartbeat segment that identifies an application instance in a slave state.

4. The non-transitory machine-readable storage medium comprising instructions of claim 1, wherein generating the heartbeat message includes a field, in a heartbeat segment, that identifies a configuration associated with the application instance.

5. The non-transitory machine-readable storage medium comprising instructions of claim 1, wherein generating the heartbeat message comprises generating the heartbeat message with a single heartbeat segment after a master application instance has been promoted to a master state, wherein the single heartbeat segment identifies the master application instance.

6. The non-transitory machine-readable storage medium comprising instructions of claim 1, wherein generating the heartbeat message comprises generating the heartbeat message with multiple heartbeat segments.

7. The non-transitory machine-readable storage medium comprising instructions of claim 1, wherein generating the heartbeat message comprises generating each heartbeat segment with a field that identifies the application instance associated with the heartbeat segment.

8. The non-transitory machine-readable storage medium comprising instructions of claim 7, wherein generating the heartbeat message comprises generating each heartbeat segment with a second field that identifies a current configuration associated with the application instance.

9. The non-transitory machine-readable storage medium comprising instructions of claim 1, further comprising:

determining whether each heartbeat segment within the heartbeat message would be the same as previously sent heartbeat segments; and

if each heartbeat segment would be substantially the same, sending the previously sent heartbeat segments.

10. The non-transitory machine-readable storage medium comprising instructions of claim 1, further comprising:

determining whether a problem has occurred with a master application instance in the master state; and

if a problem has occurred, removing a heartbeat segment for the master application instance in the heartbeat message.

11. The non-transitory machine-readable storage medium comprising instructions of claim 1, wherein the one or more networks include multiple networks, and wherein sending the heartbeat message comprises alternating which of the one or more networks the node sends the heartbeat message out over.

12. A non-transitory machine-readable storage medium comprising instructions, which when implemented by one or more machines that cause the one or more machines to perform the following operations:

receiving, by the local node, a heartbeat message from a remote node, wherein the heartbeat message has a format that allows multiple heartbeat segments to be included within the heartbeat message, wherein each heartbeat segment of the multiple heartbeat segments is respectively associated with an application instance being managed by the node and includes one or more fields including a field identifying the associated application instance; and

processing a heartbeat segment of the heartbeat message based on the field identifying the application instance being managed by the remote node.

13. The non-transitory machine-readable storage medium comprising instructions of claim 12, wherein processing the heartbeat segment comprises:

sending a quit message to the remote node when the remote node considers itself to be a master for the application instance, and the local node considers itself to be the master for the application instance.

14. The non-transitory machine-readable storage medium comprising instructions of claim 12, wherein processing the heartbeat segment comprises:

restarting a timer associated with the application instance when the local node considers itself to be a slave for the application instance.

15. The non-transitory machine-readable storage medium comprising instructions of claim 12, wherein processing the heartbeat segment comprises:

determining, from the heartbeat segment, whether the local node has current configuration information for the application instance; and

if the local node does not have the current configuration information, indicating that new configuration information is needed.

16. The non-transitory machine-readable storage medium comprising instructions of claim 12, wherein processing the heartbeat segment comprises:

storing a node identifier for the remote node when the local node considers itself to be a master for the application instance associated with the heartbeat segment.

17. An apparatus comprising:

at least one processor configured to generate a heartbeat message, which has a format that allows multiple heartbeat segments to be included within the heartbeat message, wherein each heartbeat segment of the multiple heartbeat segments is respectively associated with an application instance being managed by the node and includes one or more fields including a field identifying the associated application instance; and

at least one interface configured to send the heartbeat message to one or more other nodes over one or more networks to be processed by the one or more other nodes.

18. The apparatus of claim 17, wherein the at least one processor is configured to generate the heartbeat message with a heartbeat segment that identifies an application instance in a master state.

19. The apparatus of claim 17, wherein the at least one processor is configured to generate the heartbeat message with a heartbeat segment that identifies an application instance in a slave state.

20. The apparatus of claim 17, wherein the heartbeat message comprises a field, in a heartbeat segment, that identifies a configuration associated with the application instance.

21. The apparatus of claim 17, wherein the at least one processor is configured to generate the heartbeat message with a single heartbeat segment after a master application instance has been promoted to a master state, wherein the single heartbeat segment identifies the master application instance.

22. The apparatus of claim 17, wherein the at least one processor is configured to generate the heartbeat message with multiple heartbeat segments.

23. The apparatus of claim 17, wherein the at least one processor is configured to generate each heartbeat segment with a field that identifies the application instance associated with the heartbeat segment.

24. The apparatus of claim 23, wherein the at least one processor is configured to generate each heartbeat segment with a second field that identifies a current configuration associated with the application instance.

25. The apparatus of claim 17, wherein the at least one processor is configured to:

determine whether each heartbeat segment within the heartbeat message would be the same as previously sent heartbeat segments; and

if each heartbeat segment would be substantially the same, send the previously sent heartbeat segments.

26. The apparatus of claim 17, wherein the at least one processor is configured to:

determine whether a problem has occurred with a master application instance in the master state; and

if a problem has occurred, remove a heartbeat segment for the master application instance in the heartbeat message.

27. The apparatus of claim 17, wherein the one or more networks include multiple networks, and wherein sending the heartbeat message comprises alternating which of the one or more networks the node sends the heartbeat message out over.

28. An apparatus comprising:

at least one interface residing on a local node configured to receive a heartbeat message from a remote node, the heartbeat message having a format that allows multiple heartbeat segments to be included within the heartbeat message, wherein each heartbeat segment of the multiple heartbeat segments is respectively associated with an application instance being managed by the node and includes one or more fields including a field identifying the associated application instance; and

at least one processor configured to process a heartbeat segment of the heartbeat message based on the field identifying the application instance being managed by the remote node.

29. The apparatus of claim 28, wherein the at least one processor is configured to send a heartbeat segment having a quit message to the remote node when the remote node considers itself to be a master for the application instance, and the local node considers itself to be the master for the application instance.

30. The apparatus of claim 28, wherein the at least one processor is configured to restart a timer associated with the application instance when the local node considers itself to be a slave for the application instance.

31. The apparatus of claim 28, wherein the at least one processor is configured to process a heartbeat segment that comprises:

a determination, from the heartbeat segment, whether the local node has current configuration information for the application instance; and

if the local node does not have the current configuration information, an indication that new configuration information is needed.

32. The apparatus of claim 28, wherein the at least one processor is configured to process the heartbeat segment so as to store a node identifier for the remote node when the local node considers itself to be a master for the application instance associated with the heartbeat segment.

33. An apparatus comprising:

at least one processor residing on a first node configured to generate a heartbeat message, which has a format that allows multiple heartbeat segments to be included within the heartbeat message, wherein a heartbeat segment includes a field identifying a current configuration associated with an application instance being managed by the first node; and

at least one interface configured to send the heartbeat message to one or more other nodes over the one or more networks to be processed by the one or more other nodes.

34. The apparatus of claim 33, wherein the at least one processor is configured to generate the heartbeat message with an indication of the current configuration for some or all application instances.