Managing recovery of service components and notification of service errors and failures

• US 6,718,376 B1
• Filed: 12/15/1998
• Issued: 04/06/2004
• Est. Priority Date: 12/15/1998
• Status: Expired due to Term

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Thomas, Leslie Alan, Chu, Jie, Sitaraman, Aravind
• Primary Examiner(s): Jean, Frantz B.
• Application Number:
  US09/213,304
• Time in Patent Office: 1,939 Days
• Field of Search: 709/224, 709/223, 714/4, 714/48, 370/216
• US Class Current: 709/223
• CPC Class Codes: H04L 41/052 using standardised network management architectures, e.g. telecommunication management network [TMN] or unified network management architecture [UNMA]
• CPC Class Codes: H04L 41/06 Management of faults, events, alarms or notifications
• CPC Class Codes: H04L 41/082 the condition being updates or upgrades of network functionality
• CPC Class Codes: H04L 43/00 Arrangements for monitoring or testing data switching networks
• CPC Class Codes: H04L 43/0817 by checking functioning
• CPC Class Codes: H04L 43/10 Active monitoring, e.g. heartbeat, ping or trace-route
• CPC Class Codes: H04L 43/106 using time related information in packets, e.g. by adding timestamps
• CPC Class Codes: H04L 69/28 Timers or timing mechanisms used in protocols
• CPC Class Codes: H04L 69/329 in the application layer [OSI layer 7]

Abstract

A method and apparatus for providing management and maintenance to a node within a data communications network and to the composite data communications network. A network management application is started on a host which may be located at a network operation center. The management application is in communication with network nodes and services through adapters. A master daemon located at a node is activated. The master daemon starts a control adapter running on the node and if the control adapter fails the master daemon restarts the control adapter. The control adapter is capable of starting and stopping all services running on the node. Signals are communicated between the management application, the node and the services by way of adapters. Signaling provides for the exchange of useful event data related to the nodes and services running on the nodes.

Claims

1. A method for managing a node of a data communication network, comprising:

starting a master daemon;

starting a control adapter with said master daemon;

starting at least one service with said control adapter, said service including a service adapter in communication with said control adapter;

generating a plurality of heartbeat events with said control adapter and said service adapter; and

publishing said plurality of heartbeat events to an information bus such that subscribing adapters may receive said plurality of heartbeat events.

2. The method of claim 1 further comprising:

restarting said control adapter with said master daemon should said control adapter ever stop publishing said heartbeat events for longer than a predetermined time.

3. The method of claim 1 further comprising:

restarting said service adapter with said control adapter should said service adapter ever stop publishing said heartbeat events for longer than a predetermined time.

4. The method of claim 1 further comprising:

starting an Authentication, Authorization and Accounting (AAA) service with said control adapter, said AAA service including a service adapter in communication with said control adapter.

5. The method of claim 1 further comprising:

starting a Domain Name Service (DNS) with said control adapter, said DNS including a service adapter in communication with said control adapter.

6. The method of claim 1 further comprising:

starting a Dynamic Host Configuration Protocol (DHCP) service with said control adapter, said DHCP service including a service adapter in communication with said control adapter.

7. The method of claim 1 further comprising:

starting a protocol gateway service with said control adapter, said protocol gateway service including a service adapter in communication with said control adapter.

8. The method of claim 1 further comprising:

starting a cache service with said control adapter, said cache service including a service adapter in communication with said control adapter.

9. The method of claim 1 further comprising:

signaling at said control adapter an error occurrence within said control adapter; and

publishing an exception event on to said information bus.

10. The method of claim 1 further comprising:

signaling at said control adapter when said control adapter receives two or more conflicting signals from two or more sources within a specified period of time; and

publishing a race event on to said information bus.

11. The method of claim 1 further comprising:

signaling at said control adapter the identity of said control adapter in response to said control adapter receiving a signal requiring identity status; and

publishing an identity event of said control adapter on to said information bus.

12. The method of claim 1 further comprising:

signaling at said control adapter the configuration data for said service adapter; and

publishing a configuration event on to said information bus.

13. The method of claim 1 further comprising:

signaling at said service adapter an error occurrence within said service; and

publishing an exception event on to said information bus.

14. The method of claim 1 further comprising:

signaling at said service adapter the identity of said service adapter in response to said service adapter receiving a signal requiring identity status; and

publishing an identity event on to said information bus.

15. The method of claim 1 wherein said heartbeat events include Global Unique Identifiers of publishers.

16. The method of claim 1 wherein said heartbeat events include time stamps.

17. The method of claim 1 wherein said heartbeat events include information regarding a number of packets received and processed by the node.

18. The method of claim 1 wherein said heartbeat events include information regarding a number of packets in queue in the node.

19. The method of claim 1 wherein said heartbeat events include information regarding a number of packets timed out.

20. The method of claim 1 wherein said heartbeat events include information regarding a rate at which packets are received by the node.

21. A method for managing a data communication network, comprising:

starting a master daemon;

starting a control adapter with said master daemon;

starting at least one service with said control adapter, said service including a service adapter in communication with said control adapter;

generating a plurality of heartbeat events with said control adapter and said service adapter; and

publishing said plurality of heartbeat events to an information bus such that subscribing adapters may receive said plurality of heartbeat events, said information bus in communication with a network management application and an access database adapter, said access data base adapter in communication with said network management application.

22. The method of claim 21 further comprising:

restarting said control adapter with said master daemon should said control adapter ever stop publishing said heartbeat events for longer than a predetermined time.

23. The method of claim 21 further comprising:

restarting said service adapter with said control adapter should said service adapter ever stop publishing said heartbeat events for longer than a predetermined time.

24. The method of claim 21 further comprising:

starting an Authentication, Authorization and Accounting (AAA) service with said control adapter, said AAA service including a service adapter in communication with said control adapter.

25. The method of claim 21 further comprising:

starting a Domain Name Service (DNS) with said control adapter, said DNS including a service adapter in communication with said control adapter.

26. The method of claim 21 further comprising:

starting a Dynamic Host Configuration Protocol (DHCP) service with said control adapter, said DHCP service including a service adapter in communication with said control adapter.

27. The method of claim 21 further comprising:

starting a protocol gateway service with said control adapter, said protocol gateway service including a service adapter in communication with said control adapter.

28. The method of claim 21 further comprising:

starting a cache service with said control adapter, said cache service including a service adapter in communication with said control adapter.

29. The method of claim 21 further comprising:

signaling at said control adapter an error occurrence within said control adapter; and

publishing an exception event on to said information bus.

30. The method of claim 21 further comprising:

signaling at said control adapter when said control adapter receives two or more conflicting signals from two or more sources within a specified period of time; and

publishing a race event on to said information bus.

31. The method of claim 21 further comprising:

signaling at said control adapter the identity of said control adapter in response to said control adapter receiving a signal requiring identity status; and

publishing an identity event of said control adapter on to said information bus.

32. The method of claim 21 further comprising:

signaling at said control adapter the configuration data for said service adapter; and

publishing a configuration event on to said information bus.

33. The method of claim 21 further comprising:

signaling at said service adapter an error occurrence within said service; and

publishing an exception event on to said information bus.

34. The method of claim 21 further comprising:

signaling at said service adapter the identity of said service adapter in response to said service adapter receiving a signal requiring identity status; and

publishing an identity event on to said information bus.

35. The method of claim 21 further comprising:

signaling at said access database adapter configuration data for said service adapter; and

publishing a configure event to said information bus.

36. The method of claim 21 further comprising:

signaling at said access database adapter configuration data for said control adapter; and

publishing a configure event to said information bus.

37. The method of claim 21 further comprising:

signaling at said access database adapter for said service to begin operating; and

publishing a start event on said information bus.

38. The method of claim 21 further comprising:

signaling at said access database adapter for said service to stop; and

publishing a stop event on to said information bus.

39. The method of claim 21 further comprising:

signaling at said access database adapter for the identity of said control adapter; and

publishing an identity event on to said information bus.

40. The method of claim 21 further comprising:

signaling at said access database adapter for the identity of said service adapter; and

publishing an identity event on to said information bus.

41. The method of claim 21 further comprising:

signaling at said access database adapter for said control adapter to execute a script; and

publishing a DoCommand event on to said information bus.

42. The method of claim 21 wherein said heartbeat events include Global Unique Identifiers of publishers.

43. The method of claim 21 wherein said heartbeat events include time stamps.

44. The method of claim 21 wherein said heartbeat events include information regarding a number of packets received and processed by the node.

45. The method of claim 21 wherein said heartbeat events include information regarding a number of packets in queue in the node.

46. The method of claim 21 wherein said heartbeat events include information regarding a number of packets timed out.

47. The method of claim 21 wherein said heartbeat events include information regarding a rate at which packets are received by the node.

48. A node within a data communication network, comprising:

a master daemon;

a control adapter activated by said master daemon, said control adapter in communication with an information bus over which are published heartbeat events from said control adapter such that subscribing adapters may receive said heartbeat events;

a service adapter in communication with said control adapter and said information bus over which are published heartbeat events from said service adapter such that subscribing adapters may receive said plurality of heartbeat events; and

at least one service running on said node, said service operatively coupled to said service adapter.

49. The node of claim 48 wherein said service is an Authentication, Authorization and Accounting (AAA) service, said AAA having a service adapter in communication with said information broker.

50. The node of claim 48 wherein said service is a Domain Name Service (DNS), said DNS having a service adapter in communication with said control adapter.

51. The node of claim 48 wherein said service is a Dynamic Host Configuration Protocol (DHCP) service, said DHCP service having a service adapter in communication with said control adapter.

52. The node of claim 48 wherein said service is a protocol gateway service, said protocol gateway service having a service adapter in communication with said control adapter.

53. The node of claim 48 wherein said service is a cache service, said cache service having a service adapter in communication with said control adapter.

54. The node of claim 48 wherein said heartbeat events include Global Unique Identifiers of publishers.

55. The node of claim 48 wherein said heartbeat events include time stamps.

56. The node of claim 48 wherein said heartbeat events include information regarding a number of packets received and processed by the node.

57. The node of claim 48 wherein said heartbeat events include information regarding a number of packets in queue in the node.

58. The node of claim 48 wherein said heartbeat events include information regarding a number of packets timed out.

59. The node of claim 48 wherein said heartbeat events include information regarding a rate at which packets are received by the node.

60. A data communication network, comprising:

a first processor having;

a network management application;

an access data base adapter in communication with said network management application and an information bus; and

a database in communication with said network management application and said access database adapter; and

a second processor having;

a master daemon;

a control adapter activated by said master daemon, said control adapter in communication with an information bus over which are published heartbeat events from said control adapter such that subscribing adapters may receive said heartbeat events;

at least one service running on said second processor; and

a service adapter in communication with said service, said service adapter in communication with said control adapter and said information bus over which are published heartbeat events from said service adapter such that subscribing adapters may receive said plurality of heartbeat events.

61. The data communication network of claim 60 further comprising:

an information broker in communication with said information bus, said first processor, and said second processor.

62. The data communication network of claim 60 wherein said heartbeat events include Global Unique Identifiers of publishers.

63. The data communication network of claim 60 wherein said heartbeat events include time stamps.

64. The data communication network of claim 60 wherein said heartbeat events include information regarding a number of packets received and processed by the node.

65. The data communication network of claim 60 wherein said heartbeat events include information regarding a number of packets in queue in the node.

66. The data communication network of claim 60 wherein said heartbeat events include information regarding a number of packets timed out.

67. The data communication network of claim 60 wherein said heartbeat events include information regarding a rate at which packets are received by the node.

68. An apparatus for managing a node of a data communication network, comprising:

means for starting a master daemon;

means for starting a control adapter with said master daemon;

means for starting at least one service with said control adapter, said service including a service adapter in communication with said control adapter;

means for generating a plurality of heartbeat events with said control adapter and said service adapter; and

means for publishing said plurality of heartbeat events to an information bus such that subscribing adapters may receive said plurality of heartbeat events.

69. The apparatus of claim 68 further comprising:

means for restarting said control adapter with said master daemon should said control adapter ever stop publishing said heartbeat events for longer than a predetermined time.

70. The apparatus of claim 68 further comprising:

means for restarting said service adapter with said control adapter should said service adapter ever stop publishing said heartbeat events for longer than a predetermined time.

71. The apparatus of claim 68 further comprising:

means for starting an Authentication, Authorization and Accounting (AAA) service with said control adapter, said AAA service including a service adapter in communication with said control adapter.

72. The apparatus of claim 68 further comprising:

means for starting a Domain Name Service (DNS) with said control adapter, said DNS including a service adapter in communication with said control adapter.

73. The apparatus of claim 68 further comprising:

means for starting a Dynamic Host Configuration Protocol (DHCP) service with said control adapter, said DHCP service including a service adapter in communication with said control adapter.

74. The apparatus of claim 68 further comprising:

means for starting a protocol gateway service with said control adapter, said protocol gateway service including a service adapter in communication with said control adapter.

75. The apparatus of claim 68 further comprising:

means for starting a cache service with said control adapter, said cache service including a service adapter in communication with said control adapter.

76. The apparatus of claim 68 further comprising:

means for signaling at said control adapter an error occurrence within said control adapter; and

means for publishing an exception event on to said information bus.

77. The apparatus of claim 68 further comprising:

means for signaling at said control adapter when said control adapter receives two or more conflicting signals from two or more sources within a specified period of time; and

means for publishing a race event on to said information bus.

78. The apparatus of claim 68 further comprising:

means for signaling at said control adapter the identity of said control adapter in response to said control adapter receiving a signal requiring identity status; and

means for publishing an identity event of said control adapter on to said information bus.

79. The apparatus of claim 68 further comprising:

means for signaling at said control adapter the configuration data for said service adapter; and

means for publishing a configuration event on to said information bus.

80. The apparatus of claim 68 further comprising:

means for signaling at said service adapter an error occurrence within said service; and

means for publishing an exception event on to said information bus.

81. The apparatus of claim 68 further comprising:

means for signaling at said service adapter the identity of said service adapter in response to said service adapter receiving a signal requiring identity status; and

means for publishing an identity event on to said information bus.

82. The apparatus of claim 68 wherein said heartbeat events include Global Unique Identifiers of publishers.

83. The apparatus of claim 68 wherein said heartbeat events include time stamps.

84. The apparatus of claim 68 wherein said heartbeat events include information regarding a number of packets received and processed by the node.

85. The apparatus of claim 68 wherein said heartbeat events include information regarding a number of packets in queue in the node.

86. The apparatus of claim 68 wherein said heartbeat events include information regarding a number of packets timed out.

87. The apparatus of claim 68 wherein said heartbeat events include information regarding a rate at which packets are received by the node.

88. An apparatus for managing a data communication network, comprising:

means for means for starting a master daemon;

means for means for starting a control adapter with said master daemon;

means for means for starting at least one service with said control adapter, said service including a service adapter in communication with said control adapter;

means for generating a plurality of heartbeat events with said control adapter and said service adapter; and

means for publishing said plurality of heartbeat events to an information bus such that subscribing adapters may receive said plurality of heartbeat events, said information bus in communication with a network management application and an access database adapter, said access data base adapter in communication with said network management application.

89. The apparatus of claim 88 further comprising:

means for restarting said control adapter with said master daemon should said control adapter ever stop publishing said heartbeat events for longer than a predetermined time.

90. The apparatus of claim 88 further comprising:

means for restarting said service adapter with said control adapter should said service adapter ever stop publishing said heartbeat events for longer than a predetermined time.

91. The apparatus of claim 88 further comprising:

means for starting an Authentication, Authorization and Accounting (AAA) service with said control adapter, said AAA service including a service adapter in communication with said control adapter.

92. The apparatus of claim 88 further comprising:

means for starting a Domain Name Service (DNS) with said control adapter, said DNS including a service adapter in communication with said control adapter.

93. The apparatus of claim 88 further comprising:

means for starting a Dynamic Host Configuration Protocol (DHCP) service with said control adapter, said DHCP service including a service adapter in communication with said control adapter.

94. The apparatus of claim 88 further comprising:

means for starting a protocol gateway service with said control adapter, said protocol gateway service including a service adapter in communication with said control adapter.

95. The apparatus of claim 88 further comprising:

means for starting a cache service with said control adapter, said cache service including a service adapter in communication with said control adapter.

96. The apparatus of claim 88 further comprising:

means for signaling at said control adapter an error occurrence within said control adapter; and

means for publishing an exception event on to said information bus.

97. The apparatus of claim 88 further comprising:

means for signaling at said control adapter when said control adapter receives two or more conflicting signals from two or more sources within a specified period of time; and

means for publishing a race event on to said information bus.

98. The apparatus of claim 88 further comprising:

means for signaling at said control adapter the identity of said control adapter in response to said control adapter receiving a signal requiring identity status; and

means for publishing an identity event of said control adapter on to said information bus.

99. The apparatus of claim 88 further comprising:

means for signaling at said control adapter the configuration data for said service adapter; and

means for publishing a configuration event on to said information bus.

100. The apparatus of claim 88 further comprising:

means for signaling at said service adapter an error occurrence within said service; and

means for publishing an exception event on to said information bus.

101. The apparatus of claim 88 further comprising:

means for signaling at said service adapter the identity of said service adapter in response to said service adapter receiving a signal requiring identity status; and

means for publishing an identity event on to said information bus.

102. The apparatus of claim 88 further comprising:

means for signaling at said access database adapter configuration data for said service adapter; and

means for publishing a configure event to said information bus.

103. The apparatus of claim 88 further comprising:

means for signaling at said access database adapter configuration data for said control adapter; and

means for publishing a configure event to said information bus.

104. The apparatus of claim 88 further comprising:

means for signaling at said access database adapter for said service to begin operating; and

means for publishing a start event on said information bus.

105. The apparatus of claim 88 further comprising:

means for signaling at said access database adapter for said service to stop; and

means for publishing a stop event on to said information bus.

106. The apparatus of claim 88 further comprising:

means for signaling at said access database adapter for the identity of said control adapter; and

means for publishing an identity event on to said information bus.

107. The apparatus of claim 88 further comprising:

means for signaling at said access database adapter for the identity of said service adapter; and

means for publishing an identity event on to said information bus.

108. The apparatus of claim 88 further comprising:

means for signaling at said access database adapter for said control adapter to execute a script; and

means for publishing a DoCommand event on to said information bus.

109. The apparatus of claim 88 wherein said heartbeat events include Global Unique Identifiers of publishers.

110. The apparatus of claim 88 wherein said heartbeat events include time stamps.

111. The apparatus of claim 88 wherein said heartbeat events include information regarding a number of packets received and processed by the node.

112. The apparatus of claim 88 wherein said heartbeat events include information regarding a number of packets in queue in the node.

113. The apparatus of claim 88 wherein said heartbeat events include information regarding a number of packets timed out.

114. The apparatus of claim 88 wherein said heartbeat events include information regarding a rate at which packets are received by the node.

115. A program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine to perform a method for managing a node of a data communication network, said method comprising:

starting a master daemon;

starting a control adapter with said master daemon;

starting at least one service with said control adapter, said service including a service adapter in communication with said control adapter;

generating a plurality of heartbeat events with said control adapter and said service adapter; and

publishing said plurality of heartbeat events to an information bus such that subscribing adapters may receive said plurality of heartbeat events.

116. A program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine to perform a method for managing a data communication network, said method comprising:

starting a master daemon;

starting a control adapter with said master daemon;

starting at least one service with said control adapter, said service including a service adapter in communication with said control adapter;

generating a plurality of heartbeat events with said control adapter and said service adapter; and

publishing said plurality of heartbeat events to an information bus such that subscribing adapters may receive said plurality of heartbeat events, said information bus in communication with a network management application and an access database adapter, said access data base adapter in communication with said network management application.

117. A node within a data communications network, comprising:

a master daemon process running on a processor within the node;

a control adapter process activated by the master daemon, said control adapter process in communication with an information bus on which said control adapter process publishes heartbeat events generated by said control adapter process while it is operating such that subscribing adapters may receive said heartbeat events;

a service adapter process activated by the control adapter, said service adapter process in communication with said control adapter process to which said service adapter process transmits heartbeat events generated by said service adapter process while it is operating; and

a service process activated by the service adapter process, said service process in communication with said service adapter process.

118. The node of claim 117 wherein said service is an Authentication, Authorization and Accounting (AAA) service, said AAA having a service adapter in communication with said information broker.

119. The node of claim 117 wherein said service is a Domain Name Service (DNS), said DNS having a service adapter in communication with said control adapter.

120. The node of claim 117 wherein said service is a Dynamic Host Configuration Protocol (DHCP) service, said DHCP service having a service adapter in communication with said control adapter.

121. The node of claim 117 wherein said service is a protocol gateway service, said protocol gateway service having a service adapter in communication with said control adapter.

122. The node of claim 117 wherein said service is a cache service, said cache service having a service adapter in communication with said control adapter.

123. The apparatus of claim 117 wherein said heartbeat events include Global Unique Identifiers of publishers.

124. The apparatus of claim 117 wherein said heartbeat events include time stamps.

125. The apparatus of claim 117 wherein said heartbeat events include information regarding a number of packets received and processed by the node.

126. The apparatus of claim 117 wherein said heartbeat events include information regarding a number of packets in queue in the node.

127. The apparatus of claim 117 wherein said heartbeat events include information regarding a number of packets timed out.

128. The apparatus of claim 117 wherein said heartbeat events include information regarding a rate at which packets are received by the node.