Managing recovery of service components and notification of service errors and failures

• US 7,370,102 B1
• Filed: 02/13/2004
• Issued: 05/06/2008
• Est. Priority Date: 12/15/1998
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Thomas, Leslie Alan, Sitaraman, Aravind, Chu, Jie
• Primary Examiner(s): Jean; Frantz B.
• Application Number:
  US10/778,749
• Time in Patent Office: 1,544 Days
• Field of Search: 709/223, 709/224, 709/217, 709/229, 714/4, 714/48, 370/216, 719/315, 719/332
• US Class Current: 709/223
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0715 in a system implementing multitasking multitasking per se G06F9/46
• CPC Class Codes: G06F 11/0757 by exceeding a time limit, i.e. time-out, e.g. watchdogs
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: H04L 41/0233 Object-oriented techniques, for representation of network management data, e.g. common object request broker architecture [CORBA]
• CPC Class Codes: H04L 43/50 Testing arrangements

Abstract

A method and apparatus for providing management and maintenance to a node within a data communications network and to the composite data communications network. A network management application is started on a host which may be located at a network operation center. The management application is in communication with network nodes and services through adapters. A master daemon located at a node is activated. The master daemon starts a control adapter running on the node and if the control adapter fails the master daemon restarts the control adapter. The control adapter is capable of starting and stopping all services running on the node. Signals are communicated between the management application, the node and the services by way of adapters. Signaling provides for the exchange of useful event data related to the nodes and services running on the nodes.

Claims

1. A method for managing a node of a data communication network, comprising:

starting a master daemon;

starting a control adapter using said master daemon;

starting at least one service using said control adapter, said service including a service adapter in communication with said control adapter;

generating one or more heartbeat events with said control adapter and one or more heartbeat events with said service adapter; and

publishing said heartbeat events to an information bus.

2. The method of claim 1 further comprising:

restarting said control adapter using said master daemon should said control adapter ever stop publishing said heartbeat events for longer than a predetermined time.

3. The method of claim 1 further comprising:

signaling at said control adapter an error occurrence within said control adapter; and

publishing an exception event on to said information bus.

4. The method of claim 1 further comprising:

signaling at said control adapter when said control adapter receives two or more conflicting signals from two or more sources within a specified period of time; and

publishing a race event on to said information bus.

5. The method of claim 1 further comprising:

signaling at said control adapter the identity of said control adapter in response to said control adapter receiving a signal requiring identity status; and

publishing an identity event of said control adapter on to said information bus.

6. The method of claim 1 wherein said heartbeat events include Global Unique Identifiers of publishers.

7. The method of claim 1 wherein said heartbeat events include time stamps.

8. The method of claim 1 wherein said heartbeat events include information regarding a number of packets received and processed by the node.

9. The method of claim 1 wherein said heartbeat events include information regarding a number of packets in queue in the node.

10. The method of claim 1 wherein said heartbeat events include information regarding a number of packets timed out.

11. The method of claim 1 wherein said heartbeat events include information regarding a rate at which packets are received by the node.

12. A node within a data communication network, comprising:

a master daemon;

a control adapter configured to be started using said master daemon, said control adapter configured to communicate with an information bus over which are published one or more heartbeat events from said control adapter, said control adapter further configured to signal an error occurrence within said control adapter, said apparatus further configured to publish an exception event on to said information bus;

a service adapter in communication with said control adapter and said information bus over which are published heartbeat events from said service adapter; and

at least one service running on said node, said service operatively coupled to said service adapter.

13. The node of claim 12 wherein said heartbeat events include Global Unique Identifiers of publishers.

14. The node of claim 12 wherein said heartbeat events include time stamps.

15. The node of claim 12 wherein said heartbeat events include information regarding a number of packets received and processed by the node.

16. The node of claim 12 wherein said heartbeat events include information regarding a number of packets in queue in the node.

17. The node of claim 12 wherein said heartbeat events include information regarding a number of packets timed out.

18. The node of claim 12 wherein said heartbeat events include information regarding a rate at which packets are received by the node.

19. A data communication network, comprising:

a first processor having;

a network management application;

an access data base adapter in communication with said network management application and an information bus; and

a database in communication with said network management application and said access database adapter; and

a second processor having;

a master daemon;

a control adapter configured to be started using said master daemon, said control adapter configured to communicate with an information bus over which are published one or more heartbeat events from said control adapter, said control adapter further configured to signal an error occurrence within said control adapter, said apparatus further configured to publish an exception event on to said information bus;

at least one service running on said second processor; and

a service adapter in communication with said service, said service adapter in communication with said control adapter and said information bus over which are published heartbeat events from said service adapter.

20. The data communication network of claim 19 further comprising:

an information broker in communication with said information bus, said first processor, and said second processor.

21. The data communication network of claim 19 wherein said heartbeat events include Global Unique Identifiers of publishers.

22. The data communication network of claim 19 wherein said heartbeat events include time stamps.

23. The data communication network of claim 19 wherein said heartbeat events include information regarding a number of packets received and processed by the node.

24. The data communication network of claim 19 wherein said heartbeat events include information regarding a number of packets in queue in the node.

25. The data communication network of claim 19 wherein said heartbeat events include information regarding a number of packets timed out.

26. The data communication network of claim 19 wherein said heartbeat events include information regarding a rate at which packets are received by the node.

27. An apparatus for managing a node of a data communication network, comprising:

means for starting a master daemon;

means for starting a control adapter using said master daemon;

means for starting at least one service using said control adapter, said service including a service adapter in communication with said control adapter;

means for generating one or more heartbeat events with said control adapter and one or more heartbeat events with said service adapter; and

means for publishing said heartbeat events to an information bus.

28. The apparatus of claim 27, further comprising:

means for restarting said control adapter using said master daemon should said control adapter ever stop publishing said heartbeat events for longer than a predetermined time.

29. The apparatus of claim 27 further comprising:

means for signaling at said control adapter when said control adapter receives two or more conflicting signals from two or more sources within a specified period of time; and

means for publishing a race event on to said information bus.

30. The apparatus of claim 27 wherein said heartbeat events include Global Unique Identifiers of publishers.

31. The apparatus of claim 27 wherein said heartbeat events include time stamps.

32. The apparatus of claim 27 wherein said heartbeat events include information regarding a number of packets received and processed by the node.

33. The apparatus of claim 27 wherein said heartbeat events include information regarding a number of packets in queue in the node.

34. The apparatus of claim 27 wherein said heartbeat events include information regarding a number of packets timed out.

35. The apparatus of claim 27 wherein said heartbeat events include information regarding a rate at which packets are received by the node.

36. A program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine to perform a method for managing a node of a data communication network, said method comprising:

starting a master daemon;

starting a control adapter using said master daemon;

starting at least one service using said control adapter, said service including a service adapter in communication with said control adapter;

generating one or more heartbeat events with said control adapter and one or more heartbeat events with said service adapter; and

publishing said heartbeat events to an information bus.

37. The program storage device method of claim 36, wherein the method further comprises:

restarting said control adapter with said master daemon should said control adapter ever stop publishing said heartbeat events for longer than a predetermined time.

38. A node within a data communications network, comprising:

a master daemon process running on a processor within the node;

a control adapter process activated by the master daemon, said control adapter process in communication with an information bus on which said control adapter process publishes heartbeat events generated by said control adapter process while it is operating;

a service adapter process activated by the control adapter, said service adapter process in communication with said control adapter process to which said service adapter process transmits heartbeat events generated by said service adapter process while it is operating; and

a service process activated by the service adapter process, said service process in communication with said service adapter process.

39. The node of claim 38 wherein said heartbeat events include Global Unique Identifiers of publishers.

40. The node of claim 38 wherein said heartbeat events include time stamps.

41. The node of claim 38 wherein said heartbeat events include information regarding a number of packets received and processed by the node.

42. The node of claim 38 wherein said heartbeat events include information regarding a number of packets in queue in the node.

43. The node of claim 38 wherein said heartbeat events include information regarding a number of packets timed out.

44. The node of claim 38 wherein said heartbeat events include information regarding a rate at which packets are received by the node.

45. An apparatus for managing a node of a communication network, comprising:

a master daemon; and

a control adapter configured to be started using said master daemon, said control adapter configured to communicate with an information bus over which are published one or more heartbeat events from said control adapter, said control adapter further configured to signal an error occurrence within said control adapter, said apparatus further configured to publish an exception event on to said information bus.

46. The apparatus of claim 45, further comprising:

a service adapter in communication with said control adapter and said information bus over which are published heartbeat events from said service adapter.

47. A method for managing a node of a data communication network, comprising:

starting a master daemon;

starting a control adapter using said master daemon;

generating one or more heartbeat events with said control adapter;

publishing said heartbeat events to an information bus;

signaling at said control adapter an error occurrence within said control adapter; and

publishing an exception event on to said information bus.

48. The method of claim 47, further comprising:

restarting said control adapter using said master daemon should said control adapter ever stop publishing said heartbeat events for longer than a predetermined time.

49. The method of claim 47, further comprising:

signaling at said control adapter when said control adapter receives two or more conflicting signals from two or more sources within a specified period of time; and

publishing a race event on to said information bus.

50. An apparatus for managing a node of a data communication network, comprising:

means for starting a master daemon;

means for starting a control adapter using said master daemon;

means for generating one or more heartbeat events with said control adapter;

means for publishing said heartbeat events to an information bus;

means for signaling at said control adapter an error occurrence within said control adapter; and

means for publishing an exception event on to said information bus.

51. The apparatus of claim 50, further comprising:

means for restarting said control adapter using said master daemon should said control adapter ever stop publishing said heartbeat events for longer than a predetermined time.

52. The apparatus of claim 50, further comprising:

means for signaling at said control adapter when said control adapter receives two or more conflicting signals from two or more sources within a specified period of time; and

means for publishing a race event on to said information bus.

53. A program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine to perform a method for managing a node of a data communication network, said method comprising:

starting a master daemon;

starting a control adapter using said master daemon;

generating one or more heartbeat events with said control adapter;

publishing said heartbeat events to an information bus;

signaling at said control adapter an error occurrence within said control adapter; and

publishing an exception event on to said information bus.

54. The program storage device of claim 53, wherein the method further comprises:

restarting said control adapter using said master daemon should said control adapter ever stop publishing said heartbeat events for longer than a predetermined time.

55. The program storage device of claim 53, wherein the method further comprises:

signaling at said control adapter when said control adapter receives two or more conflicting signals from two or more sources within a specified period of time; and

publishing a race event on to said information bus.

56. A method for managing a node of a data communication network, comprising:

starting a master daemon;

starting a control adapter using said master daemon;

starting at least one service using said control adapter, said service including a service adapter in communication with said control adapter;

generating one or more heartbeat events with said control adapter and one or more heartbeat events with said service adapter;

publishing said heartbeat events to an information bus; and

restarting said service adapter using said control adapter should said service adapter ever stop publishing said heartbeat events for longer than a predetermined time.

57. An apparatus for managing a node of a data communication network, comprising:

means for starting a master daemon;

means for starting a control adapter using said master daemon;

means for starting at least one service using said control adapter, said service including a service adapter in communication with said control adapter;

means for generating one or more heartbeat events with said control adapter and one or more heartbeat events with said service adapter;

means for publishing said heartbeat events to an information bus; and

means for restarting said service adapter using said control adapter should said service adapter ever stop publishing said heartbeat events for longer than a predetermined time.

58. A program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine to perform a method for managing a node of a data communication network, said method comprising:

starting a master daemon;

starting a control adapter using said master daemon;

starting at least one service using said control adapter, said service including a service adapter in communication with said control adapter;

generating one or more heartbeat events with said control adapter and one or more heartbeat events with said service adapter;

publishing said heartbeat events to an information bus; and

restarting said service adapter with said control adapter should said service adapter ever stop publishing said heartbeat events for longer than a predetermined time.

59. A method for managing a node of a data communication network, comprising:

starting a master daemon;

starting a control adapter using said master daemon;

generating one or more heartbeat events with said control adapter;

publishing said heartbeat events to an information bus; and

restarting said control adapter using said master daemon should said control adapter ever stop publishing said heartbeat events for longer than a predetermined time.

60. A method for managing a node of a data communication network, comprising:

starting a master daemon;

starting a control adapter using said master daemon;

generating one or more heartbeat events with said control adapter;

publishing said heartbeat events to an information bus; and

signaling at said control adapter when said control adapter receives two or more conflicting signals from two or more sources within a specified period of time; and

publishing a race event on to said information bus.

61. An apparatus for managing a node of a data communication network, comprising:

means for starting a master daemon;

means for starting a control adapter using said master daemon;

means for generating one or more heartbeat events with said control adapter;

means for publishing said heartbeat events to an information bus; and

means for restarting said control adapter using said master daemon should said control adapter ever stop publishing said heartbeat events for longer than a predetermined time.

62. An apparatus for managing a node of a data communication network, comprising:

means for starting a master daemon;

means for starting a control adapter using said master daemon;

means for generating one or more heartbeat events with said control adapter;

means for publishing said heartbeat events to an information bus;

means for signaling at said control adapter when said control adapter receives two or more conflicting signals from two or more sources within a specified period of time; and

means for publishing a race event on to said information bus.

63. A program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine to perform a method for managing a node of a data communication network, said method comprising:

starting a master daemon;

starting a control adapter using said master daemon;

generating one or more heartbeat events with said control adapter;

publishing said heartbeat events to an information bus; and

restarting said control adapter using said master daemon should said control adapter ever stop publishing said heartbeat events for longer than a predetermined time.

64. A program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine to perform a method for managing a node of a data communication network, said method comprising:

starting a master daemon;

starting a control adapter using said master daemon;

generating one or more heartbeat events with said control adapter;

publishing said heartbeat events to an information bus;

signaling at said control adapter when said control adapter receives two or more conflicting signals from two or more sources within a specified period of time; and

publishing a race event on to said information bus.