Arrangement for automated fault detection and fault resolution of a network device

• US 20060277299A1
• Filed: 05/26/2006
• Published: 12/07/2006
• Est. Priority Date: 04/12/2002
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: De Kegel, Erik, Sinclair, Keith, Mc Daniel, Dean, Baekelmans, John
• Granted Patent: US 7,509,415 B2
• Time in Patent Office: Days
• US Class Current: 709/224
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: H04L 41/064 involving time analysis

Abstract

A network based management system enables network devices installed at a customer'"'"'s premises to be automatically monitored and managed for automated support services. Each network device includes a monitoring agent configured for generating a notification message based upon an event where device operations exceed prescribed monitored parameters. A troubleshooting resource at the customer premises requests selected device state attributes from the network device and that are forwarded to a data collection resource that serves as an interface to the manufacturer'"'"'s back-end infrastructure. The troubleshooting system accesses databases that store the configuration information for the affected network device, and uses the configuration information for accessing problem/resolution information. The rules based troubleshooting system, upon correlating the databases for the problem/resolution information, provides a service notification message as a response to the event, and selectively includes the problem/resolution information for the network device.

Claims

1. A system comprising:

a network device having an identifiable configuration and including a monitoring agent configured for monitoring device operations relative to monitored parameters, the monitoring agent configured for generating a notification message based on an event where the device operations exceed at least one of the monitored parameters;

a troubleshooting resource configured for requesting selected device state attributes associated with the event from the network device based on reception of the notification message;

at least one data collection resource configured for collecting the selected device state attributes associated with the event into an event message, the at least one data collection resource configured for collecting the selected device state attributes for events from a respective plurality of the troubleshooting resources via respective secure connections via an Internet Protocol (IP) network,

a first database configured for storing identifiable configuration information describing the corresponding identifiable configuration for the network device;

a second database configured for storing problem/resolution information for the network device according to known configurations; and

a rules-based troubleshooting system configured for determining a presence in the second database of problem/resolution information for the network device based on correlating the identifiable configuration information retrieved from the first database and the corresponding selected device state attributes specified in the event message, the rules-based troubleshooting system configured for providing a service notification message as a response to the event and that selectively includes the problem/resolution information for the network device.

2. The system of claim 1, wherein the monitored parameters include CPU utilization, and memory usage.

3. The system of claim 2, further comprising a service appliance configured for executing the troubleshooting resource and establishing a secure tunnel connection with the network device for retrieval of the selected device state attributes from the network device.

4. The system of claim 2, wherein the network device is configured for local execution of the troubleshooting resource.

5. The system of claim 2, wherein the selected device state attributes include an identification of each process executed by the network device upon generation of the notification message, and memory resource utilization for each corresponding identified process.

6. The system of claim 1, further comprising a plurality of the data collection resources, each configured for receiving the selected device state attributes from a corresponding group of the troubleshooting resources via the IP network and based on a corresponding geographic region.

7. The system of claim 6, wherein each troubleshooting resource is configured for initially accessing one of a prescribed list of the data collection resources.

8. The system of claim 6, wherein each data collection resource is configured for supplying received resource updates to the corresponding group of troubleshooting resources.

9. The system of claim 6, wherein the monitoring agent is configured for supplying, upon initialization, the identifiable configuration information to the corresponding one data collection resource via the corresponding secure connection for storage in the first database.

10. The system of claim 9, wherein the monitoring agent is configured for supplying updated identifiable configuration information to the corresponding one data collection resource for storage in the first database in response to detecting a change in the network device from the identifiable configuration information to the updated identifiable configuration information.

11. The system of claim 9, wherein the rules-based troubleshooting system includes:

a rules manager configured for correlating the second database for the problem/resolution information for the network device based on correlating the identifiable configuration information, and the selected device state attributes, relative to network management rules specified according to an extensible markup language, the network management rules including network infrastructure rules for describing a network infrastructure and network element rules for describing network elements, including the network device, within the corresponding network infrastructure;

a notification resource module configured for identifying a manager contact for the network device, the notification resource module configured for sending notification information, including the service notification message, to the manager contact; and

a central communications system configured for routing information between the at least one data collection resource, the first database, and the second database, the central communications system including;

(1) an inventory management resource configured for loading the identifiable configuration information for the network device, received from a first message via the corresponding data collection resource, into the first database according to a prescribed database format,

(2) a device troubleshooting resource configured for responding to the event message by fetching the identifiable configuration information for the network device associated with the event message, sending a query to the rules manager for correlation of the problem/resolution information, and sending results of the query to the notification resource module; and

(3) a message router configured for routing messages between the data collection resources, the inventory management resource, the notification resource module, the device troubleshooting resource, and the rules manager.

12. The system of claim 11, wherein the rules-based troubleshooting system further includes a device update resource configured for identifying, based on new problem/resolution information added to the second database, network devices affected by the new problem/resolution information based on correlating the new problem/resolution information to the respective identifiable information stored in the first database, the device update resource configured for outputting to the notification resource module an update notification for the network devices affected by the new problem/resolution information.

13. The system of claim 12, wherein the rules-based troubleshooting system further includes a user contact database configured for storing entries specifying, for each said network device, the corresponding manager contact for the network device, the notification resource module configured for sending the update notification to each corresponding manager contact for each corresponding network device affected by the new problem/resolution information.

14. The system of claim 11, wherein the rules-based troubleshooting system further includes a best practices module configured for identifying optimum device configuration settings, the best practices module configured for outputting to the notification resource module a practice advisory message for the network device, based on correlating the corresponding identifiable configuration information with the stored problem/resolution information.

15. A method of managing a network device, the method comprising:

storing identifiable configuration information for the network device in a first database configured for storing the identifiable configuration information for multiple network devices;

generating, in the network device, a notification message based on an event where device operations monitored by a monitoring agent exceed at least one of a plurality of monitored parameters;

requesting, by a troubleshooting resource, selected device state attributes associated with the event from the device based on reception of the notification message;

collecting the selected device state attributes associated with the event from the troubleshooting resource by a data collection resource via a secure connection via an Internet Protocol (IP) network, and outputting an event message that includes the selected device state attributes to a rules-based troubleshooting system;

determining, by the rules-based troubleshooting system, a presence in a second database of problem/resolution information for the device based on correlating the identifiable configuration information retrieved from the first database and the corresponding selected device state attributes specified in the event message, the second database configured for storing problem/resolution information for the network devices according to respective known configurations; and

providing, by the rules-based troubleshooting system, a service notification message as a response to the event and that selectively includes the problem/resolution information for the device.

16. The method of claim 15, wherein the monitored parameters include CPU utilization, and memory usage.

17. The method of claim 16, further comprising establishing a secure tunnel connection between the troubleshooting resource and the network device for retrieval of the selected device state attributes from the network device.

18. The method of claim 16, wherein the network device is configured for local execution of the troubleshooting resource.

19. The method of claim [[6]] 16, further comprising supplying by the monitoring agent the selected device state attributes, including an identification of each process executed by the network device upon generation of the notification message, and memory resource utilization for each corresponding identified process, to the troubleshooting resource.

20. The method of claim 15, further comprising supplying the selected device state attributes to the data collection resource selected from a plurality of the data collection resources, each said data collection resource configured for receiving the selected device state attributes from a corresponding group of the troubleshooting resources via the IP network and based on a corresponding geographic region.

21. The method of claim 20, wherein the supplying step includes initially selecting the data collection resource from a prescribed list of the data collection resources.

22. The method of claim 20, further comprising supplying received resource updates, received by the data collection resource, to the troubleshooting resource.

23. The method of claim 20, wherein the storing step includes supplying, by the monitoring agent upon initialization, the identifiable configuration information to the corresponding one data collection resource via the corresponding secure connection for storage in the first database.

24. The method of claim 23, further comprising supplying, by the monitoring agent, updated identifiable configuration information to the corresponding one data collection resource for storage in the first database in response to detecting a change in the network device from the identifiable configuration information to the updated identifiable configuration information.

25. The method of claim 23, wherein:

the rules-based troubleshooting system includes a rules manager, a notification resource module, and a central communications system having an inventory management resource and a device troubleshooting resource, the central communications system configured for routing information between the at least one data collection resource, the first database, and the second database;

the determining step includes correlating the second database, by the rules manager, for the problem/resolution information for the network device based on correlating the identifiable configuration information, and the selected device state attributes, relative to network management rules specified according to an extensible markup language, the network management rules including network infrastructure rules for describing a network infrastructure and network element rules for describing network elements, including the network device, within the corresponding network infrastructure;

the providing step includes identifying, by the notification resource module, a manager contact for the network device and sending notification information, including the service notification message, to the manager contact;

the storing step includes loading the identifiable configuration information for the network device, received by the inventory management resource from a first message via the corresponding data collection resource, into the first database according to a prescribed database format; and

the determining step further includes responding, by the device troubleshooting resource, to the event message by fetching the identifiable configuration information for the network device associated with the event message, sending a query to the rules manager for correlation of the problem/resolution information, and sending results of the query to the notification resource module.

26. The method of claim 25, further comprising:

identifying, based on new problem/resolution information added to the second database, network devices affected by the new problem/resolution information based on correlating the new problem/resolution information to the respective identifiable information stored in the first database; and

outputting to the notification resource module an update notification for the network devices affected by the new problem/resolution information.

27. The system of claim 25, further comprising identifying optimum device configuration settings for the network device by outputting to the notification resource module a practice advisory message for the network device, based on correlating the corresponding identifiable configuration information with the stored problem/resolution information.

28. A network device comprising:

a monitoring agent configured for registering with a prescribed troubleshooting system based on outputting, upon initialization of the network device, identifiable configuration information that describes an identifiable configuration of the network device and that includes network topology information related to the network device, the monitoring agent further configured for monitoring device operations relative to monitored parameters, the monitoring agent configured for generating a notification, for delivery to the prescribed troubleshooting system, based on detecting an event where the device operations exceed at least one of the monitored parameters.

29. The device of claim 28, wherein the notification includes selected device state attributes associated with the event, the selected device state attributes including an identification of each process executed by the network device upon generation of the notification message, and memory resource utilization for each corresponding identified process.

30. The device of claim 28, wherein the monitoring agent is configured for supplying, to the prescribed troubleshooting system, updated identifiable configuration information in response to detecting a change in the network device from the identifiable configuration information to the updated identifiable configuration information.

31. The device of claim 28, wherein the network device is configured for receiving a response to the event, from the prescribed troubleshooting system, that specifies problem/resolution information for responding to the event.

32. The device of claim 28, wherein the network device is a router.

33. A method in a network device, the method comprising:

registering with a prescribed troubleshooting system, by a monitoring agent in the network device, based on outputting, upon initialization of the network device, identifiable configuration information that describes an identifiable configuration of the network device and that includes network topology information related to the network device,

monitoring device operations relative to monitored parameters by the monitoring agent, and

generating a notification by the monitoring agent, for delivery to the prescribed troubleshooting system, based on the monitoring agent detecting an event where the device operations exceed at least one of the monitored parameters.

34. The method of claim 33, wherein the generating includes supplying selected device state attributes associated with the event, the selected device state attributes including an identification of each process executed by the network device upon generation of the notification message, and memory resource utilization for each corresponding identified process.

35. The method of claim 33, further comprising supplying, to the prescribed troubleshooting system, updated identifiable configuration information in response to detecting a change in the network device from the identifiable configuration information to the updated identifiable configuration information.

36. The method of claim 33, further comprising receiving a response to the event, from the prescribed troubleshooting system, that specifies problem/resolution information for responding to the event.

37. The method of claim 33, wherein the network device is a router.

38. An apparatus comprising:

a rules-based troubleshooting resource configured for determining problem/resolution information for a network device, the rules-based troubleshooting resource configured for retrieving identifiable configuration information, having been supplied by the network device, in response to an event message having been initiated by the network device and including device state attributes of the network device, the rules-based troubleshooting resource configured for determining the problem/resolution information based on correlating the identifiable configuration information and the device state attributes relative to stored problem/resolution information that is stored in a database, the database storing the stored problem/resolution information for the network device according to known configurations;

the rules-based troubleshooting resource selectively outputting a service notification message as a response to the event message and that includes the problem/resolution information for the network device.

39. The apparatus of claim 38, wherein the rules-based troubleshooting resource is configured for correlating the identifiable configuration information, and the device state attributes, relative to network management rules specified according to an extensible markup language, the network management rules including network infrastructure rules for describing a network infrastructure and network element rules for describing network elements, including the network device, within the corresponding network infrastructure.

40. The apparatus of claim 38, wherein the rules-based troubleshooting resource is configured for retrieving the identifiable configuration information from a second database, configured for storing the identifiable configuration information for a respective plurality of network devices, in response to the event message.

41. The apparatus of claim 38, wherein the device state attributes include an identification of each process executed by the network device upon generation of the event message, and memory resource utilization for each corresponding identified process.

42. A method comprising:

retrieving, by a rules-based troubleshooting resource, identifiable configuration information having been supplied by a network device in response to an event message having been initiated by the network device and including device state attributes of the network device,

determining problem/resolution information for the network device, by the rules-based troubleshooting resource, based on correlating the identifiable configuration information and the device state attributes relative to stored problem/resolution information that is stored in a database, the database storing the stored problem/resolution information for the network device according to known configurations, and

selectively outputting, by the rules-based troubleshooting resource, a service notification message as a response to the event message and that includes the problem/resolution information for the network device.

43. The method of claim 42, wherein the determining includes correlating the identifiable configuration information, and the device state attributes, relative to network management rules specified according to an extensible markup language, the network management rules including network infrastructure rules for describing a network infrastructure and network element rules for describing network elements, including the network device, within the corresponding network infrastructure.

44. The method of claim 42, wherein the retrieving includes retrieving the identifiable configuration information from a second database, configured for storing the identifiable configuration information for a respective plurality of network devices, in response to the event message.

45. The method of claim 42, wherein the device state attributes include an identification of each process executed by the network device upon generation of the event message, and memory resource utilization for each corresponding identified process.