Integration of cluster information with root cause analysis tool

• US 7,721,152 B1
• Filed: 12/21/2004
• Issued: 05/18/2010
• Est. Priority Date: 12/21/2004
• Status: Expired due to Fees

• Current Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Original Assignee: Symantec Operating Corporation (Gen Digital Inc.)
• Inventors: Shankar, Asim, Joshi, Darshan B., Senicka, James A.
• Primary Examiner(s): LE, DIEU MINH T
• Application Number:
  US11/017,895
• Time in Patent Office: 1,974 Days
• Field of Search: 714/25, 714/26, 714/27, 714/15, 714/16, 717/124, 717/126, 717/128
• US Class Current: 714/25
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: H04L 41/0604 using filtering, e.g. reduction of information by using priority, element types, position or time
• CPC Class Codes: H04L 41/0631 using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
• CPC Class Codes: H04L 41/0893 Assignment of logical groups to network elements

Abstract

A method, system, computer system, and computer-readable medium to perform root cause analysis of failure of an application program in a clustering environment. Upon the occurrence of a problem or event of interest, cluster configuration data can be obtained to provide a common context for events occurring at different software layers supporting an application. Diagnostic information produced by the different software layers can be obtained from various log files, which are typically in different formats and on different nodes in the cluster. The diagnostic information can be viewed in the context of the cluster and filtered to identify events related to the failure. The related events can be presented in a time-ordered sequence to assist in analysis of the event of interest. Patterns of events that led to the failure can be identified and documented for use in further problem analysis and for taking preventative and/or corrective measures.

Claims

1. A method comprising:

obtaining a selected event from a plurality of events related to a cluster of nodes;

identifying a resource in the cluster of nodes, wherein

the resource is related to the selected event;

selecting a subset of the plurality of events, wherein

the selecting comprises

using cluster configuration data for the cluster of nodes,

identifying a filter related to the resource, and

applying the filter to the plurality of events, and

the identifying the filter is based on the cluster configuration data and the selected event; and

providing the subset of the plurality of events, wherein

the subset of the plurality of events is configured to be used in determining a cause of the selected event.

2. The method of claim 1, wherein

the plurality of events is obtained from a plurality of logs,

the plurality of events comprises the selected event and a second event,

the selected event occurs at a first software layer of a first node of the cluster of nodes,

the second event occurs at a second software layer of a second node of the cluster of nodes, and

a comprehensive event log comprises the selected event and the second event.

3. The method of claim 2, wherein

the plurality of logs is produced by a plurality of software layers comprising the first software layer and the second software layer, and

the method further comprises ordering events in the comprehensive event log by a resource in a time sequence.

4. The method of claim 2, wherein

the plurality of logs comprises at least one of:

a first application log on the first node,

a first database log on the first node,

a first cluster log on the first node,

a first file system log on the first node,

a first operating system event log on the first node,

a second application log on the second node,

a second database log on the second node,

a second cluster log on the second node,

a second file system log on the second node, and

a second operating system event log on the second node.

5. The method of claim 1, further comprising:

determining the cause of the selected event, and

analyzing the subset of the plurality of events for a pattern to use in analyzing a future failure.

6. The method of claim 1, further comprising:

obtaining the cluster configuration data for the cluster of nodes.

7. The method of claim 1, further comprising:

identifying a resource in the cluster of nodes related to the selected event; and

selecting the subset of the plurality of events by searching for an event related to the resource.

8. A system comprising:

a selected node among a cluster of nodes, the selected node comprising a processor;

obtaining means for obtaining a selected event from a plurality of events related to the cluster of nodes, wherein

the obtaining means is coupled to the processor;

identifying means for identifying a resource in the cluster of nodes, wherein

the resource is related to the selected event;

selecting means for selecting a subset of the plurality of events, wherein

the selecting means is coupled to the processor,

the selecting means comprises means for using cluster configuration data for the cluster of nodes,

the selecting means further comprises

means for identifying a filter related to the resource, and

means for applying the filter to the plurality of events, and

the means for identifying the filter comprises means for determining the filter based on the cluster configuration data and the selected event; and

providing means for providing the subset of the plurality of events, wherein

the providing means is coupled to the processor, and

the subset of the plurality of events is configured to be used in determining a cause of the selected event.

9. The system of claim 8, further comprising:

searching means, coupled to the processor, for selecting the subset of the plurality of events by searching for an event related to the resource.

10. A computer-readable storage medium comprising:

obtaining instructions configured to obtain a selected event from a plurality of events related to a cluster of nodes;

identifying instructions configured to identify a resource in the cluster of nodes, wherein

the resource is related to the selected event;

selecting instructions configured to select a subset of the plurality of events, wherein

the selecting instructions comprise instructions for using cluster configuration data for the cluster of nodes,

the selecting instructions comprise;

instructions for identifying a filter related to the resource, and

instructions for applying the filter to the plurality of events, and

the instructions for identifying the filter comprise instructions for identifying the filter based on the cluster configuration data and the selected event;

and

providing instructions configured to provide the subset of the plurality of events, wherein the subset of the plurality of events is configured to be used in determining a cause of the selected event.

11. The computer-readable storage medium of claim 10, wherein

the plurality of events is obtained from a plurality of logs,

the plurality of events comprises the selected event and a second event,

the selected event occurs at a first software layer of a first node of the cluster of nodes,

the second event occurs at a second software layer of a second node of the cluster of nodes, and

a comprehensive event log comprises the selected event and the second event.

12. The computer-readable storage medium of claim 11, wherein

the plurality of logs is produced by a plurality of software layers comprising the first software layer and the second software layer, and

the providing instructions further comprise instructions for ordering events in the comprehensive event log by a resource in a time sequence.

13. The computer-readable storage medium of claim 11 wherein

the plurality of logs comprises at least one of:

a first application log on the first node,

a first database log on the first node,

a first cluster log on the first node,

a first file system log on the first node,

a first operating system event log on the first node,

a second application log on the second node,

a second database log on the second node,

a second cluster log on the second node,

a second file system log on the second node, and

a second operating system event log on the second node.

14. The computer-readable storage medium of claim 10, further comprising:

identifying instructions configured to identify a resource in the cluster of nodes related to the selected event; and

searching instructions configured to select the subset of the plurality of events by searching for an event related to the resource.

15. A system comprising:

a selected node among a cluster of nodes, the selected node comprising a processor;

an identifying module, stored in a memory coupled to the processor, configured to identify a resource in the cluster of nodes, wherein

the resource is related to the selected event;

an obtaining module, stored in the memory coupled to the processor, configured to obtain a selected event from a plurality of events related to the cluster of nodes;

a selecting module, stored in the memory coupled to the processor, configured to select a subset of the plurality of events, wherein

the selecting module comprises a cluster configuration data use module configured to use cluster configuration data for the cluster of nodes,

the selecting module comprises a filtering module configured to

identify a filter related to the resource, and

apply the filter to the plurality of events, and

the filtering module is configured to identify the filter based on the cluster configuration data and the selected event; and

a providing module, stored in the memory coupled to the processor, configured to provide the subset of the plurality of events, and

the subset of the plurality of events is configured to be used in determining a cause of the selected event.

16. The system of claim 15, wherein

the plurality of events is obtained from a plurality of logs,

the plurality of events comprises the selected event and a second event,

the selected event occurs at a first software layer of a first node of the cluster of nodes,

the second event occurs at a second software layer of a second node of the cluster of nodes, and

a comprehensive event log comprises the selected event and the second event.

17. The system of claim 16, wherein

the plurality of logs is produced by a plurality of software layers comprising the first software layer and the second software layer, and

the system further comprises an ordering module configured to order a plurality of events in the comprehensive event log by a resource in a time sequence.

18. The system of claim 16 wherein

the plurality of logs comprises at least one of:

a first application log on the first node,

a first database log on the first node,

a first cluster log on the first node,

a first file system log on the first node,

a first operating system event log on the first node,

a second application log on the second node,

a second database log on the second node,

a second cluster log on the second node,

a second file system log on the second node, and

a second operating system event log on the second node.

19. The system of claim 15, further comprising:

an identifying module, coupled to the processor, configured to identify a resource in the cluster of nodes related to the selected event; and

a searching module, coupled to the processor, configured to select the subset of the plurality of events by searching for an event related to the resource.