Error reporting to diagnostic engines based on their diagnostic capabilities

• US 7,328,376 B2
• Filed: 10/31/2003
• Issued: 02/05/2008
• Est. Priority Date: 10/31/2003
• Status: Active Grant

• Current Assignee: Oracle America, Inc. (Oracle Corporation)
• Original Assignee: Sun Microsystems Incorporated (Oracle Corporation)
• Inventors: McGuire, Cynthia A., Rudoff, Andrew M., Haley, Timothy P., Shapiro, Michael W., Simmons, Matthew T.
• Primary Examiner(s): Chu; Gabriel
• Assistant Examiner(s): SCHELL, JOSEPH O
• Application Number:
  US10/698,989
• Publication Number:
  US 20050102567A1
• Time in Patent Office: 1,558 Days
• Field of Search: 714/25, 714/26, 714/48, 714/57
• US Class Current: 714/48
• CPC Class Codes: G06F 11/2257 using expert systems

Abstract

A method, apparatus, and computer program product diagnosing and resolving faults is disclosed. A disclosed fault management architecture includes a fault manager suitable having diagnostic engines and fault correction agents. The diagnostic engines receive error information and identify associated fault possibilities. The fault possibility information is passed to fault correction agents, which diagnose and resolve the associated faults. The architecture uses logs to track the status of error information, the status of fault management exercises, and the fault status of system resources. Additionally, a soft error rate discriminator can be employed to track and resolve soft (correctible) errors in the system. The architecture is extensible allowing additional diagnostic engines and agents to be plugged in to the architecture without interrupting the normal operational flow of the computer system.

Claims

1. A computer network system having a fault management architecture configured for use in a computer network system, the computer network system comprising:

a plurality of nodes interconnected in a network; and

a fault manager mounted at a node on the network and configured to diagnose and resolve faults occurring at said node, wherein the fault manager is suitable for interfacing with diagnostic engines and fault correction agents, the fault manager being suitable for receiving error information and passing this information to diagnostic engines that have subscribed to receive the error information, and wherein the fault manager publishes error reports; and

wherein each diagnostic engine subscribes to selected error reports associated with fault diagnosis capabilities of said diagnostic engine so that when the fault manager publishes error reports only subscribing diagnostic engines receive the selected error reports;

at least one diagnostic engine for receiving error information and identifying a set of fault possibilities associated with errors contained in the error information;

at least one fault correction agent for receiving the set of fault possibilities from the at least one diagnostic engine and then selecting a diagnosed fault, and then taking appropriate fault resolution action concerning the selected diagnosed fault; and

logs for tracking a status of the error information, a status of the fault management exercises, and a fault status of the resources of the computer system.

2. The fault management architecture of claim 1 wherein the fault manager is configured to accommodate additional diagnostic engines and fault correction agents that can be added at a later time.

3. The fault management architecture of claim 2 wherein the fault manager is configured so that said additional diagnostic engines and additional fault correction agents can be added while the computer system is operating without interrupting its operation.

4. The fault management architecture of claim 1 wherein the fault correction agents resolve faults by initiating at least one of:

executing a corrective action on a selected diagnosed fault and generating a message identifying the selected diagnosed fault so that further action can be taken.

5. The fault management architecture of claim 4 wherein generating a message identifying the selected diagnosed fault so that further action can be taken includes identifying faulted resource and identifying a problem with the faulted resource.

6. The fault management architecture of claim 1 wherein the architecture further includes a data capture engine configured to obtain error information from the computer system and generate an error report that is provided to the fault manager.

7. The fault management architecture of claim 6 wherein the diagnostic engines and the agents are configured so that the fault manager continuously accumulates error reports from the data capture engine until enough error information is accumulated so that the diagnostic engines and the agents can successfully diagnose a fault associated with the error reports.

8. The fault management architecture of claim 6 wherein the fault manager stores the error reports generated by the data capture engine to an error report log of the logs;

wherein the at least one diagnostic engine stores fault management exercise information in a fault management exercise log of the logs; and

wherein the at least one fault correction agent stores fault status information concerning resources of the computer system in a resource cache of the logs.

9. The fault management architecture of claim 8 wherein information from the error report log and the fault management exercise log are stored in the resource cache.

10. The fault management architecture of claim 9 wherein resource cache is configured so that in the event of a computer system failure, the system can be restarted and information can be downloaded from the resource cache to reconstruct error history, fault management exercise history, and resource status, and use this information to conduct fault diagnosis.

11. The fault management architecture of claim 9 wherein resource cache is configured so that in the event of a computer system failure, the system can be restarted and information can be uploaded from the resource cache, the error report log, and

the fault management exercise log is to reconstruct error history, fault management exercise history, and resource status, and use this information to conduct fault diagnosis.

12. The fault management architecture of claim 1 wherein the diagnostic engine determines a probability of occurrence associated with each identified fault possibility.

13. The fault management architecture of claim 12 wherein the at least one fault correction agent for receiving the set of fault possibilities receives a relative probability of occurrence associated with each identified fault possibility from the diagnostic engines and then resolves a fault using a protocol.

14. The fault management architecture of claim 13 wherein the at least one fault correction agent resolves a set of fault possibilities using a protocol that incorporates at least one of:

an analysis of at least one of computer resource failure history, system management policy, and relative probability of occurrence for each fault possibility.

15. The fault management architecture of claim 1 wherein the fault manager stores provided error reports in a log comprising an error report log and wherein the error report log tracks the status of the provided error reports.

16. The fault management architecture of claim 1 wherein the fault manager includes a soft error rate discriminator that:

receives error information concerning correctable errors;

wherein the soft error rate discriminator is configured so that when the number and frequency of correctable errors exceeds a predetermined threshold number of correctable errors over a predetermined threshold amount of time, these errors are deemed recurrent correctable errors that are sent to the diagnostic engines for further analysis;

wherein the diagnostic engine receives a recurrent correctible error message and

diagnoses a set of fault possibilities associated with the recurrent correctible error message; and

wherein a fault correction agent receives the set of fault possibilities from the diagnostic engines and then resolves the diagnosed fault.

17. The fault management architecture of claim 16 wherein the soft error rate discriminator receives error information concerning correctable errors from the diagnostic engine.

18. The fault management architecture of claim 16 wherein the diagnostic engine that identifies a set of fault possibilities associated with the recurrent correctable error message further determines associated probabilities of occurrence for the set of fault possibilities associated with the recurrent correctable error message.

19. The fault management architecture of claim 18 wherein the a fault correction agent receives the set of fault possibilities and associated probabilities of occurrence from the diagnostic engines and the agent then takes appropriate action to resolve the set of fault possibilities.

20. The fault management architecture of claim 1 wherein the fault manager includes a soft error rate discriminator that:

receives error information concerning soft errors;

wherein the soft error rate discriminator is configured so that when the number and frequency of soft errors exceeds a predetermined threshold number of soft errors over a predetermined threshold amount of time, these soft errors are deemed recurrent soft errors that are sent to the diagnostic engines for further analysis;

wherein the diagnostic engine receives a recurrent soft error message and diagnoses a set of fault possibilities associated with the recurrent correctable error message; and

wherein a fault correction agent receives the set of fault possibilities from the diagnostic engines and then resolves the diagnosed fault.

21. The fault management architecture of claim 1 further including a fault management administrative tool that is configured to enable a user to access the logs to determine the fault status and error history of resources in the computer system.

22. The fault management architecture of claim 1 further including a fault management statistical file that can be reviewed to determine the effectiveness of the diagnostic engines and fault correction agents at diagnosing faults and resolving faults.

23. The fault management architecture of claim 1 wherein the computer system comprises a single computer device.

24. The fault management architecture of claim 1 wherein the computer system comprises a plurality of computers forming a network.