System and method for performing automated system management

• US 7,120,559 B1
• Filed: 06/29/2004
• Issued: 10/10/2006
• Est. Priority Date: 06/29/2004
• Status: Active Grant

• Current Assignee: Oracle America, Inc. (Oracle Corporation)
• Original Assignee: Sun Microsystems Incorporated (Oracle Corporation)
• Inventors: Williams, Emrys, Rudoff, Andrew
• Primary Examiner(s): Nghiem, Michael
• Assistant Examiner(s): LAU, TUNG S
• Application Number:
  US10/880,030
• Time in Patent Office: 833 Days
• Field of Search: 702/185
• US Class Current: 702/185
• CPC Class Codes: G06F 11/0706 the processing taking place on a specific hardware platform or in a specific software environment
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36

Abstract

A method and system are provided for performing automated management on a system. The method includes providing a fault tree representation of the system. The fault tree specifies propagations of errors generated in the system by problems to produce error reports. The method further includes receiving one or more error reports which are analyzed using the fault tree representation to determine one or more suspected problems. The suspected problems represent those problems that could have generated errors to produce the received error reports compatible with the propagations in the fault tree. The method further includes deconfiguring or reconfiguring one or more components in the system in response to the determination of the suspected problems.

Claims

1. A method for performing automated management on a system, comprising:

providing a fault tree representation of the system, wherein said fault tree representation indicates hierarchical cause and effect relationships between system events including propagations of errors generated in the system by problems;

receiving one or more error reports for the system;

analyzing the received one or more error reports using the fault tree representation to determine one or more suspected problems, said suspected problems representing those problems that could have generated errors to produce the received error reports compatible with the propagations in the fault tree representation;

storing data representing the one or more suspected problems; and

reconfiguring one or more components in the system in response to the determination of the one or more suspected problems.

2. The method of claim 1, wherein said reconfiguring comprises isolating one of the one or more components so that one or more errors generated within the isolated component do riot propagate elsewhere in the system.

3. The method of claim 1, wherein said analyzing comprises identifying an automatic system reconfiguration unit (ASRU) containing a suspected problem.

4. The method of claim 3, wherein said reconfiguring comprises deconfiguring the identified ASRU.

5. The method of claim 4, wherein deconfiguring the identified ASRU is subject to a configuration policy.

6. The method of claim 3, wherein said suspected problem is specified in the fault tree representation as being located in said identified ASRU.

7. The method of claim 3, wherein said identifying an ASRU comprises examining constraints on propagation in the fault tree representation.

8. The method of claim 1, wherein said reconfiguring is performed dependent upon determining consequences of reconfiguration using the fault tree representation.

9. The method of claim 8, wherein said determining comprises evaluating one or more problems known to exist already in the system.

10. The method of claim 1, further comprising remedying one or more errors in the system caused by said one or more suspected problems.

11. The method of claim 1, wherein said analyzing is performed on the system that produced the one or more received error reports.

12. The method of claim 1, further comprising forwarding error reports from the system that produced the error reports to another system;

wherein the other system is configured to perform said analyzing.

13. The method of claim 1, further comprising analyzing a newly received error report to determine whether it could have been caused by a previously suspected problem, and if so, attributing the newly received error report to said previously suspected problem.

14. The method of claim 1, wherein the one or more suspected problems comprise a suspect list, the method further comprising refining said suspect list based on historical knowledge regarding the system.

15. The method of claim 14, wherein said historical knowledge comprises information concerning problems previously diagnosed or repaired for the system.

16. The method of claim 1, wherein the one or more suspected problems comprise a suspect list, the method further comprising refining said suspect list based on a service policy for the system.

17. The method of claim 16, wherein said service policy indicates how the system should be reconfigured if multiple suspected problems are determined to be present.

18. The method of claim 1, wherein said reconfiguring comprises passing a new setting for a configuration variable to a configuration manager.

19. The method of claim 18, wherein the system stores information regarding values of said configuration variables over time.

20. A method for automatically assessing a proposed configuration of a system, comprising:

providing a fault tree representation of the system, wherein said fault tree representation indicates hierarchical cause and effect relationships between system events including propagations in the system of errors generated by problems;

providing a proposed system configuration; and

analyzing the proposed system configuration using the fault tree representation in accordance with one or more predetermined criteria;

wherein said analyzing comprises generating and storing a predicted result of implementing the proposed system configuration.

21. The method of claim 20, further comprising implementing the proposed system configuration only if said one or more predetermined criteria are met.

22. The method of claim 20, wherein said predetermined criteria relate to existence of one or more single points of failure.

23. The method of claim 20, wherein said predetermined criteria relate to an estimated failure in time rate.

24. The method of claim 23, wherein said fault tree representation includes information relating to said estimated failure in time rate.

25. The method of claim 20, wherein said analyzing comprises evaluating one or more problems suspected to exist already in the system.

26. The method of claim 20, wherein said one or more criteria are determined based on annotations indicating acceptability of particular errors.

27. The method of claim 26, wherein said annotations are based on customer input.

28. The method of claim 20, wherein the proposed configuration is rejected if it will change the system to a state where a further new problem could produce an unacceptable error.

29. Apparatus for performing automated management on a system, comprising:

a device configured to access a fault tree representation of the system, wherein said fault tree representation indicates hierarchical cause and effect relationships between system events including propagations of errors generated in the system by problems;

wherein the device is configured to implement;

a diagnosis system configured to analyze one or more error reports generated in the system using the fault tree representation to determine one or more suspected problems, said suspected problems representing those problems that could have generated errors to produce the one or more error reports compatible with the propagations in the fault tree representation; and

a configuration manager configured to reconfigure one or more components in the system in response to the determination of the one or more suspected problems.

30. The apparatus of claim 29, wherein to reconfigure the one or more components, said configuration manager is configured to isolate one of the one or more components so that one or more errors generated within the isolated component do not propagate elsewhere in the system.

31. The apparatus of claim 30, wherein to analyze the one or more error reports, the diagnosis system is configured to identify an automatic system reconfiguration unit (ASRU) containing a suspected problem.

32. The apparatus of claim 31, wherein to reconfigure the one or more components, the configuration manager is configured to deconfigure the identified ASRU.

33. The apparatus of claim 32, wherein to deconfigure the identified ASRU, the configuration manager is configured to follow a configuration policy.

34. The apparatus of claim 31, wherein said suspected problem is specified in the fault tree representation as being located in said identified ASRU.

35. The apparatus of claim 31, wherein to identify said ASRU, said diagnosis system is configured to examine constraints on propagation in the fault tree representation.

36. The apparatus of claim 29, wherein the configuration manager is configured to reconfigure the one or more components subject to a prior determination of consequences of reconfiguration performed using the fault tree representation.

37. The apparatus of claim 36, wherein to determine the consequences, the configuration manager is configured to evaluate one or more problems known to exist already in the system.

38. The apparatus of claim 29, wherein the device is further configured to implement a clean-up function, wherein the clean-up function is operable to remedy one or more errors in the system caused by said one or more suspected problems.

39. The apparatus of claim 29, wherein said device resides on the system that generated the one or more error reports.

40. The apparatus of claim 29, wherein said device is configured to receive one or more error reports forwarded from the system in which the error reports were generated to another system comprising the device.

41. The apparatus of claim 29, wherein the diagnosis system is configured to analyze a newly received error report to determine whether it could have been caused by a previously suspected problem, and if so, attributing the newly received error report to said previously suspected problem.

42. The apparatus of claim 29, wherein the one or more suspected problems comprise a suspect list, and wherein said diagnosis system is configured to refine said suspect list based on historical knowledge regarding the system.

43. The apparatus of claim 42, wherein said historical knowledge comprises information concerning problems previously diagnosed or repaired for the system.

44. The apparatus of claim 29, wherein the one or more suspected problems comprise a suspect list, and wherein said diagnosis system is configured to refine said suspect list based on a service policy for the system.

45. The apparatus of claim 44, wherein said service policy indicates how the system should be reconfigured if multiple suspected problems are determined to be present.

46. The apparatus of claim 29, wherein to reconfigure the one or more components, said configuration manager is configured to indicate a new setting for a configuration variable.

47. The apparatus of claim 29, wherein the configuration manager is configured to store information regarding values of said configuration variables over time in a configuration database.

48. Apparatus for automatically assessing a proposed configuration of a system, comprising:

a device configured to access a fault tree representation of the system, wherein said fault tree representation indicates hierarchical cause and effect relationships between system events including propagations in the system of errors generated by problems;

wherein said device is configured to implement;

a configuration manager configured to receive a proposed system configuration; and

an analysis engine configured to assess the proposed system configuration using the fault tree representation in accordance with one or more predetermined criteria.

49. The apparatus of claim 48, wherein to assess the proposed system configuration, the analysis engine is configured to:

determine if said one or more predetermined criteria are met; and

recommend implementing the proposed system configuration only if said one or more predetermined criteria are determined to be met.

50. The apparatus of claim 48, wherein said predetermined criteria relate to existence of one or more single points of failure.

51. The apparatus of claim 48, wherein said predetermined criteria relate to an estimated failure in time rate.

52. The apparatus of claim 51, wherein said fault tree representation includes information relating to said estimated failure in time rate.

53. The apparatus of claim 48, wherein to assess the proposed system configuration, said analysis engine is configured to evaluate one or more problems suspected to exist already in the system.

54. The apparatus of claim 48, wherein said one or more criteria are determined based on annotations indicating acceptability of particular errors.

55. The apparatus of claim 54, wherein said annotations are based on customer input.

56. The apparatus of claim 48, wherein the proposed configuration is rejected if it will change the system to a state where a further new problem could produce an unacceptable error.

57. A tangible computer program product, comprising program instructions computer-executable for performing automated management on a system, wherein said automated management comprises:

providing a fault tree representation of the system, wherein said fault tree representation indicates hierarchical cause and effect relationships between system events including propagations of errors generated in the system by problems;

receiving one or more error reports for the system;

analyzing the received one or more error reports using the fault tree representation to determine one or more suspected problems, said suspected problems representing those problems that could have generated errors to produce the received error reports compatible with the propagations in the fault tree representation;

storing data representing the one or more suspected problems; and

reconfiguring one or more components in the system in response to the determination of the one or more suspected problems.

58. A tangible computer program products comprising program instructions computer-executable for performing automated management on a system, wherein said automated management comprises:

providing a fault tree representation of the system, wherein said fault tree representation indicates hierarchical cause and effect relationships between system events including propagations in the system of errors generated by problems;

providing a proposed system configuration; and

analyzing the proposed system configuration using the fault tree representation in accordance with one or more predetermined criteria;

wherein said analyzing comprises generating and storing a predicted result of implementing the proposed system configuration.