Adaptive problem determination and recovery in a computer system

• US 7,194,445 B2
• Filed: 09/20/2002
• Issued: 03/20/2007
• Est. Priority Date: 09/20/2002
• Status: Expired due to Fees

• Current Assignee: Lenovo PC International Limited (Lenovo Group Ltd.)
• Original Assignee: Lenovo Singapore Pte Limited (Lenovo Group Ltd.)
• Inventors: Kephart, Jeffrey Owen, Chan, Hoi Yeung, Segal, Alla, Das, Rajarshi
• Primary Examiner(s): Vincent; David
• Assistant Examiner(s): COUGHLAN, PETER D
• Application Number:
  US10/252,979
• Publication Number:
  US 20040059966A1
• Time in Patent Office: 1,642 Days
• Field of Search: 706/1, 706/15, 706/45, 706/13, 706/14, 706/20, 706/47, 700/1, 700/90, 700/26, 709/219, 709/223, 395/76
• US Class Current: 706/20
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0775 Content or structure details of the error report, e.g. specific table structure, specific error fields
• CPC Class Codes: G06F 11/0781 Error filtering or prioritizing based on a policy defined by the user or on a policy defined by a hardware/software module, e.g. according to a severity level
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/2257 using expert systems
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 2201/86 Event-based monitoring

Abstract

A method, computer program product, and data processing system for recognizing, tracing, diagnosing, and repairing problems in an autonomic computing system is disclosed. Rules and courses of actions to follow in logging data, in diagnosing faults (or threats of faults), and in treating faults (or threats of faults) are formulated using an adaptive inference and action system. The adaptive inference and action system includes techniques for conflict resolution that generate, prioritize, modify, and remove rules based on environment-specific information, accumulated time-sensitive data, actions taken, and the effectiveness of those actions. Thus, the present invention enables a dynamic, autonomic computing system to formulate its own strategy for self-administration, even in the face of changes in the configuration of the system.

Claims

1. A computer-based method for providing problem determination and error recovery features to a computing environment, the method comprising:

receiving information regarding a status of the computing environment;

identifying at least three applicable rules from a knowledge base of rules, wherein the at least three applicable rules are applicable to the status of the computing environment; and

applying the at least three applicable rules to obtain a result,

wherein the at least three applicable rules from the knowledge base of rules includes a logging logic rule used by a logging logic module and associated inference engine in specifying that particular events should be logged by system components under particular circumstances, a problem determination logic rule used by a problem determination logic module and associated inference engine in specifying that a presence of particular information contained within event logs indicates a particular problem, and an error recovery logic rule used by an error recovery logic module and associated inference engine in specifying that a particular problem implies a particular solution to the particular problem should be followed.

2. The method of claim 1, wherein a result of the logging rule is a logging policy specifying how particular events should be logged by the system components to generate the event logs.

3. The method of claim 2, wherein the logging policy includes an identification of the system components to perform logging of the particular events.

4. The method of claim 2, wherein the logging policy includes specified conditions under which the particular events are to be logged.

5. The method of claim 2, wherein the logging policy includes a specified level of detail at which the particular events are to be logged.

6. The method of claim 2, wherein a result of the problem determination logic rule is a problem diagnosis of a future problem likely to occur and is used by the error recovery logic rule to reduce chances that the future problem will actually occur.

7. The method of claim 6, wherein the at least one problem determination logic rule correlates particular information contained within event logs to problems using a statistical test.

8. The method of claim 6, wherein the at least one problem determination logic rule correlates particular information contained within event logs to problems using a machine learning algorithm.

9. The method of claim 6, wherein one potential result of the error recovery logic rule is a course of action to follow in resolving a problem and another potential result of the error recovery logic rule is adapting the logging logic module to specify different particular events that should be logged by the system components.

12. The method of claim 1, further comprising:

applying a machine learning algorithm to add additional rules to the knowledge base.

13. The method of claim 12, wherein the machine learning algorithm is an inductive logic programming algorithm.

14. The method of claim 1, further comprising:

determining a degree of relevance of a rule in the knowledge base; and

in response to a determination that the rule has a low degree of relevance, removing the rule.

15. The method of claim 1, further comprising:

determining a degree of relevance of a rule in the knowledge base; and

in response to a determination that the rule has a low degree of relevance, assigning the rule a low priority in the knowledge base.

16. The method of claim 1, further comprising:

applying a machine learning algorithm to modify rules within the knowledge base.

17. The method of claim 16, wherein the machine learning algorithm is a supervised learning algorithm.

18. The method of claim 17, wherein the supervised learning algorithm is one of a neural network, a Bayesian network, and Support Vector Machines.

19. The method of claim 16, wherein the machine learning algorithm is an unsupervised learning algorithm.

20. The method of claim 19, wherein the unsupervised learning algorithm is one of k-means clustering, hierarchical clustering, and principal component analysis.

21. The method of claim 1, wherein receiving the information regarding the status of to computing environment, identifying the at least one applicable rule, and applying the at least one applicable rule are performed in response to a request from a client.

10. A computer-based method for providing problem determination and error recovery features to a computing environment, the method comprising:

receiving information regarding a status of the computing environment;

identifying at least three applicable rules from a knowledge base of rules, wherein the at least three applicable rules are applicable to the status of the computing environment;

applying the at least three applicable rules to obtain a result, wherein the at least three applicable rules, utilized by corresponding inference engines includes a logging logic rule specifying that particular events should be logged by system components under particular circumstances, a problem determination logic rule specifying that a presence of particular information contained within event logs indicates a particular problem, and an error recovery logic rule specifying that a particular problem implies a particular solution to the particular problem should be followed, wherein result of the error recovery logic rule is a course of action to follow in resolving a problem;

following the course of action to resolve the problem; and

in response to following the course of action, determining a degree of success of the course of action.

11. The method of claim 10, further comprising:

prioritizing the rules in the knowledge base in response to the degree of success of the course of action.

22. A computer program product in a computer-readable medium comprising functional descriptive material that, when executed by a computer, enables the computer to perform acts including:

receiving information regarding a status of the computing environment;

identifying at least three applicable rules from a knowledge base of rules, wherein the at least three applicable rules are applicable to the status of the computing environment; and

applying the at least three applicable rules to obtain a result,

wherein the at least three applicable rules from the knowledge base of rules includes a logging logic rule used by a logging logic module and associated inference engine in specifying that particular events should be logged by system components under particular circumstances, a problem determination logic rule used by a problem determination logic module and associated inference engine in specifying that a presence of particular information contained within event logs indicates a particular problem, and an error recovery logic rule used by an error recovery logic module and associated inference engine in specifying that a particular problem implies a particular solution to the particular problem should be followed.

23. The computer program product of claim 22, wherein a result of the logging logic rule is a logging policy specifying how particular events should be logged by the system components to generate the event logs.

24. The computer program product of claim 23, wherein the logging policy includes an identification of the system components to perform logging of the particular events.

25. The computer program product of claim 23, wherein the logging policy includes specified conditions under which the particular events are to be logged.

26. The computer program product of claim 23, wherein the logging policy includes a specified level of detail at which the particular events are to be logged.

27. The computer program product of claim 23, wherein a result of the problem determination logic rule is a problem diagnosis of a future problem likely to occur and is used by the error recovery logic rule to reduce chances that the future problem will actually occur.

28. The computer program product of claim 27, wherein the at least one problem determination logic rule correlates particular information contained within event logs to problems using a statistical test.

29. The computer program product of claim 27, wherein the at least one problem determination logic rule correlates particular information contained within event logs to problems using a machine learning algorithm.

30. The computer program product of claim 27, wherein one potential result of the error recovery logic rule is a course of action to follow in resolving a problem and another potential result of the error recovery logic rule is adapting the logging logic module to specify different particular events that should be logged by the system components.

33. The computer program product of claim 22, comprising additional functional descriptive material that, when executed by the computer, enables the computer to perform additional acts including:

applying a machine learning algorithm to add additional rules to the knowledge base.

34. The computer program product of claim 33, wherein the machine learning algorithm is an inductive logic programming algorithm.

35. The computer program product of claim 22, comprising additional functional descriptive material that, when executed by the computer, enables the computer to perform additional acts including:

determining a degree of relevance of a rule in the knowledge base; and

in response to a determination that the rule has a low degree of relevance, removing the rule.

36. The computer program product of claim 22, comprising additional functional descriptive material that, when executed by the computer, enables the computer to perform additional acts including:

determining a degree of relevance of a rule in the knowledge base; and

in response to a determination that the rule has a low degree of relevance, assigning the rule a low priority in the knowledge base.

37. The computer program product of claim 22, comprising additional functional descriptive material that, when executed by the computer;

enables the computer to perform additional acts including;

applying a machine learning algorithm to modify rules within the knowledge base.

38. The computer program product of claim 37, wherein the machine learning algorithm is a supervised learning algorithm.

39. The computer program product of claim 38, wherein the supervised learning algorithm is one of a neural network, a Bayesian network, and Support Vector Machines.

40. The computer program product of claim 37, wherein the machine learning algorithm is an unsupervised learning algorithm.

41. The computer program product of claim 40, wherein the unsupervised learning algorithm is one of k-means clustering, hierarchical clustering, and principal component analysis.

42. The computer program product of claim 22, wherein receiving the information regarding the status of the computing environment, identifying the at least one applicable rule, and applying the at least one applicable rule are performed in response to a request from a client.

31. A computer program product in a computer-readable medium comprising functional descriptive material that, when executed by a computer, enables the computer to perform acts including:

receiving information regarding a status of the computing environment;

identifying at least three applicable rules from a knowledge base of rules, wherein the at least three applicable rules are applicable to the status of the computing environment;

applying the at least three applicable rules to obtain a result, wherein the at least three applicable rules from the knowledge base of rules, utilized by corresponding inference engines includes a logging logic rule specifying that particular events should be logged by system components under particular circumstances, a problem determination logic rule specifying that a presence of particular information contained within event logs indicates a particular problem, and an error recovery logic rule specifying that a particular problem implies a particular solution to the particular problem should be followed, wherein a result of the error recovery logic rule is a course of action to follow in resolving a problem;

following the course of action to resolve the problem; and

in response to following the course of action, determining a degree of success of the course of action.

32. The computer program product of claim 31, comprising additional functional descriptive material that, when executed by the computer, enables the computer to perform additional acts including:

prioritizing the rules in the knowledge base in response to the degree of success of the course of action.

43. A data processing machine comprising:

means for receiving information regarding a status of the computing environment;

means for identifying at least three applicable rules from a knowledge base of rules, wherein the at least three applicable rules are applicable to the status of the computing environment; and

means for applying the at least three applicable rules to obtain a result,

wherein the at least three applicable rules from the knowledge base of rules includes a logging logic rule used by a logging logic module and associated inference engine in specifying that particular events should be logged by system components under particular circumstances, a problem determination logic rule used a problem determination logic module and associated inference engine in specifying that a presence of particular information contained within event logs indicates a particular problem, and an error recovery logic rule used by an error recovery logic module and associated inference engine in specifying that a particular problem implies a particular solution to the particular problem should be followed.

44. The data processing machine of claim 43, wherein a result of the logging logic rule is a logging policy specifying how particular events should be logged by the system components to generate the event logs.

45. The data processing machine of claim 44, wherein the logging policy includes an identification of the system components to perform logging of the particular events.

46. The data processing machine of claim 44, wherein the logging policy includes specified conditions under which the particular events are to be logged.

47. The data processing machine of claim 44, wherein the logging policy includes a specified level of detail at which the particular events are to be logged.

48. The data processing machine of claim 44, wherein a result of the problem determination logic rule is a problem diagnosis of a future problem likely to occur and is used by the error recovery logic rule to reduce chances that the future problem will actually occur.

49. The data processing machine of claim 48, wherein the at least one problem determination logic rule correlates particular information contained within event logs to problems using a statistical test.

50. The data processing machine of claim 48, wherein the at least one problem determination logic rule correlates particular information contained within event logs to problems using a machine learning algorithm.

51. The data processing machine of claim 48, wherein one potential result of the error recovery logic rule is a course of action to follow in resolving a problem and another potential result of the error recovery logic rule is adapting the logging logic module to specify different particular events that should be logged by the system components.

54. The data processing machine of claim 43, further comprising:

means for applying a machine learning algorithm to add additional rules to the knowledge base.

55. The data processing machine of claim 54, wherein the machine learning algorithm is an inductive logic programming algorithm.

56. The data processing machine of claim 43, further comprising:

means for determining a degree of relevance of a rule in the knowledge base; and

means, responsive to a determination that the rule has a low degree of relevance, for removing the rule.

57. The data processing machine of claim 43, further comprising:

means for determining a degree of relevance of a rule in the knowledge base; and

means, responsive to a determination that the rule has a low degree of relevance, for assigning the rule a low priority in the knowledge base.

58. The data processing machine of claim 43, further comprising:

means for applying a machine learning algorithm to modify rules within the knowledge base.

59. The data processing machine of claim 58, wherein the machine learning algorithm is a supervised learning algorithm.

60. The data processing machine of claim 59, wherein the supervised learning algorithm is one of a neural network, a Bayesian network, and Support Vector Machines.

61. The data processing machine of claim 58, wherein the machine learning algorithm is an unsupervised learning algorithm.

62. The data processing machine of claim 61, wherein the unsupervised learning algorithm is one of k-means clustering, hierarchical clustering, and principal component analysis.

63. The data processing machine of claim 43, wherein receiving the information regarding the status of the computing environment, identifying the at least one applicable rule, and applying the at least one applicable rule are performed in response to a request from a client.

52. A data processing machine comprising:

means for receiving information regarding a status of the computing environment;

means for identifying at least three applicable rules from a knowledge base of rules, wherein the at least three applicable rules are applicable to the status of the computing environment;

means for applying the at least three applicable rules to obtain a result, wherein the at least three applicable rules from the knowledge base of rules, utilized by corresponding inference engines includes a logging logic rule specifying that particular events should be logged by system components under particular circumstances, a problem determination logic rule specifying that a presence of particular information contained within event logs indicates a particular problem, and an error recovery logic rule specifying that a particular problem implies a particular solution to the particular problem should be followed, wherein a result of the error recovery logic rule is a course of action to follow in resolving a problem;

means for following the course of action to resolve the problem; and

means, responsive to following the course of action, for determining a degree of success of the course of action.

53. The data processing machine of claim 52, further comprising:

means for prioritizing the rules in the knowledge base in response to the degree of success of the course of action.