Computer recovery method and system for recovering automatically from fault, and fault monitoring apparatus and program used in computer system

• US 20030070114A1
• Filed: 10/07/2002
• Published: 04/10/2003
• Est. Priority Date: 10/05/2001
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Yasuda, Shigeru
• Granted Patent: US 7,069,473 B2
• Time in Patent Office: Days
• US Class Current: 714/20
• CPC Class Codes: G06F 11/0712 in a virtual computing platform, e.g. logically partitioned systems
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40

Abstract

A fault monitoring apparatus is connected to computer systems and monitors a fault in the computer systems. The fault monitoring apparatus is provided with a storage section for storing and holding fault recovery information including rules for defining recovery operations when faults occur in the computer systems, and when a fault occurs in the computer systems, retrieves the rules previously set in the fault recovery information and instructs the computer systems so as to perform a recovery operation corresponding to a rule matching to the fault which occurs in the computer systems.

Claims

1. A computer recovery system for recovering automatically from a fault comprising at least one computer system and a fault monitoring apparatus for monitoring a fault in said at least one computer system,

wherein said fault monitoring apparatus comprises a storage section for storing and holding fault recovery information including rules for defining recovery operations when faults occur in said at least one computer system, and a recovery instructing section, when a fault occurs in said at least one computer system, for retrieving said rules previously set in said fault recovery information and for instructing said at least one computer system in such a manner that a recovery operation corresponding a rule matching to said fault which occurs in said at least one computer system is carried out.

2. The computer recovery system for recovering automatically from a fault according to claim 1, wherein said fault recovery information includes a rule in which a condition and a procedure for executing said recovery operation from said fault are designated using a rule designation language which is previously set.

3. The computer recovery system for recovering automatically from a fault according to claim 1, wherein said at least one computer system is provided with fault monitoring agent communicating with said fault monitoring apparatus, and wherein when a fault occurs in said at least one computer system, said fault monitoring agent acting in said at least one computer system notify said fault monitoring apparatus of fault information concerning said fault which occurs.

4. The computer recovery system for recovering automatically from a fault according to claim 1, wherein said fault recovery information includes component configuration information having information about each component making up said at least one computer system as an object to be monitored by said fault monitoring apparatus, wherein when said at least one computer system becomes said object to be monitored by said fault monitoring apparatus, said fault monitoring apparatus instructs to a fault monitoring agent acting in said at least one computer system to notify said information concerning each component in said at least one computer system and configuration information concerning connection of said each component, and wherein said fault monitoring apparatus automatically obtains said information concerning said each component from said at least one computer system which is said object of monitoring and sets said information concerning said each component in said component configuration information.

5. The computer recovery system for recovering automatically from a fault according to claim 1, wherein said fault recovery information includes component characteristic information including information showing characteristics of components making up said at least one computer system to be monitored by said fault monitoring apparatus, and wherein said fault monitoring apparatus, when instructing a fault recovery operation to said at least one computer system in which said fault occurs, considers efficiencies concerning components included in said at least one computer system based on said component characteristic information, and instructs said at least one computer system in which said fault occurs so as to select components which are used efficiently.

6. The computer recovery system for recovering automatically from a fault according to claim 1, wherein said fault recovery information includes classification of each of faults which occurred previously and configuration information of said at least one computer system at a time at which a fault occurs as fault example storage information, and wherein said fault monitoring apparatus further comprises an avoidance instructing section which, when instructing a fault recovery operation to said at least one computer system in which said fault occurs, refers to fault information of past occurrences, and instructs said at least one computer system in which experienced said fault so as to avoid a computer system configuration in which a fault is apt to occur.

7. The computer recovery system for recovering automatically from a fault according to claim 1, wherein said fault recovery information includes a system requisite rule which is a rule of a computer system to be met by said at least one computer system which is an object of monitoring and information for defining an operation for satisfying said rule as computer system configuration rule information, and wherein said fault monitoring apparatus further comprises a change instructing section which, when fault recovery of said at least one computer system in which a fault occurs, instructs said at least one computer system to change a computer system configuration after said fault recovery operation in accordance with a request specification of a whole system of said at least one computer system based on said system requisite rule.

8. The computer recovery system for recovering automatically from a fault according to claim 1, wherein in said fault monitoring apparatus, said fault recovery information includes a fault type judging rule in which, when a fault occurs in said at least one computer system, a rule for judging at which position said fault occurs and what type of a fault as a recovery rule, and an operation specifying section in which an operation is specified when said fault occurs, wherein when a fault occurs in said at least one computer system, said fault information indicating a fault cause is notified from said fault monitoring agent to said fault monitoring apparatus, wherein said fault monitoring apparatus which receives said fault information refers to said fault recovery rule, retrieves a fault type judging rule corresponding to a condition of said fault which occurs, and wherein said fault monitoring apparatus instructs said fault monitoring agent of an operation of contents described in said operation specifying section corresponding to said fault type judging rule matching said fault.

9. The computer recovery system for recovering automatically from a fault according to claim 1, wherein said fault monitoring apparatus comprises:

a first processing section, when a fault occurs in said at least one computer system, for checking whether said fault which occurs is a component fault or not, and for, when said fault is said component fault, storing system configuration information at a time when said fault occurs in a storage area for memorizing fault examples as fault example storage information;

a second processing section for referring to said fault example storage information to refer to past fault examples, for checking whether or not there is a same fault example that has occurred this time, for, when there is said same fault example, comparing system configuration information in the past same fault example with a computer system configuration in which said fault occurs, for extracting a feature of said computer system configuration, and for memorizing said characteristic related to said fault information as fault example storage information;

a counting section for counting a frequency of fault occurrences for every feature of said computer system configuration when said fault occurs based on an extracted feature of computer system configuration; and

a third processing section for checking a frequency of fault occurrences for every feature of said computer system configuration, and for registering a rule for avoiding an extracted feature of a computer system configuration in said component characteristic information, when said frequency of fault occurrences is more than a predetermined number.

10. The computer recovery system for recovering automatically from a fault according to claim 8, wherein in said fault type judging rule, it is specified whether said fault type judging rule is applied to all of computer systems monitored by said fault monitoring apparatus or said fault type judging rule is applied only to at least one specific computer system.

11. The computer recovery system for recovering automatically from a fault according to claim 10, wherein in said fault type judging rule, an order of priority is set, said fault type judging rule is retrieved in said order of priority, and a fault type judging rule which coincides first is selected.

12. The computer recovery system for recovering automatically from a fault according to claim 8, wherein a fault type judging rule used in a case of an unknown fault occurrence is previously prepared, and an operation specifying section corresponding to said fault type judging rule is registered in a lowest order of priority.

13. The computer recovery system for recovering automatically from a fault according to claim 8, wherein in said fault monitoring apparatus, a condition where a load of an operating system exceeds a predetermined load state is previously registered as said fault type judging rule, and an operation is defined in which a CPU (Central Processing Unit) board is added to a corresponding computer system as a fault recovery operation corresponding to said fault type judging rule.

14. The computer recovery system for recovering automatically from a fault according to claim 5, wherein said at least one computer system has a plurality of partitions respectively made up of a sub-computer system, and wherein said partitions are defined in said component characteristic information as alternative components, and when a fault occurs in an arbitrary component making up one of said plurality of partitions, said component is automatically changed to an alternative component.

15. The computer recovery system for recovering automatically from a fault according to claim 8, wherein said at least one computer system has a plurality of partitions respectively made up of a sub-computer system, wherein said fault monitoring apparatus is provided with said fault type judging rule and said operation specifying section which are different for each of said at least one computer systems, and wherein when said operating system differs for each of said partitions, said fault monitoring apparatus integrates each of partitions and executes an automatic fault recovery operation.

16. The computer recovery system for recovering automatically from a fault according to claim 1, wherein said fault monitoring apparatus is connected to a plurality of computer systems via a network, wherein a backup component used in said computer systems, wherein fault recovery operation information, when a fault occurs in a component in one system of said computer systems, for removing said component automatically from said system and for arranging an alternative component in said system, is previously defined in said fault monitoring apparatus.

17. The computer recovery system for recovering automatically from a fault according to claim 1, wherein a plurality of computer systems is provided, and wherein a standby computer system is operated instead of an active computer system in which a fault occurs.

18. The computer recovery system for recovering automatically from a fault according to claim 1, wherein each of said plurality computer systems makes up a cluster system whereby a node is configured, and wherein said fault monitoring apparatus includes at least one piece of node information, information showing that each node is capable of being a cluster with which node, and communication speed information of each network in said fault recovery information.

19. A computer recovery method for recovering automatically from a fault, wherein, in a fault monitoring apparatus for monitoring a fault in at least one computer system, fault recovery information including a rule for defining a recovery operation when a fault occurs in said at least one computer system is memorized, and wherein when said fault occurs in said at least one computer system, said fault monitoring apparatus instructs said at least one computer system to refer to said rule which is previously set in said fault recovery information, to perform a fault recovery operation corresponding to said fault which occurs and to execute said recovery operation corresponding to said rule.

20. The computer recovery system for recovering automatically from a fault according to claim 19, wherein said fault recovery information includes a rule in which a condition and a procedure for executing said recovery operation from said fault are designated using a rule designation language which is previously set.

21. A computer recovery method for recovering automatically from a fault comprising:

a first step, when a fault occurs in at least one computer system, of notifying a fault monitoring apparatus of fault information by a fault monitoring agent in said at least one computer system in which said fault occurs;

a second step, by said fault monitoring apparatus, of storing said fault information in a fault example storage area, and of extracting a feature of a computer system configuration for said fault information;

a third step, by said fault monitoring apparatus, of referring to a fault recovery rule, of retrieving a fault type judging rule corresponding to a condition, and of instructing a fault monitoring agent to execute an operation described in a corresponding operation specifying section; and

a fourth step, by said fault monitoring apparatus, of referring to a system configuration rule, of checking whether all of system requisite rules are met or not, and of instructing said fault monitoring agent to execute an operation described in said operation specifying section corresponding to said system requisite rule when there exists system requisite rule which said at least one computer system does not met.

22. The computer recovery method for recovering automatically from a fault according to claim 21, wherein said fault monitoring apparatus, comprises a fault recovery information section,

said fault recovery information including:

component configuration information in which information concerning each component in said at least one computer system as an object to be monitored by said fault monitoring apparatus is registered,

component characteristic information including information showing characteristics of components making said at least one computer system as said object to be monitored,

a fault example storage area for memorizing a history of said fault information which occurs in said at least one computer system as said object to be monitored,

a system configuration rule including a system requisite rule which is a rule of a computer system configuration to be met by said at least one computer system as said object to be monitored and an operation specifying section for defining an operation satisfying said system configuration rule, and

a fault recovery rule including a fault type judging rule in which when a fault occurs in said at least one computer system, a rule for judging at which position said fault occurs and what type of said fault and an operation specifying section in which an operation to be executed when a fault occurs is specified.

23. The computer recovery method for recovering automatically from a fault according to claim 22, wherein said second step comprises:

a fifth step, when a fault occurs in said at least one computer system, by said fault monitoring apparatus, of checking whether said fault which occurs is a component fault or not, and of memorizing system configuration information at a time of fault occurrence in said fault example storage area in a case of said component fault;

a sixth step, by said fault monitoring apparatus, of referring to said fault example storage area to refer to past fault examples, of checking whether or not there exists a same fault example that has occurred this time, and when there is said same past fault example, of comparing a computer system configuration in said same past fault example with a computer system configuration in which said fault has occurred this time, of extracting a feature of said computer system configuration, and of memorizing said characteristic in said fault example storage area related to said fault information;

a seventh step, by said fault monitoring apparatus, based on extracted features of said computer system configuration, of counting a frequency of fault occurrences for every feature of said computer system configuration when said fault occurs;

an eighth step, by said fault monitoring apparatus, of checking a frequency of fault occurrences for every feature of said computer system configuration, and of registering a rule for avoiding an extracted feature of a computer system configuration in said component characteristic information, when said frequency of fault occurrences is more than a predetermined number.

24. The computer recovery method for recovering automatically from a fault according to claim 22, wherein said third step in said fault monitoring apparatus, comprises:

a ninth step of retrieving said fault recovery rule, checking whether there is an unchecked fault type judging rule or not, and of finishing a process when all of fault type judging rules are checked;

a tenth step of checking whether said fault which occurs matches with a fault type judging rule or not when there exists an unchecked fault type judging rule;

an eleventh step of returning to said step ninth when a fault content does not match with a fault type judging rule, and of instructing a fault monitoring agent to execute contents of said operation specifying section corresponding to said fault judging rule matching with said fault content when there is a fault judging rule matching with said fault content; and

a twelfth step of checking whether an instructed operation is executed normally by said fault monitoring agent or not, of finishing a process when said instructed operation is executed normally, and of returning to said step ninth to repeat a same operation for said unchecked fault type judging rule when said instructed operation is not executed normally.

25. The computer recovery method for recovering automatically from a fault according to claim 24, wherein said eleventh step in said fault monitoring apparatus, comprises:

a thirteenth step of selecting an operation having a highest order of priority among operations described in said operation specifying section;

a fourteenth step of instructing said fault monitoring agent to execute a selected operation; and

a fifteenth step of checking whether said selected operation is executed normally or not, of reporting the normal execution to a higher process and of finishing a process when said selected operation is executed normally, of checking whether an operation having a second higher order priority is specified in said operation specifying section or not when said operation fails, of reporting that said operation described in said operation specifying section fails to said higher process and of finishing said process.

26. The computer recovery method for recovering automatically from a fault according to claim 25, wherein said fourteenth step in said fault monitoring apparatus, comprises:

a sixteenth step of checking whether a system to be an object of an operation is specified during operation defining or not;

a seventeenth step of making a specified system as an object system when said object system is specified;

an eighteenth step of making a system in which a fault occurs as said object system when no system is specified.

27. The computer recovery method for recovering automatically from a fault according to claim 26, wherein said fourteenth step in said fault monitoring apparatus, comprises:

a nineteenth step of checking whether an operation which will be executed is to change said at least one computer system or not, and of advancing to a twenty second step when said operation is not to change said at least one computer system;

a twentieth step of checking whether an object component of a configuration change operation is specified or not when said operation is to change said at least one computer system;

a twenty first step of referring to said component characteristic information to decide an object component of said configuration change operation when a component to be an object of said configuration change is not specified;

the twenty second step of instructing a specified operation to said fault monitoring agent of said at least one computer system to be recovered;

a twenty third step of checking whether an operation instructed in said at least one computer system is normally executed or not, of reporting to a higher process that said operation is successful, of checking whether an operation object component selected in said twenty first step remains or not when an instructed operation fails, and of returning to said twenty second step when remaining; and

a twenty fourth step of reporting that said operation has failed when said operation for all object components has failed.

28. A fault monitoring apparatus connected to at least one computer system and monitoring a fault in said at least one computer system, comprising:

a storage section for memorizing and holding fault recovery information including a rule which defines a recovery operation when a fault occurs in said at least one computer system,

wherein said fault monitoring apparatus, when a fault occurs in said at least one computer system, retrieves a rule previously set in said fault recovery information and instructs said at least one computer system to execute a recovery operation corresponding to said fault which occurs in said at least one computer system.

29. The fault monitoring apparatus according to claim 28, wherein said fault recovery information includes a condition for executing said recovery operation for said fault, said recovery operation, and a rule specified in a rule specifying language which is previously set.

30. The fault monitoring apparatus according to claim 28, wherein said fault recovery information includes component configuration information having information about each component making up said at least one computer system to be monitored by said fault monitoring apparatus, wherein when said at least one computer system becomes monitored as an object to be monitored by said fault monitoring apparatus, said fault monitoring apparatus instructs to a fault monitoring agent acting in said at least one computer system to notify said information concerning each component in at least one computer system and configuration information concerning connection of said each component, and wherein said fault monitoring apparatus automatically obtains said information concerning each component from said at least one computer system which is said object of monitoring and sets said information concerning each component in said component configuration information.

31. The fault monitoring apparatus according to claim 28, wherein said fault recovery information includes component characteristic information including information showing characteristics of components included in said at least one computer system monitored by said fault monitoring apparatus, and wherein said fault monitoring apparatus, when instructing a fault recovery operation to said at least one computer system in which said fault occurs, considers efficiencies concerning components included in said at least one computer system based on said component characteristic information, and instructs said at least one computer system in which said fault occurs so as to select components which are used efficiently.

32. The fault monitoring apparatus according to claim 28, wherein types of past faults which occurred and configuration information of said at least one computer system at a time at which a fault has occurred are registered in said storage section as fault example storage information, and wherein said fault monitoring apparatus, when instructing a fault recovery operation to said at least one computer system in which said fault has occurred, refers to fault information of past occurrences in said fault example storage information, and instructs said at least one computer system in which said fault occurs so as to avoid a computer system in which a fault is apt to occur.

33. The fault monitoring apparatus according to claim 28, wherein a system requisite rule which is a rule of a computer system to be met by a at least one computer system as an object to be monitored and information for defining an operation for meeting said rule are registered in said storage section as system configuration rule information, and wherein there is provided a controller that, when fault recovery of said at least one computer system in which a fault occurs, instructs said at least one computer system to change a computer system configuration after said fault recovery operation in accordance with a request specification of a whole system of said at least one computer system based on said system requisite rule.

34. The fault monitoring apparatus according to claim 28, further comprising:

a first processing section, when a fault occurs in said at least one computer system, for checking whether said fault which occurs is a component fault or not, and for, when said fault is said component fault, storing system configuration information at a time when said fault occurs in fault example storage information;

a second processing section for referring to said fault example storage information to refer to past fault examples, for checking whether or not there is a same fault example that has occurred this time, for, when there is the same fault example, comparing system configuration information in said past same fault example with a computer system configuration in which said fault has occurred this time, for extracting a feature of said computer system configuration, and for memorizing said characteristic related to said fault information as fault example storage information in said storage section;

a counting section for counting a frequency of fault occurrences for every feature of said at least one computer system when said fault occurs based on an extracted feature of computer system configuration; and

a third processing section for checking a frequency of fault occurrences for every feature of said computer system configuration, and for registering a rule for avoiding an extracted feature of a computer system configuration in said component characteristic information, when said frequency of fault occurrences is more than a predetermined number.

35. The fault monitoring apparatus according to claim 28, wherein said fault recovery information includes a fault type judging rule in which, when a fault occurs in said at least one computer system, a rule for judging at which position said fault occurs and what type of a fault, and an operation specifying section in which an operation is specified when said fault occurs,

whereby when a fault occurs in said at least one computer system, said fault information indicating a fault cause is notified from said fault monitoring agent to said fault monitoring apparatus, which refers to said fault recovery rule, retrieves a fault type judging rule corresponding to a condition of said fault which occurs, and instructs said fault monitoring agent of an operation of contents described in said operation specifying section corresponding to said fault type judging rule matching said fault.

36. A program being used in a fault monitoring apparatus connected to computer system, has fault recovery information including a rule for defining a recovery operation when a fault occurs in said computer system,

said program to cause a computer to execute a process wherein when a fault occurs in said computer system, of referring to said rule which is previously set in said fault recovery information, and of instructing said computer system, to perform a fault recovery operation corresponding to said fault which occurs and to execute a recovery operation corresponding to said rule.

37. The program according to claim 36, further causing said computer to execute:

a process, when a fault occurs in said computer system, of storing fault information notified from a fault monitoring agent in said computer system in which said fault occurs in a fault example storage area, and of extracting a feature of a computer system configuration for said fault information;

a process, when said fault occurs in said computer system, of referring to a fault recovery rule including a fault type judging rule for judging which position said fault occurs and what type of said fault and an operation specifying section in which an operation to be executed when a fault occurs, of retrieving a fault type judging rule corresponding to a condition, and of instructing a fault monitoring agent to execute an operation described in a corresponding operation specifying section; and

a process of referring to a system configuration rule including a system requisite rule which is a rule of a computer system configuration to met by a computer system to be monitored and an operation specifying section for defining an operation to satisfy said rule, of checking whether all of system requisite rules are met or not, and of instructing said fault monitoring agent to execute an operation described in said operation specifying section corresponding to said system requisite rule when there is a non-met system requisite rule.

38. A medium storing a program being used in a fault monitoring apparatus connected to computer system, has fault recovery information including a rule for defining a recovery operation when a fault occurs in said computer system,

said program to cause a computer to execute a process wherein when a fault occurs in said computer system, of referring to said rule which is previously set in said fault recovery information, and of instructing said computer system, to perform a fault recovery operation corresponding to said fault which occurs and to execute a recovery operation corresponding to said rule.