Failure recovery apparatus, failure recovery method, manager, and program

• US 7,441,152 B2
• Filed: 06/01/2005
• Issued: 10/21/2008
• Est. Priority Date: 06/02/2004
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Kato, Kiyoshi
• Primary Examiner(s): DUNCAN, MARC M
• Application Number:
  US11/141,398
• Publication Number:
  US 20050283638A1
• Time in Patent Office: 1,238 Days
• Field of Search: 714/26, 706/46, 706/47
• US Class Current: 714/26
• CPC Class Codes: G06F 11/0706 the processing taking place on a specific hardware platform or in a specific software environment
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40

Abstract

The failure knowledge storage 5 stores failure knowledge information describing countermeasure command selection information, of which a diversion to a failure other than failures described in a failure countermeasure rule in the rule storage 2 can be surmised. When the failure occurring in the service executor 11 matches the failure described in the failure countermeasure rule, the countermeasure retriever 3 executes the countermeasure command in the failure countermeasure rule on the service executor 11. It is decided whether or not a failure other than the failures described in the failure countermeasure rule matches a failure described in the failure knowledge information in the failure knowledge storage 5. When there is a match, the countermeasure command is read out of the rule storage 2 based on the selection information of the matched failure knowledge information and is executed on the service executor 11.

Claims

1. A failure recovery apparatus comprising:

a rule storage for storing a failure countermeasure rule which describes a failure type and a countermeasure command corresponding to said failure type; and

a failure knowledge storage for storing failure knowledge information, which describes a failure type and countermeasure command selection information;

wherein when a failure occurring in a service executor matches a failure described in said failure countermeasure rule in said rule storage, a countermeasure command in a matched failure countermeasure rule is read out of said rule storage and is executed on said service executor;

wherein when there is not a match, it is decided whether or not a failure occurring in said service executor matches a failure described in failure knowledge information in said failure knowledge storage;

wherein when there is a match, a countermeasure command is selected from said rule storage based on countermeasure command selection information corresponding to the matched failure knowledge information and is executed on said service executor.

2. The failure recovery apparatus defined in claim 1, further comprising:

means for creating a failure countermeasure rule and adding said failure countermeasure rule into said rule storage, said failure countermeasure rule describing a failure type described in said failure knowledge information and a countermeasure command selected from said rule storage based on said countermeasure command selection information;

wherein when a failure of the same type occurs again, said failure is dealt with according to said failure countermeasure rule stored in said rule storage.

3. A failure recovery apparatus comprising:

a rule storage for storing a failure countermeasure rule containing a conditional expression for decision of a failure type and a countermeasure command;

a failure knowledge storage for storing failure knowledge information containing a failure state and countermeasure command selection information;

a running status detector for detecting a running state of a service executor;

a failure decider for retrieving failure knowledge information from said failure knowledge storage, said failure knowledge information containing a failure state matching a running state detected by said running status detector;

a countermeasure retriever for retrieving, from said rule storage, a failure countermeasure rule containing a conditional expression matching a running state detected by said running status detector and outputting a countermeasure command in a failure countermeasure rule obtained through retrieval when said retrieval is successful or selectively outputting a countermeasure command from said failure countermeasure rule in said rule storage based on a countermeasure command selection information in said failure knowledge information when there is failure knowledge information retrieved by said failure decider in failure of said retrieval; and

a command executor for receiving a countermeasure command from said countermeasure retriever and executing said countermeasure command on said service executor.

4. The failure recovery apparatus defined in claim 3, wherein countermeasure command selection information in said failure knowledge information contains a parameter related to a failure state in said failure knowledge information of parameters appearing in a conditional expression in a failure countermeasure rule stored in said rule storage;

and

wherein said countermeasure retriever selects a countermeasure command in a failure countermeasure rule, in which a conditional expression contains a parameter described in countermeasure command selection information.

5. The failure recovery apparatus defined in claim 3, further comprising an effect decider for deciding a success or failure of recovery based on said executed countermeasure command and changing said priority information according to the decision result;

wherein said rule storage stores priority information on an application order of failure countermeasure rules; and

wherein said countermeasure retriever sequentially selectively outputs said countermeasure commands based on said priority information until said failure recovers when plural countermeasure commands selected by said countermeasure command selection information exist.

6. The failure recovery apparatus defined in claim 3, wherein said countermeasure retriever creates a failure countermeasure rule and adding said failure countermeasure rule to said rule storage, said failure countermeasure rule describing a failure state described in said failure knowledge information and a countermeasure command selected from said rule storage based on said countermeasure command selection information;

wherein when the same failure state occurs again, said failure is dealt with according to said failure countermeasure rule stored in said rule storage.

7. The failure recovery apparatus defined in claim 6, further comprising an interactive controller for outputting a failure countermeasure rule created by said countermeasure retriever and an effect decision result thereof, to a user input/output device, according to said failure knowledge information, and correcting a failure countermeasure rule stored in said rule storage and failure knowledge information stored in said failure knowledge storage, according to an input from said user input/output device.

8. The failure recovery apparatus defined in claim 3, further comprising a peculiar status detector for detecting a peculiar state of said service executor based on a history of a running state detected by said running status detector and outputting peculiar state information;

wherein said failure decider receives said peculiar state information and creates failure knowledge information, including a failure state and countermeasure command selection information, and stores said failure knowledge information into said failure knowledge storage, said peculiar state including a failure state, said failure state being said peculiar state, said countermeasure command selection information being a parameter related to said peculiar state among parameters contained in a conditional expression in a failure countermeasure rule stored in said rule storage.

9. The failure recovery apparatus defined in claim 8, further comprising an interactive controller for outputting a failure countermeasure rule to a user input/output device, said failure countermeasure rule including failure knowledge information created by said failure decider according to said peculiar state information and a countermeasure command selected from said rule storage by said countermeasure retriever according to said failure knowledge information, and correcting a failure countermeasure rule stored in said rule storage and failure knowledge information stored in said failure knowledge storage according to an input from said user input/output device.

10. The failure recovery apparatus defined in claim 3, further comprising an interactive controller, disposed between said countermeasure retriever and said command executor, for inquiring from a user go or no-go of execution of a countermeasure command selectively output by said countermeasure retriever through a user input/output device, based on countermeasure command selection information of said failure knowledge information.

11. A failure recovery method comprising the steps of:

using a rule storage for storing a failure countermeasure rule describing a type of failure and a countermeasure command against the type of failure and a failure knowledge storage for storing failure knowledge information describing a type of failure and countermeasure command selection information;

when a failure occurring in a service executor matches a failure described in a failure countermeasure rule stored in said rule storage, reading a countermeasure command corresponding to a matched failure countermeasure rule from said rule storage and executing said command on said service executor;

when there is no match, deciding whether or not a failure occurring in said service executor matches a failure described to said failure knowledge information in said failure knowledge storage; and

when there is a match, selecting a countermeasure command from said rule storage based on countermeasure command selection information of said matched failure knowledge information and executing said countermeasure command on said service executor.

12. The failure recovery method defined in claim 11, wherein said countermeasure retriever creates a failure countermeasure rule, said failure countermeasure rule describing a type of failure described in said failure knowledge information and a countermeasure command selected from said rule storage based on said countermeasure command selection information and adds said countermeasure command rule to said rule storage;

whereby when a failure of the same type occurs again, countermeasure is performed according to said failure countermeasure rule stored in said rule storage.

13. A failure recovery method comprising the steps of:

a) detecting a running state of a service executor, using a running status detector;

b) retrieving failure knowledge information including a failure state matching said detected running state from a failure knowledge storage, which stores failure knowledge information including a failure state and countermeasure command selection information, using a failure decider;

c) when said detected running state matches a conditional expression described in any failure countermeasure rule in a rule storage storing failure countermeasure rules including a failure decision conditional expression and a countermeasure command, outputting a countermeasure command of a matched failure countermeasure rule, using said countermeasure retriever, or when there is no match and there is failure knowledge information retrieved in the step (b), outputting a countermeasure command selected from a failure countermeasure rule in said rule storage based on countermeasure command selection information in the failure knowledge information; and

d) receiving said output countermeasure command using a command executor and then executing said countermeasure command on said service executor.

14. The failure recovery method defined in claim 13, wherein countermeasure command selection information in said failure knowledge information includes a parameter related to a failure state in said failure knowledge information among parameters appearing in a conditional expression in a failure countermeasure rule stored in said rule storage and wherein a countermeasure command in a failure countermeasure rule including a conditional expression having parameters described in countermeasure command selection information is selected in the step (c).

15. The failure recovery method defined in claim 13, further comprising the step of:

e) deciding a success or failure of recovery by said executed countermeasure command, using an effect decider, and changing said priority information according to the decision result;

wherein said rule storage stores priority information on an application order of failure countermeasures rules;

wherein said countermeasures commands outputs sequentially and selectively based on said priority information until recovery of said failure when plural countermeasure commands selectable with said countermeasure command selection information exists in said step (c).

16. The failure recovery method defined in claim 13, further comprising the step of:

f) creating a failure countermeasure rule using said countermeasure retriever, said failure countermeasure rule describing a failure state described in said failure knowledge information and a countermeasure command selected from said rule storage based on said countermeasure command selection information; and

adding said failure countermeasure rule to said rule storage;

whereby when the same failure state occurs again, said failure is dealt with according to said failure countermeasure rule stored in said rule storage.

17. The failure recovery method defined in claim 16, comprising the steps of, by means of an interactive controller,

outputting a failure countermeasure rule created by said countermeasure retriever according to said failure knowledge information and the effect decision result to a user input/output device, using an interactive controller;

and

correcting a failure countermeasure rule stored in said rule storage and failure knowledge information stored in said failure knowledge storage, according to an input from said user input/output device.

18. The failure recovery method defined claim 13, further comprising the steps of:

g) detecting a peculiar state of said service executor based on a history of said detected running state, by means of a peculiar status detector, and then outputting peculiar state information; and

h) receiving said peculiar state information using said failure decider and creating failure knowledge information, said failure knowledge information including a failure state being said peculiar state and countermeasure command selection information being a parameter related to said peculiar state among parameters appearing in a conditional expression in a failure countermeasure rule stored in said rule storage, and then storing said failure knowledge information into said failure knowledge storage.

19. The failure recovery method defined in claim 18, further comprising the steps of, by means of an interactive controller,:

outputting a failure countermeasure rule to a user input/output device, said failure countermeasure rule including failure knowledge information created by said failure decider based on said peculiar state information and a countermeasure command selected from said rule storage by said countermeasure retriever based on said failure knowledge information; and

correcting a failure countermeasure rule stored in said rule storage and failure knowledge information stored in said failure knowledge storage, according to an input from said user input/output device.

20. The failure recovery method defined in claim 13, further comprising the step of:

inquiring, of a user through a user input/output device, whether or not an interactive controller disposed between said countermeasure retriever and said command executor can execute a countermeasure command selectively output by means of said countermeasure retriever based on countermeasure command selection information in said failure knowledge information.

21. A manager comprising:

a rule storage for storing a failure countermeasure rule, which includes a conditional expression for failure decision and a countermeasure command;

a failure knowledge storage for storing failure knowledge information, which includes a failure state and countermeasure command selection information;

a running status detector for detecting a running state of a service executor;

a failure decider for retrieving failure knowledge information from said failure knowledge storage, said failure knowledge information including a failure state matching a running state detected by said running status detector;

a countermeasure retriever for retrieving a failure countermeasure rule including a conditional expression matching a running state detected by said running status detector, from said rule storage, and outputting, when said retrieval is successful, a countermeasure command in a failure countermeasure rule obtained through retrieval, or selectively outputting, when said retrieval is not successful, a countermeasure command from a failure countermeasure rule in said rule storage based on countermeasure command selection information in the failure knowledge information, provided that there is failure knowledge information retrieved by said failure decider; and

a command executor for receiving a countermeasure command from said countermeasure retriever and executing said countermeasure command on said service executor.

22. The manager defined in claim 21, wherein countermeasure command selection information in said failure knowledge information includes a parameter related to a failure state in said failure knowledge information among parameters appearing in a conditional expression in a failure countermeasure rule stored in said rule storage;

and wherein said countermeasure retriever selects a countermeasure command in a failure countermeasure rule including a parameter described in countermeasure command selection information in a conditional expression.

23. The manager defined claim 21 or 22, further comprising:

an effect decider for deciding a success or failure of recovery by said executed countermeasure command and changing said priority information according to the decision result;

wherein said rule storage stores priority information on an application order of failure countermeasure rules; and

wherein said countermeasure retriever outputs said countermeasure command selectively and sequentially based on said priority information until said failure recovers when plurality countermeasure commands to be selected with said countermeasure command selection information exist.

24. The manager defined in claim 21, wherein said countermeasure retriever creates a failure countermeasure rule, which describes a failure state described in said failure knowledge information and a countermeasure command selected from said rule storage based on said countermeasure command selection information, and adds said failure countermeasure rule to said rule storage, and deals with said failure according to said failure countermeasure rule stored in said rule storage when the same failure state occurs again.

25. The manager defined in claim 24, further comprising an interactive controller for outputting a failure countermeasure rule created by said countermeasure retriever according to said failure knowledge information and the effect decision result thereof to a user input/output device and correcting a failure countermeasure rule stored in said rule storage according to an input from said user input/output device and failure knowledge information stored in said failure knowledge storage.

26. The manager defined in claim 21, further comprising:

a peculiar status detector for detecting a peculiar state of said service executor from a running state history detected by said running status detector and then outputting peculiar state information;

wherein said failure decider receives said peculiar state information and creates failure knowledge information, said failure knowledge information including said peculiar state being a failure state and a parameter related to said peculiar state among parameters included in a conditional expression in a failure countermeasure rule stored in said rule storage, being countermeasure command selection information, and stores said failure knowledge information into said failure knowledge storage.

27. The manager defined in claim 26, further comprising an interactive controller for outputting a failure countermeasure rule to a user input/output device, said failure countermeasure rule including failure knowledge information created by said failure decider based on said peculiar state information and a countermeasure command selected from said rule storage by said countermeasure retriever based on said failure knowledge information and correcting a failure countermeasure rule stored in said rule storage and failure knowledge information stored in said failure knowledge storage according to an input from said user input/output device.

28. The manager defined in claim 21, further comprising:

an interactive controller disposed between said countermeasure retriever and said command executor, for inquiring of a user through a user input/output device the go or no-go of a countermeasure command selectively output from said countermeasure retriever based on countermeasure command selection information in said failure knowledge information.

29. A computer readable medium containing a computer program, which when executed, controls a computer as a manager, which includes a failure decider and a countermeasure retriever;

said failure decider for retrieving failure knowledge information from a failure knowledge storage, said failure knowledge information including a failure state matching a running state detected by said running status detector which detects a running state of a service executor, said failure knowledge storage storing failure knowledge information including a failure state and countermeasure command selection information; and

wherein when a running state detected by said running status detector matches a conditional expression described in any failure countermeasure rule in a rule storage storing a failure countermeasure rule including a conditional expression for failure decision and a countermeasure command, said countermeasure retriever outputs a countermeasure command in a failure countermeasure rule to a command executor executing a command on said service executor;

wherein when there is no match, said countermeasure retriever outputs a countermeasure command selected from a failure countermeasure rule in said rule storage to said command executor based on countermeasure command selection information in the failure knowledge information, provided that there is failure knowledge information retrieved by said failure decider.