Fault recovery system and method for adaptively updating order of command executions acoording to past results

• US 20050015665A1
• Filed: 07/16/2004
• Published: 01/20/2005
• Est. Priority Date: 07/16/2003
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Kato, Kiyoshi, Hiraike, Ryuichi
• Granted Patent: US 7,620,849 B2
• Time in Patent Office: Days
• US Class Current: 714/15
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0781 Error filtering or prioritizing based on a policy defined by the user or on a policy defined by a hardware/software module, e.g. according to a severity level
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40

Abstract

In a memory a number of entries are defined for mapping reference symptom levels of a server to fault restoration commands and to priority levels. In response to a status report from a fault detector indicating an operating state of the server, one of the commands is selected according to the priority levels corresponding to the reported state. The selected command is executed, and a result of the execution is estimated. In response to a subsequent status report, a comparison is made between the estimated result and an operating state indicated in the report. The priority levels are updated according to the comparison result. In a modification, status variables are mapped to the commands. A command is selected according to the status variables of entries to which a reported state corresponds. A success value is determined based on a result of execution of the command. The status variable of the selected command is updated with the determined success value.

Claims

1. A fault recovery system comprising:

fault detecting means for transmitting a status report indicating an operating condition of a data processing apparatus;

command execution means for executing a command in said data processing apparatus; and

decision means, responsive to a status report from the fault detecting means, for selecting one of a plurality of fault restoration commands according to an operating condition indicated in the status report and according to a command selection order, and transmitting the selected command to said command execution means,

said decision means being further responsive to a subsequent status report from said fault detecting means, for evaluating the result of execution of said transmitted command according to an operating condition indicated in the subsequent status report, and updating said command selection order according to the evaluated result.

2. A fault recovery system comprising:

fault detecting means for transmitting a status report indicating an operating condition of a data processing apparatus;

command execution means for executing a command in said data processing apparatus;

a store for defining a plurality of entries and mapping in each of said entries one of a plurality of reference fault conditions of said data processing apparatus to one of a plurality of fault restoration commands and one of a plurality of priority levels; and

decision means, responsive to a status report from said fault detecting means, for detecting one of said reference fault conditions which corresponds to a fault condition indicated in the status report, selecting one of said fault restoration commands according to the priority levels of the entries which correspond to the detected reference fault condition, transmitting the selected command to said command execution means, and estimating a result of the execution of the transmitted command;

said decision means being further responsive to a subsequent status report from said fault detecting means for making a comparison between said estimated result and a fault condition indicated in the subsequent status report and updating said priority levels according to a result of the comparison

3. The fault recovery system of claim 2, wherein said decision means is configured to select one of said fault restoration commands from an entry whose priority level is lower than the priority level of the entry from which a fault restoration command was previously selected, if the fault condition indicated in said subsequent status report still corresponds to said one reference fault condition.

4. The fault recovery system of claim 2, wherein said decision means is responsive to multiple status reports from said fault detecting means for detecting a plurality of said reference fault conditions which correspond to fault conditions indicated in said multiple status reports, selecting one of said fault restoration commands which is commonly shared by a plurality of entries of said detected reference fault conditions, and transmitting the selected command to said command execution means.

5. The fault recovery system of claim 3, wherein said decision means is responsive to multiple status reports from said fault detecting means for detecting a plurality of said reference fault conditions which correspond to fault conditions indicated in said multiple status reports, selecting one of said fault restoration commands which is commonly shared by a plurality of entries of said detected reference fault conditions, and transmitting the selected command to said command execution means.

6. A fault recovery system comprising.

fault detecting means for transmitting a status report indicating a value of an operating condition of a data processing apparatus;

command execution means for executing a command in said data processing apparatus;

a store for defining a plurality of entries and mapping in each of said entries one of a plurality of reference values of operating conditions of said data processing apparatus to one of a plurality of fault restoration commands and one of a plurality of status variables; and

decision means, responsive to a first status report from said fault detecting means, for detecting one of said reference values which corresponds to a value indicated in the status report, making an estimation of success or failure of command execution by calculating a sum of a status variable mapped to the detected reference value and the value of the first status report and comparing said sum with said detected reference value to produce an estimated value, selecting a fault restoration command from the entries in which the detected reference value is contained according to said estimated value, and transmitting the selected command to said command execution means,

said decision means being further responsive to a second, subsequent status report from said fault detecting means for subtracting the value of the first status report from a value indicated in said second stats report to produce a success value, and updating said calculated status variable with the success value.

7. The fault recovery system of claim 6, wherein said decision means is configured to determine whether there is a match or a mismatch between said sum and said detected reference value and select a fault restoration command from a first one of the entries in which the detected reference value is contained if said mismatch is detected or select a fault restoration command from a second one of said entries in which said detected reference value is contained if said match is detected.

8. The fault recovery system of claim 6, wherein said decision means is configured to perform said estimation again after said status variable is updated if there is a match between the value of said second status report and the detected reference value.

9. The fault recovery system of claim 7, wherein said decision means is configured to perform said estimation again after said status variable is updated if there is a match between the value of said second status report and the detected reference value.

10. A fault recovery system comprising:

fault detecting means for transmitting a status report indicating a value of an operating condition of a data processing apparatus;

command execution means for executing a command in said data processing apparatus;

a store for defining a plurality of entries and mapping in each of said entries one of a plurality of reference values of operating conditions of said data processing apparatus to one of a plurality of fault restoration commands, one of a plurality of priority levels, and one of a plurality of status variables; and

decision means, responsive to a first status report from said fault detecting means, for detecting one of said reference values which corresponds to a value indicated in the status report, making an estimation of success or failure of command execution by calculating a sum of a status variable mapped to the detected reference value and the value of the first status report and comparing said sum with said detected reference value to produce an estimated value, selecting a fault restoration command from said entries in which the detected reference value is contained according to said estimated value and the priority levels of said entries, and transmitting the selected command to said command execution means,

said decision means being further responsive to a second, subsequent status report from said fault detecting means for subtracting the value of the first status report from a value indicated in said second status report to produce a success value, and updating said calculated status variable with the success value.

11. The fault recovery system of claim 10, wherein said decision means is configured to determine whether there is a match or a mismatch between said sum and said detected reference value and select a fault restoration command from a first one of the entries in which the detected reference value is contained if said mismatch is detected or select a fault restoration command from a second one of said entries in which said detected reference value is contained if said match is detected.

12. The fault recovery system of claim 10, wherein said decision means is configured to perform said estimation again after said status variable is updated if there is a match between the value of said second status report and the detected reference value.

13. The fault recovery system of claim 11, wherein said decision means is configured to perform said estimation again after said status variable is updated if there is a match between the value of said second status report and the detected reference value.

14. A fault recovery method for a data processing apparatus to which fault detecting means is connected for transmitting a status report indicating an operating condition of the data processing apparatus and to which a command execution means is further connected for executing a command in said data processing apparatus, the method comprising the steps of:

a) responsive to a status report from the fault detecting means, selecting one of a plurality of fault restoration commands according to an operating condition indicated in the status report and according to a command selection order;

b) transmitting the selected command to said command execution means; and

c) responsive to a subsequent status report from said fault detecting means, evaluating the result of execution of said transmitted command according to an operating condition indicated in the subsequent status report, and updating said command selection order according to the evaluated result.

15. A fault recovery method for a data processing apparatus to which fault detecting means is connected for transmitting a status report indicating an operating condition of the data processing apparatus and to which a command execution means is further connected for executing a command in said data processing apparatus, the method comprising the steps of:

a) defining a plurality of entries and mapping in each of said entries one of a plurality of reference fault conditions of said data processing apparatus to one of a plurality of fault restoration commands and one of a plurality of priority levels;

b) responsive to a status report from said fault detecting means, detecting one of said reference fault conditions which corresponds to a fault condition indicated in the status report;

c) selecting one of said fault restoration commands according to the priority levels of the entries which correspond to the detected reference fault condition and transmitting the selected command to said command execution means;

d) estimating a result of the execution of the transmitted command; and

e) responsive to a subsequent status report from said fault detecting means for making a comparison between said estimated result and a fault condition indicated in the subsequent status report and updating said priority levels according to a result of the comparison.

16. The fault recovery method of claim 15, further comprising the step of selecting one of said fault restoration commands from an entry whose priority level is lower than the priority level of the entry from which a fault restoration command was previously selected, if the fault condition indicated in said subsequent status report still corresponds to said one reference fault condition.

17. The fault recovery method of claim 15, wherein step (b) comprises the step of:

receiving multiple status reports from said fault detecting means and detecting a plurality of said reference fault conditions which correspond to fault conditions indicated in said multiple status reports, and

wherein step (c) comprises the steps of;

selecting one of said fault restoration commands which is commonly shared by a plurality of entries of said detected reference fault conditions; and

transmitting the selected command to said command execution means.

18. A fault recovery method for a data processing apparatus to which fault detecting means is connected for transmitting a status report indicating an operating condition of the data processing apparatus and to which a command execution means is further connected for executing a command in said data processing apparatus, the method comprising the steps of:

a) defining a plurality of entries and mapping in each of said entries one of a plurality of reference values of operating conditions of said data processing apparatus to one of a plurality of fault restoration commands and one of a plurality of status variables;

b) responsive to a first status report from said fault detecting means, detecting one of said reference values which corresponds to a value indicated in the status report;

c) making an estimation of success or failure of command execution by calculating a sum of a status variable mapped to the detected reference value and the value of the first status report and comparing said sum with said detected reference value to produce an estimated value;

d) selecting a fault restoration command from the entries in which the detected reference value is contained according to said estimated value, and transmitting the selected command to said command execution means; and

e) responsive to a second, subsequent status report from said fault detecting means for subtracting the value of the first status report from a value indicated in said second status report to produce a success value and updating said calculated status variable with the success value.

19. The fault recovery method of claim 18, further comprising the steps of:

determining whether there is a match or a mismatch between said sum and said detected reference value; and

selecting a fault restoration command from a first one of the entries in which the detected reference value is contained if said mismatch is detected or selecting a fault restoration command from a second one of said entries in which said detected reference value is contained if said match is detected.

20. The fault recovery method of claim 18, further comprising the steps of, after step (e) is performed, making a second estimation of success or failure of command execution by calculating a sum of a status variable which is mapped to the detected reference value and the value of the first status report and comparing said sum with said detected reference value to produce an estimated value, if there is a match between the value of said second status report and the detected reference value.

21. A fault recovery method for a data processing apparatus to which fault detecting means is connected for transmitting a status report indicating an operating condition of the data processing apparatus and to which a command execution means is further connected for executing a command in said data processing apparatus, the method comprising the steps of:

a) defining a plurality of entries and mapping in each of said entries one of a plurality of reference values of operating conditions of said data processing apparatus to one of a plurality of fault restoration commands, one of a plurality of priority levels, and one of a plurality of status variables;

b) responsive to a first status report from said fault detecting means, detecting one of said reference values which corresponds to a value indicated in the status report;

c) making an estimation of success or failure of command execution by calculating a sum of a status variable mapped to the detected reference value and the value of the first status report and comparing said sum with said detected reference value to produce an estimated value;

d) selecting a fault restoration command from said entries in which the detected reference value is contained according to said estimated value and the priority levels of said entries and transmitting the selected command to said command execution means; and

e) responsive to a second, subsequent status report from said fault detecting means, subtracting the value of the first status report from a value indicated in said second status report to produce a success value, and updating said calculated status variable with the success value.

22. The fault recovery method of claim 21, further comprising the steps of:

determining whether there is a match or a mismatch between said sum and said detected reference value; and

selecting a fault restoration command from a first one of the entries in which the detected reference value is contained if said mismatch is detected or selecting a fault restoration command from a second one of said entries in which said detected reference value is contained if said match is detected.

23. The fault recovery method of claim 21, further comprising the steps of, after step (e) is performed, making a second estimation of success or failure of command execution by calculating a sum of a status variable which is mapped to the detected reference value and the value of the first status report and comparing said sum with said detected reference value to produce an estimated value, if there is a match between the value of said second status report and the detected reference value.

24. A storage medium containing a computer-executable program for recovering a data processing apparatus from failure, wherein fault detecting means is connected to the data processing apparatus for transmitting a status report indicating an operating condition of the data processing apparatus and wherein command execution means is further connected for executing a command in said data processing apparatus, the program comprising the steps of:

a) responsive to a status report from the fault detecting means, selecting one of a plurality of fault restoration commands according to an operating condition indicated in the status report and according to a command selection order;

b) transmitting the selected command to said command execution means; and

c) responsive to a subsequent status report from said fault detecting means, evaluating the result of execution of said transmitted command according to an operating condition indicated in the subsequent status report, and updating said command selection order according to the evaluated result.

25. A storage medium containing a computer-executable program for recovering a data processing apparatus from failure, wherein fault detecting means is connected to the data processing apparatus for transmitting a status report indicating an operating condition of the data processing apparatus and wherein command execution means is further connected for executing a command in said data processing apparatus, the program comprising the steps of:

a) defining a plurality of entries and mapping in each of said entries one of a plurality of reference fault conditions of said data processing apparatus to one of a plurality of fault restoration commands and one of a plurality of priority levels;

b) responsive to a status report from said fault detecting means, detecting one of said reference fault conditions which corresponds to a fault condition indicated in the status report;

c) selecting one of said fault restoration commands according to the priority levels of the entries which correspond to the detected reference fault condition and transmitting the selected command to said command execution means;

d) estimating a result of the execution of the transmitted command; and

e) responsive to a subsequent status report from said fault detecting means for making a comparison between said estimated result and a fault condition indicated in the subsequent status report and updating said priority levels according to a result of the comparison.

26. The storage medium of claim 25, further comprising the step of selecting one of said fault restoration commands from an entry whose priority level is lower than the priority level of the entry from which a fault restoration command was previously selected, if the fault condition indicated in said subsequent status report still corresponds to said one reference fault condition.

27. The storage medium of claim 25, wherein step (b) comprises the step of:

receiving multiple status reports from said fault detecting means and detecting a plurality of said reference fault conditions which correspond to fault conditions indicated in said multiple status reports, and

wherein step (c) comprises the steps of;

selecting one of said fault restoration commands which is commonly shared by a plurality of entries of said detected reference fault conditions; and

transmitting the selected command to said command execution means.

28. A storage medium containing a computer-executable program for recovering a data processing apparatus from failure, wherein fault detecting means is connected to the data processing apparatus for transmitting a status report indicating an operating condition of the data processing apparatus and wherein command execution means is further connected for executing a command in said data processing apparatus, the program comprising the steps of:

a) defining a plurality of entries and mapping in each of said entries one of a plurality of reference values of operating conditions of said data processing apparatus to one of a plurality of fault restoration commands and one of a plurality of status variables;

b) responsive to a first status report from said fault detecting means, detecting one of said reference values which corresponds to a value indicated in the status report;

c) making an estimation of success or failure of command execution by calculating a sum of a status variable mapped to the detected reference value and the value of the first status report and comparing said sum with said detected reference value to produce an estimated value;

d) selecting a fault restoration command from the entries in which the detected reference value is contained according to said estimated value, and transmitting the selected command to said command execution means; and

e) responsive to a second, subsequent status report from said fault detecting means for subtracting the value of the first status report from a value indicated in said second status report to produce a success value and updating said calculated status variable with the success value.

29. The storage medium of claim 28, further comprising the steps of:

determining whether there is a match or a mismatch between said sum and said detected reference value; and

selecting a fault restoration command from a first one of the entries in which the detected reference value is contained if said mismatch is detected or selecting a fault restoration command from a second one of said entries in which said detected reference value is contained if said match is detected.

30. The storage medium of claim 28, further comprising the steps of, after step (e) is performed, making a second estimation of success or failure of command execution by calculating a sum of a status variable which is mapped to the detected reference value and the value of the first status report and comparing said sum with said detected reference value to produce an estimated value, if there is a match between the value of said second status report and the detected reference value.

31. A storage medium containing a computer-executable program for recovering a data processing apparatus from failure, wherein fault detecting means is connected to the data processing apparatus for transmitting a status report indicating an operating condition of the data processing apparatus and wherein command execution means is further connected for executing a command in said data processing apparatus, the program comprising the steps of:

a) defining a plurality of entries and mapping in each of said entries one of a plurality of reference values of operating conditions of said data processing apparatus to one of a plurality of fault restoration commands, one of a plurality of priority levels, and one of a plurality of status variables;

b) responsive to a first status report from said fault detecting means, detecting one of said reference values which corresponds to a value indicated in the status report;

c) making an estimation of success or failure of command execution by calculating a sum of a status variable mapped to the detected reference value and the value of the first status report and comparing said sum with said detected reference value to produce an estimated value;

d) selecting a fault restoration command from said entries in which the detected reference value is contained according to said estimated value and the priority levels of said entries and transmitting the selected command to said command execution means; and

e) responsive to a second, subsequent status report from said fault detecting means, subtracting the value of the first status report from a value indicated in said second status report to produce a success value, and updating said calculated status variable with the success value.

32. The storage medium of claim 31, further comprising the steps of:

determining whether there is a match or a mismatch between said sum and said detected reference value; and

selecting a fault restoration command from a first one of the entries in which the detected reference value is contained if said mismatch is detected or selecting a fault restoration command from a second one of said entries in which said detected reference value is contained if said match is detected.

33. The storage medium of claim 31, further comprising the steps of, after step (e) is performed, making a second estimation of success or failure of command execution by calculating a sum of a status variable which is mapped to the detected reference value and the value of the first status report and comparing said sum with said detected reference value to produce an estimated value, if there is a match between the value of said second status report and the detected reference value.