Containment and recovery of software exceptions in interacting, replicated-state-machine-based fault-tolerant components

• US 7,996,716 B2
• Filed: 06/12/2008
• Issued: 08/09/2011
• Est. Priority Date: 06/12/2008
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Butterworth, Henry
• Primary Examiner(s): GUYTON, PHILIP A
• Application Number:
  US12/137,813
• Publication Number:
  US 20090313500A1
• Time in Patent Office: 1,153 Days
• Field of Search: 714/16, 714/17, 714/41
• US Class Current: 714/17
• CPC Class Codes: G06F 11/1438 Restarting or rejuvenating
• CPC Class Codes: G06F 11/1479 Generic software techniques for error detection or fault masking
• CPC Class Codes: G06F 11/202 where processing functionality is redundant redundant communication control functionality G06F11/2005, redundant storage control functionality G06F11/2089

Abstract

A method, system and article of manufacture are disclosed for error recovery in a replicated state machine. A batch of inputs is input to the machine, and the machine uses a multitude of components for processing those inputs. Also, during this processing, one of said components generates an exception. The method comprises the steps of after the exception, rolling the state machine back to a defined point in the operation of the machine; preemptively failing said one of the components; re-executing the input batch in the state machine; and handling any failure, during the re-executing step, of the one of the components using a defined error handling procedure. The rolling, preemptively failing, re-executing and handling steps are repeated until the input batch runs to completion without generating any exception in any of the components that are not preemptively failed.

Claims

1. A method of error recovery in a replicated state machine, wherein, at a defined time in an operation of the machine, a batch of inputs are input to the machine, and the machine uses a multitude of components for processing said inputs, and wherein during said processing, one of said components generates an exception, the method comprising the steps of:

after the exception, rolling the state machine back to a defined point in the operation of the machine;

preemptively failing said one of the components;

re-executing the batch of inputs in the state machine;

handling any failure, during said re-executing step, of said one of the components using a defined error handling procedure, including using a second one of said components to handle said any failure in order to contain said exception within said one of the components; and

repeating the rolling, preemptively failing, re-executing and handling steps until the input batch runs to completion without generating any exception in any of the components that are not pre-emptively failed.

2. The method according to claim 1, wherein said defined point is at or prior to the defined time at which the batch inputs are input to the state machine.

3. The method according to claim 1, wherein the preemptively failing step includes the steps of:

assigning a respective one component ID to each of the components; and

maintaining a fail-set of components that are to be preemptively failed during input-batch processing.

4. The method according to claim 3, wherein said fail-set of components that are to be preemptively failed is empty at the start of the processing of the batch inputs.

5. The method according to claim 3, wherein the preemptively failing step includes the step of, during the processing of the batch inputs, if any component is created that is on the fail-set, preemptively failing said any component on creation thereof.

6. The method according to claim 3, wherein the preemptively failing step includes the step of, during the processing of the batch inputs, if an exception is generated by one of the components, adding the component ID of the component generating the exception to the fail-set.

7. The method according to claim 3, wherein:

the component ID for each component is sufficient to determine whether said each components was created during processing of the current input batch, or during processing of a previous input batch; and

if said each component is created during processing of the current input batch, the component ID is also sufficient to determine whether said each component was created before or after any of the other components created during the processing of the current input batch.

8. The method according to claim 1, wherein each of the plurality of components has a parent component, and the step of handling any failure of said one component includes the step of using the parent of said one component to handle said any failure.

9. The method according to claim 1, wherein the preemptively failing step includes the step of forcing said one of the components into a pending state before requests to said one of the components are failed back to others of the components.

10. The method according to claim 1, wherein the preemptively failing includes:

keeping a list of all of the components that are preemptively failed; and

adding to said list all of the components that generate an exception during said repeating step.

11. A method of error recovery in a replicated state machine, wherein, at a defined time in an operation of the machine, a batch of inputs are input to the machine, and the machine uses a multitude of components for processing said inputs, and wherein during said processing, one of said components generates an exception, the method comprising the steps of:

after the exception, rolling the state machine back to a defined point in the operation of the machine;

preemptively failing said one of the components'"'"'

re-executing the batch of inputs in the state machine;

handling any failure, during said re-executing step, of said one of the components using a defined error handling procedure; and

repeating the rolling, preemptively failing, re-executing and handling steps until the input batch runs to completion without generating any exception in any of the components that are not preemptively failed;

wherein the preemptively failing step includes the steps of;

assigning a respective one component ID to each of the components; and

maintaining a fail-set of components that are to be preemptively failed during input-batch processing; and

wherein each of the components receives a new ID when said each component is created, and as a result of a reset by a supervisor component after an exception.

12. An error recovery system in a replicated state machine, wherein, at a defined time in an operation of the machine, a batch of inputs are input to the machine, and the machine uses a multitude of components for processing said inputs, and wherein during said processing, one of said components generates an exception, the error recovery system comprising:

a computer system including one or more processor units configured for;

after the exception, rolling the state machine back to a defined point in the operation of the machine;

preemptively failing said one of the components;

re-executing the batch of inputs in the state machine;

handling any failure, during said re-executing step, of said one of the components using a defined error handling procedure, including using a second one of said components to handle said any failure in order to contain said exception within said one of the components; and

repeating the rolling, preemptively failing, re-executing and handling steps until the input batch runs to completion without generating any exception in any of the components that are not preemptively failed.

13. The error recovery system according to claim 12, wherein the preemptively failing is done by:

assigning a respective one component ID to each of the components; and

maintaining a fail-set of components that are to be preemptively failed during input-batch processing.

14. The error recovery system according to claim 13, wherein:

each of the components receives a new ID when said each component is created, and as a result of a reset by a supervisor component after an exception; and

said fail-set of components that are to be preemptively failed is empty at the start of the processing of the batch inputs.

15. The error recovery system according to claim 13, wherein the preemptively failing step includes the steps of, during the processing of the batch inputs:

if any component is created that is on the fail-set, preemptively failing said any component on creation thereof; and

if an exception is generated by one of the components, adding the component ID of the component generating the exception to the fail-set.

16. The error recovery system according to claim 13, wherein:

the component ID for each component is sufficient to determine whether said each components was created during processing of the current input batch, or during processing of a previous input batch; and

if said each component is created during processing of the current input batch, the component ID is also sufficient to determine whether said each component was created before or after any of the other components created during the processing of the current input batch.

17. An article of manufacture comprising:

at least one computer usable tangible medium having computer readable program code logic tangibly embodied therein to execute a machine instruction in a processing unit for error recovery in a replicated stat machine, wherein, at a defined time in an operation of the machine, a batch of inputs are input to the machine, and the machine uses a multitude of components for processing said inputs, and wherein during said processing, one of said components generates an exception, said computer readable program code logic, when executing, performing the following steps;

after the exception, rolling the state machine back to a defined point in the operation of the machine;

preemptively failing said one of the components;

re-executing the batch of inputs in the state machine;

handling any failure, during said re-executing step, of said one of the components using a defined error handling procedure, including using a second one of said components to handle said any failure in order to contain said exception within said one of the components; and

repeating the rolling, preemptively failing, re-executing and handling steps until the input batch runs to completion without generating any exception in any of the components that are not preemptively failed.

18. The article of manufacture according to claim 17, wherein the preemptively failing step includes the steps of:

assigning a respective one component ID to each of the components; and

maintaining a fail-set of components that are to be preemptively failed during input-batch processing.

19. The article of manufacture according to claim 18, wherein the preemptively failing step includes the step of, during the processing of the batch inputs:

if any component is created that is on the fail-set, presumptively failing said any component on creation thereof; and

if an exception is generated by one of the components, adding the component ID of the component generating the exception to the fail-set.

20. The article of manufacture according to claim 17, wherein said defined point is at or prior to the defined time at which the batch inputs are input to the state machine.