Nested recovery scope management for stateless recovery agents

• US 20040220973A1
• Filed: 04/30/2003
• Published: 11/04/2004
• Est. Priority Date: 04/30/2003
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Booz, David A., Sykes, Matthew J., Kinder, Stephen J.
• Granted Patent: US 7,107,293 B2
• Time in Patent Office: Days
• US Class Current: 707/200
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 11/1474 in transactions G06F16/20 takes precedence
• CPC Class Codes: G06F 11/1479 Generic software techniques for error detection or fault masking
• CPC Class Codes: Y10S 707/99952 Coherency, e.g. same view to multiple users
• CPC Class Codes: Y10S 707/99953 Recoverability

Abstract

Nested recovery scope management systems and methods for a multiple process computer system having a dynamic multiple address space server are disclosed. Stateless recovery agents are employed, under the control of a supervisory program called Recovery Director, during initialization or restart of servers to restore recoverable data in response to identified failures or other abnormal termination. The Director controls the recovery of protected resources in a systematic manner. The Director is initialized when a first address space of a first server is started. Then, as each instance of a recoverable component is initialized, the component registers with the Director by providing a reference to a stateless recovery agent that can later perform recovery functions for it if needed. As part of the registration, a token representing the current failure scope of the registration is generated and provided to there coverable component by the Director. Work identifiers representing recoverable operations can be generated from or associated with this token by the recoverable component and may be put in hardened storage. Upon initialization of a new server region, and also upon a failure or abnormal termination, the Director groups multiple instances of the same recoverable component together, formulates an iterative recovery plan, and determines the scope of recovery by examining its hardened data for failure scopes involving a recoverable component that may need to perform some recovery operation. As such recoverable components are identified, the Director references the associated recovery agent, provides it with appropriate token as to the recovery scope. The recovery agent is then allowed to carry out those recoveries, including any recovery scopes nested within.

Claims

1. A nested recovery scope management system for use in a computer system having a dynamic multiple address space server environment, the system comprising:

a supervisory program for directing recovery of protected resources, the supervisory program including a first subcomponent for initializing a recovery log, and a second subcomponent for storing failure scope tokens and recovery agent references in a recovery log, a third subcomponent for examining a recovery log for agents registered therewith and any current failure scope, and a fourth component for selectively notifying a recovery agent to carry out a recovery procedure;

a recovery log used by the supervisory program for storing information about recovery agents registered within a recovery scope;

at least a first recoverable component having therein a first subcomponent for generating work identifiers containing a failure scope token or associated with such a token that represents recoverable operations that may need to be performed at a later time, and a second subcomponent for registering a recovery agent with the supervisory program; and

at least a first stateless recovery agent identified in response to the initialization of a first recoverable component, the first agent being operable to assist in performing recovery operations in connection with the first recoverable component when instructed to do so by the supervisory program, the first recovery agent also being operable to utilize work identifiers from the recovery log in order to perform recovery operations in connection with at least the first associated component.

2. A management system as in claim 1, wherein:

the supervisory program includes a fifth component for generating tokens used to reference stored work identifiers associated with recoverable components, and

the recovery agent is also operable to utilize a token received from the supervisory program as well as the stored work identifiers in the recovery log in order to perform recoverable operations in connection with at least the first associated component.

3. A management system as in claim 2, wherein the recovery log includes hardened work identifiers and token information, whereby data placed therein will survive a loss of power to the computer system and be intact upon a restart.

4. A management system as in claim 1, wherein the supervisory program includes a fifth subcomponent which is operable to determine whether a given recoverable component has previously registered within the current failure scope, and is also operable to record in the recovery log any new work identifiers associated with a current failure scope for a checked component.

5. A management system as in claim 1, wherein the supervisory program includes a sixth subcomponent which is operable to check to determine whether a failure scope recovery by each recovery agent dispatched by the supervisory program is successful, and is also operable to delete from the recovery log information relating to a recovery scope when its recovery is reported as having been successfully carried out.

6. A management system as in claim 1 wherein the stateless recovery agent is operable to retry recovery operations more than once before reporting that recovery has failed.

7. A management system as in claim 2 wherein the supervisory program includes a sixth subcomponent which is operable to direct the removal work identifiers and token information associated with the first recoverable component from the recovery log when recovery operations associated with the first recoverable component have been successfully completed.

8. A management system as in claim 2 wherein the supervisory program includes a fifth subcomponent to filter duplicate recovery agent references found in recovery scopes stored within the recovery log.

9. A management system as in claim 8 wherein the supervisory program includes a sixth subcomponent which is operable to build a recovery agent iterator that utilizes at least a first ordered list of agents to be dispatched to perform recovery for an identified failure scope that may be in need of recovery successively, upon the recovery agent reporting back from an earlier recovery assignment.

10. A management system as in claim 9, wherein the supervisory program includes a seventh subcomponent which is operable to direct the removal of work identifiers and token information associated with the first recoverable component associated there with from the recovery log when recovery operations associated with the first recoverable component have been successfully completed.

11. A recovery scope management method for use in a multiple process computer system having a plurality of server regions assigned to a shared resource group which regions include a dynamic multiple address space, the method being for assisting in the recovery and restoration of protected resources available to the shared resource group, the method comprising:

(a) installing a supervisory program for directing the recovery of protected resources whose processing was abnormally terminated, and initializing the supervisory program when a first address space of a server region within the computer system is started;

(b) as each instance of a recoverable component is initialized, registering that component with the supervisory program by providing the supervisory program a reference to a stateless recovery agent that is able to perform recovery functions associated with that instance of the recoverable component;

(c) for each recoverable component, creating work identifiers for recoverable operations to be performed if an abnormal termination occurs using at least in part information associated with the reference or token provided to the recoverable component;

(d) employing the supervisory program to identify and group multiple instances of the same recoverable component;

(e) upon the occurrence of an abnormal termination, having the supervisory program determine a recovery scope by examining stored data for failure scopes to identify data processing operations that may not have been completed and have not been recovered, which identified data processing operations are referred to as incomplete failure scopes, and

(f) as incomplete failure scopes are identified, having the supervisory program obtain the reference to each stateless recovery agent associated with such scopes;

(g) after obtaining each such reference, using the supervisory program to call each such stateless recovery agent and providing such agent with a token representing the recovery scope; and

(h) passing control to a recovery agent for long enough to allow the agent to perform recovery for the specified recovery scope it received via the token, including any recovery scopes nested within it.

12. The recovery scope management method of claim 11, wherein step (e) is performed by the recoverable component itself.

13. The recovery scope management method of claim 11, wherein step (d) is performed at least in part by having the supervisory program cause a token to be returned that references a current failure scope of that instance of the recoverable component.

14. The recovery scope management method of claim 11, wherein at least one stateless recovery agent facilitates peer level recovery across the resource sharing group.

15. The recovery scope management method of claim 11, wherein, as part of step (e), the supervisory program determines when some form of recovery somewhere within the resource sharing group is required.

16. The recovery scope management method of claim 11, further comprising the step of:

(i) through the supervisory program, driving an instance of the stateless recovery agent into at least two active process areas within the resource sharing group.

17. The recovery scope management method of claim 16, wherein the active process areas within the resource sharing group are selected from the group of active process areas including, but not limited to, a cluster, a server, a server region, a thread pool, a thread, and a unit of work.

18. The recovery scope management method of claim 11, wherein, as part of step (h) the stateless recovery agent utilizes a recoverable component that looks for work within the failure scope specified via the token that may need recovery.

19. The recovery scope management method of claim 11, wherein, as part of step (d) having the supervisory program nest the failure scopes, such that larger recovery scopes encompass smaller recovery scopes in accordance with a following large-to-small paradigm of cluster, server, server region, thread pool, pool and unit of work, whereby restart and recovery issues associated with component initialization in the Nth address space of a server no longer exist.

20. A computer program product, for and to be used within a multiple processor computer system having a shared dynamic multiple address space server environment, in order to implement a nested recovery scope management method which employs at least one stateless agent, the software product comprising:

a storage medium readable by at least one processing circuit and storing instructions for execution by the processing circuit for performing a method comprising the steps of

(a) installing a supervisory program for directing the recovery of protected resources whose processing was abnormally terminated, and initializing the supervisory program when a first address space of a server within the computer system is started;

(b) as each instance of a recoverable component is initialized, registering that component with the supervisory program, by providing the supervisory program with a reference to a stateless recovery agent that is able to perform recovery functions associated with that instance of the recoverable component;

(c) for each recoverable component, creating work identifiers for recoverable operations to be performed if an abnormal termination occurs using at least in part information associated with the reference provided to the recoverable component;

(d) employing the supervisory program to identify and group multiple instances of the same recoverable component;

(e) upon the occurrence of an abnormal termination, having the supervisory program determine a recovery scope by examining stored data for failure scopes to identify data processing operations that may not have been completed and have not been recovered, which identified data processing operations are referred to as incomplete failure scopes, and

(f) as incomplete failure scopes are identified, having the supervisory program obtain the reference to each stateless recovery agent associated with such scopes;

(g) after obtaining each such reference, using the supervisory program to call each such stateless recovery agent and providing such agent with a token to the recovery scope; and

(h) passing control to a recovery agent for long enough to allow the agent to perform recovery for the specified recovery scope it received via the token, including any recovery scopes nested within it.