Method for monitoring and recovery of subsystems in a distributed/clustered system

• US 5,805,785 A
• Filed: 02/27/1996
• Issued: 09/08/1998
• Est. Priority Date: 02/27/1996
• Status: Expired due to Term

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Leff, Avraham, King, Richard Pervin, Dias, Daniel Manuel
• Primary Examiner(s): Beausoliel, Jr., Robert W.
• Assistant Examiner(s): Iqbal, Nadeem
• Application Number:
  US08/606,765
• Time in Patent Office: 924 Days
• Field of Search: 395/182.02, 395/182.18, 395/182.13, 395/182.14, 395/200.13, 395/200.18, 395/200.2, 395/575, 395/182.1, 395/183.15, 395/184.01 , 395/185.01, 395/185.04, 395/700, 395/600, 395/427, 395/703, 395/500, 395/680, 395/670, 395/650
• US Class Current: 714/4.4
• CPC Class Codes: G06F 11/1438 Restarting or rejuvenating

Abstract

A system and method for a general and extensible infrastructure providing monitoring and recovery of interdependent systems in a distributed/clustered system is disclosed. Subsystems, built without provision for high availability, are incorporated into the infrastructure without modification to core subsystem function. The infrastructure is comprised of one or more computing nodes connected by one or more interconnection networks, and running one or more distributed subsystems. The infrastructure monitors the computing nodes using one or more heartbeat and membership protocols, and monitors the said distributed subsystems by subsystem-specific monitors. Events detected by monitors are sent to event handlers. Event handlers process events by filtering them through activities such as event correlation, removal of duplicates, and rollup. Filtered events are given by Event Managers to Recovery Drivers which determine the recovery program corresponding to the event, and executing the recovery program or set of recovery actions by coordination among the recovery managers. Given failures of said event handlers or recovery managers, the infrastructure performs the additional steps of: coordinating among remaining event handlers and recovery managers to handle completion or termination of ongoing recovery actions, discovering the current state of the system by resetting the said monitors, and handling any new failure events that may have occurred in the interim.

Claims

1. A method for monitoring and recovery of subsystems in a distributed computer system comprising the steps of:

(a) executing a distributed software subsystem on the distributed system, said software subsystem not being self-recoverable from failure events;

(b) providing user-defined monitors for the software subsystem, each of the user-defined monitors including a set of user defined events to be detected; and

,

(c) responsive to an occurrence of one of the events, performing recovery actions coordinated among the nodes of the distributed computer system as controlled by a user specified recovery program.

2. The method of claim 1 comprising the further steps of:

distributed system comprising further steps of;

(d) detecting a failure of an element of a distributed recovery manager executing the distributed recovery program;

(e) responsive to detection of the failure, coordinating among remaining elements of the distributed recovery manager to handle completion of ongoing recovery actions;

(f) determining the current state of the distributed system; and

,

(g) based on a comparison of the current state of the distributed system and the user specified events detected in step (b) determining any additional recovery actions to be taken.

3. A method for operating a distributed system comprising the steps of:

executing a set of interdependent software subsystems run on nodes of the distributed system, each of said software subsystems not being self-recoverable from failure events;

providing a user-defined set of monitors that probe the health of each subsystem and report failure events;

providing a user-defined recovery program for each of a plurality of the failure events; and

,

using the user-defined recovery program, coordinating and synchronizing the recovery of the interdependent software subsystems.

4. The method of claim 3 wherein each of the software subsystems is embodied as a set of program instances with one or more program instances running on each node in the distributed system.

5. The method of claim 3 wherein the failure events are reported to an event manager and wherein the event manager reports only selected ones of the events based on a filtering criteria.

6. The method of claim 3 wherein the user defined recovery program is selected based on a set of rules provided by a user.

7. A method of providing error recovery in a distributed system, comprising the steps of:

monitoring computing nodes of the distributed system using at least one heartbeat and membership protocol,

monitoring for subsystems running on the computing nodes using user defined monitors, at least one of said subsystems not being self-recoverable from failure events;

reporting events detected by the user-defined monitors to at least one event handler;

filtering the events in the event handler so as to provide filtered events;

applying a set of rules to the filtered events to select a user-defined recovery program from a set of user-defined recovery programs; and

,

coordinating among the nodes in the distributed system to execute a selected recovery program.

8. The method of claim 7 wherein the filtering includes at least one of event correlation, removal of duplicate events, and rollup.

9. The method of claim 7 wherein the user-defined recovery programs comprise recovery steps which specify a set of nodes on which the recovery programs run, an action to be performed, a maximum time that the action can take, actions to be taken in the event that a recovery step fails, parameters required for the recovery step and which sets of recovery steps can be carried out in parallel.

10. The method of claim 9 comprising the further step of interspersing recovery steps by way of user-defined barrier commands that require that all the prior recovery steps be completed before proceeding beyond a barrier.

11. A system for providing error recovery in a distributed system, comprising:

a plurality of monitors in computing nodes of the distributed system using at least one heartbeat and membership protocol,

a plurality of user-defined monitors for subsystems running on the computing nodes, the monitors including means for detecting events and sending reports of said events to event handlers, at least one of said subsystems not being self-recoverable from failure events;

means for processing events, in the event handlers, by filtering the events by way of activities such as event correlation, removal of duplicate events, and rollup;

means providing filtered events to recovery drivers, which have a rule base which specify user-defined recovery programs corresponding to events; and

,

means for coordinating among the nodes in the distributed system to execute the recovery program.

12. The system of claim 11 wherein the recovery programs comprise recovery steps which specify the set of nodes on which they run, an action to be performed, a maximum time that the action can take, actions to be taken in the event that the recovery step fails, and parameters required for the recovery step and which sets of recovery steps can be carried out in parallel.