Managing a fault tolerant system

• US 7,299,385 B2
• Filed: 07/28/2004
• Issued: 11/20/2007
• Est. Priority Date: 07/28/2004
• Status: Active Grant

• Current Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Voigt, Douglas L.
• Primary Examiner(s): Bonzo; Bryce P
• Application Number:
  US10/900,948
• Publication Number:
  US 20060026451A1
• Time in Patent Office: 1,210 Days
• Field of Search: 714/57
• US Class Current: 714/57
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0769 Readable error formats, e.g. cross-platform generic formats, human understandable formats

Abstract

Systems and methods for managing a fault tolerant system are disclosed. In one implementation a system for managing a fault tolerant system comprises a configuration manager that receives configuration events from the fault tolerant system, a fault normalizer that receives fault events from the fault tolerant system; and a fault tolerance logic engine that constructs a model of the fault tolerant system based on inputs from the configuration manager and generates reporting events in response to inputs from the fault normalizer.

Claims

1. A method of monitoring a fault tolerant system, comprising:

maintaining an abstract model of the fault tolerant system;

monitoring operation of the fault tolerant system;

applying fault events received from the fault tolerant system to the abstract model; and

reporting one or more changes in the abstract model to a component in the fault tolerant system, wherein maintaining an abstract model of the fault tolerant system comprises;

receiving a configuration event indicating the removal of at least one component to the fault tolerant system; and

in response, removing at least one corresponding component from the abstract model; and

wherein removing at least one corresponding component from the abstract model comprises;

dissolving the logical association between the at least one corresponding component and one or more components dependent on the at least one corresponding component; and

recalculating a state of the one or more components dependent on the at least one corresponding component.

2. The method of claim 1, wherein maintaining an abstract model of the fault tolerant system comprises:

receiving a configuration event indicating the addition of at least one component to the fault tolerant system; and

in response, adding at least one corresponding component to the abstract model.

3. The method of claim 2, further comprising representing redundancy relationships between the at least one corresponding component and one or more other components in the abstract model.

4. The method of claim 3, further comprising representing dependency relationships between the at least one corresponding component and one or more other components in the abstract model.

5. The method of claim 1, wherein removing at least one corresponding component from the abstract model comprises:

dissolving one or more logical associations between the at least one corresponding component and at least one additional component in the abstract model.

6. The method of claim 1, wherein maintaining an abstract model of the fault tolerant system comprises:

receiving a configuration event indicating an addition or a removal of at least one component to a logical group in the fault tolerant system; and

in response, updating the logical group to reflect the addition or removal of the at least one component; and

recalculating a state of a group of components dependent on the at least one component.

7. The method of claim 6, wherein recalculating a state of one or more components dependent on the at least one component comprises propagating a failure state associated with the at least one component to a group of components dependent on the at least one component.

8. The method of claim 7, further comprising:

maintaining a failure indicator that represents one or more failure parameters in the group of components dependent on the at least one component; and

setting a group state parameter to indicate a group failure if the failure indicator exceeds a threshold.

9. The method of claim 7, further comprising:

maintaining a failure indicator that represents one or more failure parameters in the group of components dependent on the at least one component; and

setting a group state parameter to indicate a group failure if any group state parameter indicates a failure.

10. The method of claim 1, wherein applying fault events received from the fault tolerant system to the abstract model comprises:

receiving a state change event for one or more components in the fault tolerant system;

updating the abstract model to propagate the state change event to a group of components dependent on the at least one component.

11. The method of claim 10, wherein updating the abstract model to propagate the state change event to a group of components dependent on the at least one component comprises:

maintaining a failure indicator that represents one or more failure parameters in the group of components dependent on the at least one component; and

setting a group state parameter to indicate a group failure if the failure indicator exceeds a threshold.

12. The method of claim 10, wherein updating the abstract model to propagate the state change event to a group of components dependent on the at least one component comprises:

maintaining a failure indicator that represents one or more failure parameters in the group of components dependent on the at least one component; and

setting a group state parameter to indicate a group failure if any group state parameter indicates a failure.

13. The method of claim 1, wherein reporting one or more changes in the abstract model to a component in the fault tolerant system comprises searching an event generation registry table for entries corresponding to a specific node in the abstract model.

14. The method of claim 1, wherein the means for generating one or more reporting events in response to changes in the availability of a component in the abstract model comprises logic instructions executable on a processor that search an event generation registry table for entries corresponding to a specific node in the abstract model.

15. A method of monitoring a fault tolerant system, comprising:

maintaining an abstract model of the fault tolerant system;

monitoring operation of the fault tolerant system;

applying fault events received from the fault tolerant system to the abstract model; and

reporting one or more changes in the abstract model to a component in the fault tolerant system, wherein maintaining an abstract model of the fault tolerant system comprises;

receiving a configuration event indicating an addition or a removal of at least one component to a logical group in the fault tolerant system; and

in response, updating the logical group to reflect the addition or removal of the at least one component; and

recalculating a state of a group of components dependent on the at least one component, and wherein recalculating a state of one or more components dependent on the at least one component comprises propagating a failure state associated with the at least one component to a group of components dependent on the at least one component;

maintaining a failure indicator that represents one or more failure parameters in the group of components dependent on the at least one component; and

setting a group state parameter to indicate a group failure if the failure indicator exceeds a threshold.

16. The method of claim 15, wherein maintaining an abstract model of the fault tolerant system comprises:

receiving a configuration event indicating the addition of at least one component to the fault tolerant system; and

in response, adding at least one corresponding component to the abstract model.

17. The method of claim 16, further comprising representing redundancy relationships between the at least one corresponding component and one or more other components in the abstract model.

18. The method of claim 17, further comprising representing dependency relationships between the at least one corresponding component and one or more other components in the abstract model.

19. The method of claim 15, wherein maintaining an abstract model of the fault tolerant system comprises:

receiving a configuration event indicating the removal of at least one component to the fault tolerant system; and

in response, removing at least one corresponding component from the abstract model.

20. The method of claim 19, wherein removing at least one corresponding component from the abstract model comprises:

dissolving the logical association between the at least one corresponding component and one or more components dependent on the at least one corresponding component; and

recalculating a state of the one or more components dependent on the at least one corresponding component.

21. The method of claim 19, wherein removing at least one corresponding component from the abstract model comprises:

dissolving one or more logical associations between the at least one corresponding component and at least one additional component in the abstract model.

22. The method of claim 15, further comprising:

maintaining a failure indicator that represents one or more failure parameters in the group of components dependent on the at least one component; and

setting a group state parameter to indicate a group failure if any group state parameter indicates a failure.

23. The method of claim 15, wherein applying fault events received from the fault tolerant system to the abstract model comprises:

receiving a state change event for one or more components in the fault tolerant system;

updating the abstract model to propagate the state change event to a group of components dependent on the at least one component.

24. The method of claim 23, wherein updating the abstract model to propagate the state change event to a group of components dependent on the at least one component comprises:

maintaining a failure indicator that represents one or more failure parameters in the group of components dependent on the at least one component; and

setting a group state parameter to indicate a group failure if the failure indicator exceeds a threshold.

25. The method of claim 23, wherein updating the abstract model to propagate the state change event to a group of components dependent on the at least one component comprises:

maintaining a failure indicator that represents one or more failure parameters in the group of components dependent on the at least one component; and

setting a group state parameter to indicate a group failure if any group state parameter indicates a failure.

26. The method of claim 15, wherein reporting one or more changes in the abstract model to a component in the fault tolerant system comprises searching an event generation registry table for entries corresponding to a specific node in the abstract model.