Defining a computer recovery process that matches the scope of outage including determining a root cause and performing escalated recovery operations

• US 8,826,077 B2
• Filed: 12/28/2007
• Issued: 09/02/2014
• Est. Priority Date: 12/28/2007
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Bobak, Mythili K., McConnell, Tim A., Swanson, Michael D.
• Primary Examiner(s): SCHELL, JOSEPH O
• Application Number:
  US11/965,862
• Publication Number:
  US 20090172460A1
• Time in Patent Office: 2,440 Days
• Field of Search: 714/26, 714/37, 714/40, 714/46, 714/36
• US Class Current: 714/26
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/1482 by means of middleware or OS functionality
• CPC Class Codes: H04L 41/0654 using network fault recovery ring fault isolation or reconfiguration in loop networks without recovery actions by a network management system H04L12/437

Abstract

Recovery processing is defined that matches the scope of an outage. A programmatic analysis of the resources that have been impacted, of implications of the failure and what degradations have occurred is performed to construct an appropriate level of recovery. This includes selecting the appropriate set of resources to be recovered. Recovery operations are selected based on the current state of the environment.

Claims

1. A computer program product for facilitating recovery in an Information Technology (IT) environment, said computer program product comprising:

a non-transitory computer readable storage medium readable by a processing circuit and storing instructions for execution by the processing circuit for performing a method comprising;

programmatically analyzing, at failure time, information relating to a failure within the IT environment to determine which resource of a plurality of resources is the resource corresponding to a root cause of the failure, said information being related to at least one of;

one or more resources impacted by the failure, one or more implications of the failure, or one or more resources degraded by the failure, wherein the programmatically analyzing comprises iteratively analyzing the information to determine which resource is the resource corresponding to the root cause;

programmatically determining, at failure time, based on the programmatically analyzing, the root cause for the failure; and

programmatically defining, at failure time, by a processor, one or more resources to be included in a set of resources to be recovered and one or more operations to be used in recovering the set of resources based on the analyzed information and the determined root cause, wherein said programmatically defining comprises;

determining, at failure time, the one or more resources affected by the failure, the determining based on the analyzed information and the root cause;

including the one or more resources determined at failure time to be affected by the failure in the set of resources to be recovered, wherein the set of resources is commensurate with a scope of the failure, as determined at failure time, in that the set of resources includes only those resources affected by the failure; and

determining one or more operations to be performed on the set of resources, wherein the determining takes into consideration at least one of;

an effect an operation has on a resource of the set of resources on which the operation is performed, an impact on at least one other resource of the set of resources, or a time it takes to perform the operation, wherein the determining the one or more operations is iterative, and wherein an operation selected to be used in recovery is an escalated operation having an increased severity, in response to a previous operation failing.

2. The computer program product of claim 1, wherein the programmatically defining comprises iteratively performing the defining to provide the one or more resources to be included in the set of resources.

3. The computer program product of claim 1, wherein the determining the one or more operations to be performed on the set of resources comprises selecting one or more operations that enable a defined recovery time goal to be met.

4. The computer program product of claim 1, wherein the method further comprises ordering the one or more operations to be executed.

5. The computer program product of claim 1, wherein the method further comprises executing at least a portion of the operations in parallel.

6. The computer program product of claim 1, wherein at least one operation of the one or more operations to be performed comprises a recovery process having at least one selected recovery operation, and wherein the method further comprises:

executing one or more selected recovery operations of the recovery process; and

for each selected recovery operation that is executed, providing a completion indication to the recovery process.

7. The computer program product of claim 6, wherein the method further comprises performing one or more actions, subsequent to a completion of the recovery process.

8. The computer program product of claim 6, wherein the method further comprises providing a visualization of the recovery process.

9. The computer program product of claim 6, wherein the method further comprises monitoring an execution of at least one of the one or more selected recovery operations being executed or the recovery process.

10. The computer program product of claim 1, wherein a recovery impacts only those resources necessary to achieve a goal specified by a customer.

11. The computer program product of claim 1, wherein the failure is related to a business application of the IT environment, and wherein the business application has an availability goal associated therewith.

12. The computer program product of claim 1, wherein the set of resources comprises a non-responsive resource, and wherein the method further comprises invoking one or more operations for the non-responsive resource.

13. The computer program product of claim 1, wherein the set of resources includes only those resources affected by the failure and those resources to be recovered to meet a quantifiable management goal associated with a business application that is to use the set of resources.

14. The computer program product of claim 1, wherein the programmatically determining the root cause comprises assessing a state of one or more resources at failure time, and relationships associated with the one or more resources to determine the root cause.

15. A computer system for facilitating recovery in an Information Technology (IT) environment, said computer system comprising:

a memory; and

a processor in communication with the memory, wherein the computer system is configured to perform a method, said method comprising;

programmatically analyzing, at failure time, information relating to a failure within the IT environment, said information being related to at least one of;

one or more resources impacted by the failure, one or more implications of the failure, or one or more resources degraded by the failure;

programmatically determining, at failure time, based on the programmatically analyzing, a root cause for the failure; and

programmatically determining a set of resources to be recovered and one or more recovery operations to be used in recovering the set of resources based on the analyzed information and the determined root cause, said set of resources being commensurate with a scope of the failure and said one or more recovery operations being selected based on a current state of the IT environment, wherein the determining the one or more recovery operations is iterative, and wherein a recovery operation selected to be used in recovery is an escalated operation having an increased severity, in response to a previous recovery operation failing.

16. The computer system of claim 15, wherein the determining the one or more recovery operations comprises selecting one or more operations that enable a defined recovery time goal to be met.

17. A method of facilitating recovery in an Information Technology (IT) environment, the method comprising:

programmatically analyzing, at failure time, information relating to a failure within the IT environment, said information being related to at least one of;

one or more resources impacted by the failure, one or more implications of the failure, or one or more resources degraded by the failure;

programmatically determining, at failure time, based on the programmatically analyzing, a root cause for the failure; and

programmatically determining, by a processor, a set of resources to be recovered and one or more recovery operations to be used in recovering the set of resources based on the analyzed information and the determined root cause, said set of resources being commensurate with a scope of the failure and said one or more recovery operations being selected based on a current state of the IT environment, wherein the determining the one or more recovery operations is iterative, and wherein a recovery operation selected to be used in recovery is an escalated operation having an increased severity, in response to a previous recovery operation failing.

18. The method of claim 17, wherein the determining the one or more recovery operations comprises selecting one or more operations that enable a defined recovery time goal to be met.

19. The method of claim 17, wherein at least one recovery operation of the one or more recovery operations comprises a recovery process having at least one selected recovery operation, and wherein the method further comprises:

executing one or more selected recovery operations of the recovery process; and

for each selected recovery operation that is executed, providing a completion indication to the recovery process.

20. A computer program product for facilitating recovery in an Information Technology (IT) environment, the computer program product comprising:

a non-transitory computer readable storage medium readable by a processor and storing instructions for execution by the processor for performing a method comprising;

programmatically analyzing, at failure time, information relating to a failure within the IT environment, said information being related to at least one of;

one or more resources impacted by the failure, one or more implications of the failure, or one or more resources degraded by the failure;

programmatically determining, at failure time, based on the programmatically analyzing, a root cause for the failure; and

programmatically determining a set of resources to be recovered and one or more recovery operations to be used in recovering the set of resources based on the analyzed information and the determined root cause, said set of resources being commensurate with a scope of the failure and said one or more recovery operations being selected based on a current state of the IT environment, wherein the determining the one or more recovery operations is iterative, and wherein a recovery operation selected to be used in recovery is an escalated operation having an increased severity, in response to a previous recovery operation failing.