Managing multi-step storage management operations by using fault recovery policies

• US 9,811,421 B1
• Filed: 03/31/2011
• Issued: 11/07/2017
• Est. Priority Date: 03/31/2011
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Wu, RongZhang, Tang, Xuan, Wang, Yifan, Zhang, Yiyang
• Primary Examiner(s): Savla, Arpan P.
• Application Number:
  US13/076,594
• Time in Patent Office: 2,413 Days
• Field of Search: 711154, 711161, 711162, 714 2, 714 61- 632, 714 51, 718100
• CPC Class Codes: G06F 11/07 Responding to the occurrence of a fault, e.g. fault tolerance
• CPC Class Codes: G06F 11/0715 in a system implementing multitasking multitasking per se G06F9/46
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/14 Error detection or correction of the data by redundancy in operation G06F11/16 takes precedence
• CPC Class Codes: G06F 11/1402 Saving, restoring, recovering or retrying
• CPC Class Codes: G06F 9/485 Task life-cycle, e.g. stopping, restarting, resuming execution G06F9/4881 takes precedence

Abstract

A method is used in managing multi-step storage management operations. A policy is defined for a task of a multi-step storage management operation. The multi-step storage management operation includes multiple tasks. The policy for the task indicates directions for reacting to results of the task of the multi-step storage management operation. The task is invoked. The policy for the task is invoked based on results of the task.

Claims

1. A method for use in managing multi-step storage management operations, the method comprising:

receiving a request to execute a multi-step storage management operation, wherein the multi-step storage management operation is comprised of a set of discrete tasks, wherein the set of discrete tasks are bundled together and executed asynchronously, wherein upon failure of a discrete task, a fault recovery policy for the discrete task is invoked;

using a fault recovery policy framework to enable a user to recover from a failure occurred during execution of the multi-step storage management operation by programmatically defining a set of fault recovery policies for the set of discrete tasks included in the multi-step storage management operation, each fault recovery policy of the set of fault recovery policies being associated with a discrete task of the set of discrete tasks, wherein a fault recovery policy for a discrete task of the set of discrete tasks indicates directions for reacting to results of the discrete task of the multi-step storage management operation, wherein reacting to the results of the discrete task includes a set of recovery actions for recovering from a failure;

invoking each discrete task of the set of discrete tasks upon receiving the request to execute the multi-step storage management operation;

based on results of each discrete task of the set of discrete tasks, invoking a fault recovery policy from the set of fault recovery policies for recovering from a failure occurred upon invoking a discrete task during execution of the multi-step storage management operation, wherein the fault recovery policy is associated with the discrete task, wherein the fault recovery policy is invoked upon determining that the discrete task has failed to complete during execution of the multi-step storage management operation; and

based on the invocation of the fault recovery policy, performing a recovery action associated with the fault recovery policy upon resuming the execution of the multi-step storage management operation after the failure of the discrete task of the multi-step storage management operation, wherein a recovery action includes a continue action, a re-run action, stop on error action, and a rollback action.

2. The method of claim 1, wherein a recovery action is associated with a fault recovery policy.

3. The method of claim 2, wherein invoking the fault recovery policy for the discrete task further comprising performing the recovery action associated with the policy for the discrete task.

4. The method of claim 1, further comprising defining a policy for the multi-step storage management operation.

5. The method of claim 1, further comprising:

performing a recovery action associated with a policy for the multi-step storage management operation upon failure of the multi-step storage management operation.

6. The method of claim 1, further comprising:

performing a recovery action associated with a fault recovery policy for a discrete task of the set of discrete tasks of the multi-step storage management operation upon failure of the discrete task of the multi-step storage management operation.

7. The method of claim 1, further comprising:

creating a set of discrete tasks for the multi-step storage management operation; and

executing each discrete task of the set of discrete tasks for the multi-step storage management operation.

8. The method of claim 1, wherein a fault recovery policy for a discrete task is defined programmatically by a user.

9. The method of claim 1, further comprising storing information for the multi-step storage management operation in a persistent storage.

10. The method of claim 1, further comprising:

performing a recovery action associated with a policy for the multi-step storage management operation upon resuming the execution of the multi-step storage management operation after a failure of a discrete task of the multi-step storage management operation by using information stored in a persistent storage.

11. A system for use in managing multi-step storage management operations, the system comprising a processor configured to:

receive a request to execute a multi-step storage management operation, wherein the multi-step storage management operation is comprised of a set of discrete tasks, wherein the set of discrete tasks are bundled together and executed asynchronously, wherein upon failure of a discrete task, a fault recovery policy for the discrete task is invoked;

use a fault recovery policy framework to enable a user to recover from a failure occurred during execution of the multi-step storage management operation by programmatically defining a set of fault recovery policies for the set of discrete tasks included in the multi-step storage management operation, each fault recovery policy of the set of fault recovery policies being associated with a discrete task of the set of discrete tasks, wherein a fault recovery policy for a discrete task of the set of discrete tasks indicates directions for reacting to results of the discrete task of the multi-step storage management operation, wherein reacting to the results of the discrete task includes a set of recovery actions for recovering from a failure;

invoke each discrete task of the set of discrete tasks upon receiving the request to execute the multi-step storage management operation;

invoke, based on results of each discrete task of the set of discrete tasks, a fault recovery policy from the set of fault recovery policies for recovering from a failure occurred upon invoking a discrete task during execution of the multi-step storage management operation, wherein the fault recovery policy is associated with the discrete task, wherein the fault recovery policy is invoked upon determining that the discrete task has failed to complete during execution of the multi-step storage management operation; and

perform, based on the invocation of the fault recovery policy, a recovery action associated with the fault recovery policy upon resuming the execution of the multi-step storage management operation after the failure of the discrete task of the multi-step storage management operation, wherein a recovery action includes a continue action, a re-run action, stop on error action, and a rollback action.

12. The system of claim 11, wherein a recovery action is associated with a fault recovery policy.

13. The system of claim 12, wherein invoking the fault recovery policy for the discrete task further comprising:

perform the recovery action associated with the policy for the task.

14. The system of claim 11, further comprising:

define a policy for the multi-step storage management operation.

15. The system of claim 11, further comprising:

perform a recovery action associated with a policy for the multi-step storage management operation upon failure of the multi-step storage management operation.

16. The system of claim 11, further comprising:

perform a recovery action associated with a fault recovery policy for a discrete task of the set of discrete tasks of the multi-step storage management operation upon failure of the discrete task of the multi-step storage management operation.

17. The system of claim 11, further comprising:

create a set of discrete tasks for the multi-step storage management operation; and

execute each discrete task of the set of discrete tasks for the multi-step storage management operation.

18. The system of claim 11, wherein a fault recovery policy for a discrete task is defined programmatically by a user.

19. The system of claim 11, further comprising:

store information for the multi-step storage management operation in a persistent storage.

20. The system of claim 11, further comprising:

perform a recovery action associated with a policy for the multi-step storage management operation upon resuming the execution of the multi-step storage management operation after a failure of a discrete task of the multi-step storage management operation by using information stored in a persistent storage.