User-defined storage system failure detection and failover management

• US 9,535,806 B1
• Filed: 03/30/2015
• Issued: 01/03/2017
• Est. Priority Date: 03/30/2015
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Cong, Honglei, Wienand, Uwe, Qian, Yue, Huang, Zhou, Zhao, Mei, Xu, Ning
• Primary Examiner(s): Ko, Chae
• Application Number:
  US14/672,983
• Time in Patent Office: 645 Days
• Field of Search: 714/6.3, 714/4.11
• US Class Current: 1/1
• CPC Class Codes: G06F 11/00 Error detection; Error correction; Monitoring error detection, correction or monitoring in information storage based on relative movement between record carrier and transducer G11B20/18; monitoring, i.e. supervising the progress of recording or reproducing G11B27/36; in static stores G11C29/00
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/1435 using file system or storage system metadata
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/2069 Management of state, configuration or failover

Abstract

A storage system failure detection and failover application includes a command execution engine that loads and interprets both user-defined storage system management functions, and vendor-defined storage system management functions. The user-defined storage system management functions allow the system to detect and handle failure conditions indicating both the current state of an active storage system, and the current state of one or more applications that consume a storage service or services provided by the active storage system. The user-defined storage management functions also allow the system to perform failover processing in a way that allows applications to perform application-specific processing during storage system failover.

Claims

1. A computer-implemented method of providing user-defined failure detection and failover management, comprising executing, on at least one processor, the steps of:

providing, by an active storage system, external storage to at least one application executing on a computer, the external storage provided to the application through a storage service provided by the active storage system to the computer over a network, the active storage system obtained by a customer from a storage system vendor,

loading, into a memory of the computer

i) a plurality of user-defined storage system management functions, wherein the user-defined storage system management functions were developed by the customer of the storage system vendor, wherein the user-defined storage system management functions are communicable with the application executing on the computer and operable to monitor storage system state parameters indicating a current state of the active storage system, and wherein the storage system state parameters are stored in a set of system environment parameters located in the memory of the computer;

ii) a plurality of vendor-defined storage system management functions, wherein the vendor-defined storage system management functions were obtained by the customer from the storage system vendor, wherein the vendor-defined storage system management functions are communicable with the active storage system, and wherein the vendor-defined storage system management functions are operable to maintain the storage system state parameters by storing a current state of the active storage system into the storage system state parameters; and

detecting and processing a plurality of storage failure conditions by executing the user-defined storage system management functions and the vendor-defined storage system management functions on the computer to provide user-defined failure detection and failover management.

2. The method of claim 1, wherein detecting and processing the storage failure conditions includes:

detecting, by a first one of the vendor-defined storage system management functions, that the active storage system has failed;

in response to detecting that the active storage system has failed, a second one of the vendor-defined storage system management functions modifying the storage system state parameters to indicate that the active storage system has failed;

reading, by a first one of the user-defined storage system management functions, the storage system state parameters, and

responsive to the storage system state parameters indicating that the active storage system has failed, a second one of the user-defined storage system management functions sending an indication to the application that the active storage system has failed.

3. The method of claim 2, wherein detecting and processing the storage failure conditions further includes:

in response to detecting that the active storage system has failed, a third one of the vendor-defined storage system management function triggering a failover operation causing a backup storage system to replace the active storage system in providing the external storage to the application executing on the computer, at least in part by causing the backup storage system to begin providing the storage service from the backup storage system to the computer;

in response to completion of the failover operation, a fourth one of the vendor-defined storage system management functions modifying the storage system state parameters to indicate that the backup storage system is the new active storage system, and

in response to the storage system state parameters indicating that the backup storage system is the new active storage system, a third one of the user-defined storage system management functions sending an indication to the application that the backup storage system has begun providing the storage service to the computer.

4. The method of claim 1, wherein detecting and processing the storage failure conditions includes:

detecting, by one of the user-defined storage system management functions, that the application has experienced a storage service failure;

checking, by one of the vendor-defined storage system management functions, the operational status of the active storage system, and modifying the storage system state parameters to indicate that the active storage system has experienced a temporary failure; and

sending, by a second one of the user-defined storage system management functions, in response to the storage system state parameters indicating that the active storage system has experienced a temporary failure, an indication to the application that the active storage system has experienced a temporary failure.

5. The method of claim 4, wherein detecting and processing the storage failure conditions further includes:

re-checking the operational status of the active storage system by the vendor-defined storage management function, and in response to the active storage system indicating that the active storage system has recovered from the temporary failure, modifying the storage system state parameters to indicate that the active storage system has recovered from the temporary failure; and

responsive to the storage system state parameters indicating that the active storage system has recovered from the temporary failure, a third one of the user-defined storage system management functions indicating to the application that the active storage system has recovered from the temporary failure.

6. The method of claim 1, wherein detecting and processing the storage failure conditions includes

detecting, by one of the user-defined storage system management functions, that the application has experienced a storage service failure;

one of the vendor-defined storage system management functions determining that the active storage system has experienced a permanent failure; and

a second one of the user-defined storage system management functions sending an indication to the application that the active storage system has permanently failed.

7. The method of claim 1, wherein detecting and processing the storage failure conditions includes:

detecting, by one of the user-defined storage system management functions, that the application has experienced a critical failure; and

in response to the user-defined function detecting that the application has experienced a critical failure, one of the vendor-defined storage system management functions causing the vendor-defined storage system management function to trigger, without checking the operational status of the active storage system, a failover operation causing a backup storage system to replace the active storage system in providing the external storage to the at least one application executing on the computer, at least in part by causing the backup storage system to begin providing the storage service from the backup storage system to the computer.

8. The method of claim 1, further comprising:

wherein the user-defined storage system management functions comprise a plurality of scripts;

wherein detecting and processing the storage failure conditions includes executing the user-defined storage system management functions using a script interpreter; and

wherein loading the user-defined storage system management functions and the vendor-defined storage system management functions into the memory of the computer enables the user-defined storage system management functions to call the vendor-defined storage system management functions.

9. A computer for providing user-defined failure detection and failover management, wherein the computer is communicable with an active storage system obtained by a customer from a storage system vendor, and the active storage system provides external storage to at least one application executing on the computer, the external storage provided to the application through a storage service provided by the active storage system to the computer over a network, comprising:

at least one hardware processor;

a memory having program code stored thereon for execution on the processor, the program code including

program code for loading, into a memory of the computer

i) a plurality of user-defined storage system management functions, wherein the user-defined storage system management functions were developed by the customer of the storage system vendor, wherein the user-defined storage system management functions are communicable with the application executing on the computer and operable to monitor storage system state parameters indicating a current state of the active storage system, and wherein the storage system state parameters are stored in a set of system environment parameters located in the memory of the computer; and

ii) a plurality of vendor-defined storage system management functions, wherein the vendor-defined storage system management functions were obtained by the customer from the storage system vendor, wherein the vendor-defined storage system management functions are communicable with the active storage system, and wherein the vendor-defined storage system management functions are operable to maintain the storage system state parameters by storing a current state of the active storage system into the storage system state parameters; and

program code for detecting and processing a plurality of storage failure conditions by executing the user-defined storage system management functions and the vendor-defined storage system management functions on the computer to provide user-defined failure detection and failover management.

10. The system of claim 9, wherein the program code for detecting and processing the storage failure conditions includes program code for:

executing a first one of the vendor-defined storage system management functions to detect that the active storage system has failed;

executing a second one of the vendor-defined storage system management functions, in response to detecting that the active storage system has failed, to modify the storage system state parameters to indicate that the active storage system has failed;

executing a first one of the user-defined storage system management functions to read the storage system state parameters, and

executing a second one of the user-defined storage system management functions, responsive to the storage system state parameters indicating that the active storage system has failed, to send an indication that the active storage system has failed to the application.

11. The system of claim 10, wherein the program code for detecting and processing the storage failure conditions further includes program code for:

executing a third one of the vendor-defined storage system management functions, in response to detecting that the active storage system has failed, to trigger a failover operation causing a backup storage system to replace the active storage system in providing the external storage to the application executing on the computer, at least in part by causing the backup storage system to begin providing the storage service from the backup storage system to the computer;

executing a fourth one of the vendor-defined storage system management functions, in response to completion of the failover operation, to modify the storage system state parameters to indicate that the backup storage system is the new active storage system, and

executing a third one of the user-defined storage system management functions, in response to the storage system state parameters indicating that the backup storage system is the new active storage system, to send an indication to the application that the backup storage system has begun providing the storage service to the computer.

12. The system of claim 9, wherein the program code for detecting and processing the storage failure conditions includes program code for:

executing one of the user-defined storage system management functions to detect that the application has experienced a storage service failure;

executing one of the vendor-defined storage system management functions to check the operational status of the active storage system,

executing a second one of the vendor-defined storage system management functions to modify the storage system state parameters to indicate that the active storage system has experienced a temporary failure; and

executing a second one of the user-defined storage system management functions, in response to the storage system state parameters indicating that the active storage system has experienced a temporary failure, to send an indication that the active storage system has experienced a temporary failure to the application.

13. The system of claim 12, wherein the program code for detecting and processing the storage failure conditions further includes program code for:

executing the one of the vendor-defined storage management functions to check the operational status of the active storage system;

in response to the active storage system indicating that the active storage system has recovered from the temporary failure, executing the second one of the vendor-defined storage management functions to modify the storage system state parameters to indicate that the active storage system has recovered from the temporary failure; and

responsive to the storage system state parameters indicating that the active storage system has recovered from the temporary failure, executing a third one of the user-defined storage system management functions to indicate to the application that the active storage system has recovered from the temporary failure.

14. The system of claim 9, wherein the program code for detecting and processing the storage failure conditions includes program code for:

executing one of the user-defined storage system management functions to detect that the application has experienced a storage service failure;

executing one of the vendor-defined storage system management functions to determine that the active storage system has experienced a permanent failure; and

executing a second one of the user-defined storage system management functions to send an indication to the application that the active storage system has permanently failed.

15. The system of claim 9, wherein the program code for detecting and processing the storage failure conditions includes program code for:

executing one of the user-defined storage system management functions to detect that the application has experienced a critical failure; and

executing one of the vendor-defined storage system management functions in response to the user-defined function detecting that the application has experienced a critical failure, to trigger, without checking the operational status of the active storage system, a failover operation causing a backup storage system to replace the active storage system in providing the external storage to the at least one application executing on the computer, at least in part by causing the backup storage system to begin providing the storage service from the backup storage system to the computer.

16. The system of claim 9, further comprising:

wherein the user-defined storage system management functions comprise a plurality of scripts;

wherein the program code for detecting and processing the storage failure conditions includes a script interpreter for executing the user-defined storage system management functions on the computer; and

wherein loading the user-defined storage system management functions and the vendor-defined storage system management functions into the memory of the computer enables the user-defined storage system management functions to call the vendor-defined storage system management functions.

17. A non-transitory computer readable medium for providing user-defined failure detection and failover management in a computer, wherein the computer is communicable with an active storage system obtained by a customer from a storage system vendor, wherein the active storage system provides external storage to at least one application executing on the computer, wherein the external storage is provided to the application through a storage service provided by the active storage system to the computer over a network, the non-transitory medium having instructions stored thereon, that when executed on a processor of a computer, perform the steps of:

loading, into a memory of the computer

i) a plurality of user-defined storage system management functions, wherein the user-defined storage system management functions were developed by the customer of the storage system vendor, wherein the user-defined storage system management functions are communicable with the application executing on the computer and operable to monitor storage system state parameters indicating a current state of the active storage system, and wherein the storage system state parameters are stored in a set of system environment parameters located in the memory of the computer;

ii) a plurality of vendor-defined storage system management functions, wherein the vendor-defined storage system management functions were obtained by the customer from the storage system vendor, wherein the vendor-defined storage system management functions are communicable with the active storage system, and wherein the vendor-defined storage system management functions are operable to maintain the storage system state parameters by storing a current state of the active storage system into the storage system state parameters; and

detecting and processing a plurality of storage failure conditions by executing the user-defined storage system management functions and the vendor-defined storage system management functions on the computer to provide user-defined failure detection and failover management.