Handling zero fault tolerance events in machines where failure likely results in unacceptable loss

• US 10,691,518 B2
• Filed: 07/10/2019
• Issued: 06/23/2020
• Est. Priority Date: 12/01/2017
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Hardy, Clint A., Borlick, Matthew G., Gerhard, Adrian C., Gupta, Lokesh M.
• Primary Examiner(s): Kim, Matthew M
• Assistant Examiner(s): Chowdhury, Indranil
• Application Number:
  US16/508,291
• Publication Number:
  US 20190332457A1
• Time in Patent Office: 349 Days
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD
• CPC Class Codes: G06F 9/485 Task life-cycle, e.g. stopping, restarting, resuming execution G06F9/4881 takes precedence
• CPC Class Codes: G06F 9/5011 the resources being hardware resources other than CPUs, Servers and Terminals

Abstract

Provided are a computer program product, system, and method for managing I/O requests to a storage array of storage devices in a machine having a processor node and device adaptor. In response to initiating a rebuild of data in the storage array, the device adaptor determines whether a remaining fault tolerance at the storage array comprises a non-zero fault tolerance that permits at least one further storage device to fail and still allow recovery of data stored in the storage array. In response to determining that the remaining fault tolerance is a zero fault tolerance that does not permit at least one storage device to fail and allow recovery of data, the device adaptor sends a message to the processor node to cause the processor node to initiate an emergency protocol to terminate a mission critical operation when the processor node is performing the mission critical operation.

Claims

1. A computer program product for managing I/O requests to a storage array of storage devices in a machine having a processor node and device adaptor, the computer program product comprising a computer readable storage medium having computer readable program code embodied therein that is executable to perform operations, the operations comprising:

in response to the device adaptor initiating a rebuild of data at the storage devices in the storage array, determining whether a remaining fault tolerance at the storage array comprises a non-zero fault tolerance that permits at least one further storage device of the storage devices to fail and still allow recovery of data stored in the storage array; and

determining, by the device adaptor, whether processor utilization at the device adaptor exceeds a utilization threshold after determining that the remaining fault tolerance is not a zero fault tolerance;

initiating, by the device adaptor, an operation to reduce a rate at which I/O requests to the storage array are processed at the device adaptor in response to determining that the processor utilization at the device adaptor exceeds the utilization threshold.

2. The computer program product of claim 1, wherein the I/O requests whose rate of processing is reduced at the device adaptor comprise non-critical I/O requests.

3. The computer program product of claim 1, wherein the reducing the rate at which I/O requests are processed comprises the device adaptor delaying processing of the I/O requests.

4. The computer program product of claim 1, wherein the initiating the operation to reduce the rate at which I/O requests are processed at the device adaptor comprises:

sending, by the device adaptor, a message to the processor node to cause the processor node to reduce a rate at which requests are transferred from the processor node to the device adaptor.

5. The computer program product of claim 4, wherein the operations further comprise:

additionally determining, by the device adaptor, whether processor utilization at the device adaptor exceeds the utilization threshold after sending the message to the processor node;

reducing, by the device adaptor, a rate at which the device adaptor processes I/O requests from the processor node in response to additionally determining that the processor utilization at the device adaptor exceeds the utilization threshold.

6. The computer program product of claim 1, wherein the operations further comprise:

in response to determining that the remaining fault tolerance is a zero fault tolerance that does not permit at least one storage device to fail and allow recovery of data, sending, by the device adaptor, a second message to the processor node to cause the processor node to initiate an emergency protocol to terminate a mission critical operation when the processor node is performing the mission critical operation.

7. The computer program product of claim 1, wherein the operations further comprise:

determining whether the processor node is performing a mission critical operation in response to the rebuild of data at the storage devices in the storage array, wherein the determining whether the remaining fault tolerance at the storage array comprises the non-zero fault tolerance, determining whether the processor utilization exceeds the utilization threshold, and initiating the operation to reduce the rate at which I/O requests are processed are performed in response to determining that the processor node is performing the mission critical operation.

8. A system for managing I/O requests, comprising:

a processor node;

a storage array of storage devices;

a device adaptor managing access to the storage array, wherein the device adaptor performs operations, the operations comprising;

in response to the device adaptor initiating a rebuild of data at the storage devices in the storage array, determining whether a remaining fault tolerance at the storage array comprises a non-zero fault tolerance that permits at least one further storage device of the storage devices to fail and still allow recovery of data stored in the storage array; and

determining, by the device adaptor, whether processor utilization at the device adaptor exceeds a utilization threshold after determining that the remaining fault tolerance is not a zero fault tolerance;

initiating, by the device adaptor, an operation to reduce a rate at which I/O requests to the storage array are processed at the device adaptor in response to determining that the processor utilization at the device adaptor exceeds the utilization threshold.

9. The system of claim 8, wherein the I/O requests whose rate of processing is reduced at the device adaptor comprise non-critical I/O requests.

10. The system of claim 8, wherein the reducing the rate at which I/O requests are processed comprises the device adaptor delaying processing of the I/O requests.

11. The system of claim 8, wherein the initiating the operation to reduce the rate at which I/O requests are processed at the device adaptor comprises:

sending, by the device adaptor, a message to the processor node to cause the processor node to reduce a rate at which requests are transferred from the processor node to the device adaptor.

12. The system of claim 11, wherein the operations further comprise:

additionally determining, by the device adaptor, whether processor utilization at the device adaptor exceeds the utilization threshold after sending the message to the processor node;

reducing, by the device adaptor, a rate at which the device adaptor processes I/O requests from the processor node in response to additionally determining that the processor utilization at the device adaptor exceeds the utilization threshold.

13. The system of claim 8, wherein the operations further comprise:

in response to determining that the remaining fault tolerance is a zero fault tolerance that does not permit at least one storage device to fail and allow recovery of data, sending, by the device adaptor, a second message to the processor node to cause the processor node to initiate an emergency protocol to terminate a mission critical operation when the processor node is performing the mission critical operation.

14. The system of claim 8, wherein the operations further comprise:

determining whether the processor node is performing a mission critical operation in response to the rebuild of data at the storage devices in the storage array, wherein the determining whether the remaining fault tolerance at the storage array comprises the non-zero fault tolerance, determining whether the processor utilization exceeds the utilization threshold, and initiating the operation to reduce the rate at which I/O requests are processed are performed in response to determining that the processor node is performing the mission critical operation.

15. A method for managing I/O requests to a storage array of storage devices in a machine having a processor node and device adaptor, comprising:

in response to the device adaptor initiating a rebuild of data at the storage devices in the storage array, determining whether a remaining fault tolerance at the storage array comprises a non-zero fault tolerance that permits at least one further storage device of the storage devices to fail and still allow recovery of data stored in the storage array; and

determining, by the device adaptor, whether processor utilization at the device adaptor exceeds a utilization threshold after determining that the remaining fault tolerance is not a zero fault tolerance;

initiating, by the device adaptor, an operation to reduce a rate at which I/O requests to the storage array are processed at the device adaptor in response to determining that the processor utilization at the device adaptor exceeds the utilization threshold.

16. The method of claim 15, wherein the I/O requests whose rate of processing is reduced at the device adaptor comprise non-critical I/O requests.

17. The method of claim 15, wherein the reducing the rate at which I/O requests are processed comprises the device adaptor delaying processing of the I/O requests.

18. The method of claim 15, wherein the initiating the operation to reduce the rate at which I/O requests are processed at the device adaptor comprises:

sending, by the device adaptor, a message to the processor node to cause the processor node to reduce a rate at which requests are transferred from the processor node to the device adaptor.

19. The method of claim 18, further comprising:

additionally determining, by the device adaptor, whether processor utilization at the device adaptor exceeds the utilization threshold after sending the message to the processor node;

reducing, by the device adaptor, a rate at which the device adaptor processes I/O requests from the processor node in response to additionally determining that the processor utilization at the device adaptor exceeds the utilization threshold.

20. The method of claim 15, further comprising:

in response to determining that the remaining fault tolerance is a zero fault tolerance that does not permit at least one storage device to fail and allow recovery of data, sending, by the device adaptor, a second message to the processor node to cause the processor node to initiate an emergency protocol to terminate a mission critical operation when the processor node is performing the mission critical operation.