AUTOMATED STALLED PROCESS DETECTION AND RECOVERY

• US 20170147457A1
• Filed: 02/06/2017
• Published: 05/25/2017
• Est. Priority Date: 04/30/2015
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Iljazi, Ilir, Papadopoulos, Gregory A., Abhijeet, Kumar, Baptist, Andrew D., Resch, Jason K.
• Granted Patent: US 10,216,594 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 11/2053 where persistent mass storage functionality or persistent mass storage control functionality is redundant error detection or correction in information storage based on relative movement between record carrier and transducer G11B20/18
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: G06F 2201/82 Solving problems relating to consistency ensuring consistency in mirrored systems G06F11/2064
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G11C 29/48 Arrangements in static stores specially adapted for testing by means external to the store, e.g. using direct memory access [DMA] or using auxiliary access paths

Abstract

A dispersed storage network (DSN) includes multiple storage units. A processing unit included in the DSN detects a failing storage unit, and issues an error indicator to a recovery unit. The error indicator indicates the failing storage unit. To verify that the storage unit is failing, the recovery unit issues a test request to the failing storage unit. The recovery unit determines to implement a corrective action for recovery units that are determined to be actually failing, and facilitates implementing the corrective action.

Claims

1. A method for use in a dispersed storage network (DSN) including a plurality of storage units, the method comprising:

detecting, by a processing unit included in the DSN, a failing storage unit;

issuing, by the processing unit, an error indicator to a recovery unit, to indicate the failing storage unit;

issuing, by the recovery unit, a test request to the failing storage unit;

determining, by the recovery unit, to implement a corrective action; and

facilitating, by the recovery unit, execution of the corrective action.

2. The method of claim 1, wherein detecting a failing storage unit includes:

receiving an unfavorable response to an access request.

3. The method of claim 1, wherein detecting includes:

determining that no response to an access request directed to the failing storage unit is received within a predetermined amount of time.

4. The method of claim 1, wherein the error indicator includes:

information identifying the failing storage unit and a failure type.

5. The method of claim 4, wherein issuing a test request includes:

generating a particular test request based on the failure type; and

sending the particular test request to the failing storage unit.

6. The method of claim 1, wherein determining to implement a corrective action includes:

determining at least one of receiving an unfavorable response to the test request, or determining that a time period has expired without receiving a test response.

7. The method of claim 1, wherein facilitating execution of the corrective action includes:

issuing a corrective action command to at least one of the failing storage unit or a managing unit.

8. The method of claim 1, further comprising:

facilitating, by the recovery unit, execution of a further corrective action in response to receiving an unfavorable response to the corrective action.

9. A dispersed storage network (DSN) including a plurality of storage units, the DSN comprising:

processing unit configured to;

detect a failing storage unit;

issue an error indicator to a recovery unit, the error indicator indicate the failing storage unit;

a recovery unit configured to;

issue a test request to the failing storage unit;

determine to implement a corrective action; and

facilitate execution of the corrective action.

10. The dispersed storage network (DSN) of claim 9, wherein the processing unit is configured to detect a failing storage unit, at least in part, by:

receiving, at the processing unit, an unfavorable response to an access request.

11. The dispersed storage network (DSN) of claim 9, wherein the processing unit is configured to detect a failing storage unit, at least in part, by:

determining, at the processing unit, that no response to an access request directed to the failing storage unit is received within a predetermined amount of time.

12. The dispersed storage network (DSN) of claim 9, wherein the error indicator includes:

information identifying the failing storage unit and a failure type.

13. The dispersed storage network (DSN) of claim 12, wherein the recovery unit is configured to issue a test request, at least in part, by:

generating a particular test request based on the failure type; and

sending the particular test request to the failing storage unit.

14. The dispersed storage network (DSN)of claim 9, wherein the recovery unit is configured to issue determine to implement a corrective action, at least in part, by:

determining at least one of receiving an unfavorable response to the test request, or determining that a time period has expired without receiving a test response.

15. The dispersed storage network (DSN)of claim 9, wherein the recovery unit is configured to facilitate execution of the corrective action, at least in part, by:

issuing a corrective action command to at least one of the failing storage unit or a managing unit.

16. The dispersed storage network (DSN) of claim 9, wherein the recovery unit is further configured to:

facilitate execution of a further corrective action in response to receiving an unfavorable response to the corrective action.

17. A recovery unit included in a dispersed storage network (DSN) including a plurality of storage units, the recovery unit comprising:

a processor;

a memory coupled to the processor;

a non-transitory program of instructions configured to be stored in the memory and executed by the processor, the program of instructions including;

at least one instruction to receive, from a processing unit, an error indicator indicating a failing storage unit;

at least one instruction to verify that the error indicator by issuing a test request to the failing storage unit;

at least one instruction to determine to implement a corrective action associated with the failing storage unit; and

at least one instruction to facilitate execution of the corrective action.

18. The recovery unit of claim 17, wherein the at least one instruction to verify includes:

at least one instruction to generate a particular test request based on a failure type associated with the failing storage unit; and

at least one instruction to send the particular test request to the failing storage unit.

19. The recovery unit of claim 17, wherein the at least one instruction to facilitate execution of the corrective action includes:

issuing a corrective action command to at least one of the failing storage unit or a managing unit.

20. The recovery unit of claim 17, wherein the program of instructions further comprises:

the at least one instruction to facilitate execution of a further corrective action in response to receiving an unfavorable response to the corrective action.