Clustered disk error recovery

• US 10,698,779 B2
• Filed: 11/13/2017
• Issued: 06/30/2020
• Est. Priority Date: 11/13/2017
• Status: Active Grant

• Current Assignee: Infinidat Ltd.
• Original Assignee: Infinidat Ltd.
• Inventors: Dorfman, Mike, Loffe, Nick, Grabois, Sagi
• Primary Examiner(s): Patel, Jigar P
• Application Number:
  US15/810,174
• Publication Number:
  US 20190146890A1
• Time in Patent Office: 960 Days
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/1658 Data re-synchronization of a redundant component, or initial sync of replacement, additional or spare unit
• CPC Class Codes: G06F 11/2089 Redundant storage control functionality
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0617 in relation to availability
• CPC Class Codes: G06F 3/0635 by changing the path, e.g. traffic rerouting, path reconfiguration
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD

Abstract

A method for disk error recovery management, the method may include (A) executing by a first control node, a first disk error recovery process that involves halting access, of the first control node, to a first group of storage devices. (B) Receiving by the first control node, during the execution of the disk error recovery process, a request to access a given storage device of the first group. (C) Directing the request to access the given storage device to a second control node if the second control node is not currently involved in an error recovery processes that involves halting access to the given storage device. (D) Accessing the given storage device by the second control node, during the execution of the disk error recovery process.

Claims

1. A method for disk error recovery management, the method comprises:

executing by a first control node of a storage system, a first disk error recovery process;

wherein the first disk error recovery process involves halting access, of the first control node, to a first group of storage devices;

receiving by the first control node, during the execution of the first disk error recovery process, a request to access a given storage device of the first group;

directing, by the first control node, the request to access the given storage device to a second control node of the storage system if the second control node is not currently involved in an error recovery process that involves halting access to the given storage device; and

accessing the given storage device by the second control node, during the execution of the first disk error recovery process.

2. The method according to claim 1, wherein the storage devices comprise disks.

3. The method according to claim 1, comprising denying the request to access the given storage device if the second control node is currently involved in the error recovery process that involves halting access to the given storage device.

4. The method according to claim 1, comprising relaying the request to access the given storage device to a third control node if (i) the third control node is not currently involved in the error recovery process that involves halting access to the given storage device, and (ii) if the second control node is currently involved in the error recovery process that involves halting access to the given storage device.

5. The method according to claim 1, comprising:

receiving, during the execution of the first disk error recovery process, a request to perform, by a second control node of the storage system, a second disk error recovery process that involves halting access, of the second control node, to at least one storage device of the first group; and

denying the request to perform the second disk error recovery process, at least until a completion of the first disk error recovery process.

6. The method according to claim 1 comprising:

requesting by the first control node, from an arbiter module of the storage system, a permission to perform the first disk error recovery process; and

executing the first disk error recovery process, when the arbiter module grants the permission to perform the first disk error recovery process.

7. The method according to claim 1 comprising:

requesting by the first control node, from an arbiter module of the storage system a first permission to perform the first disk error recovery process;

requesting by the second control node, from the arbiter module, a second permission to perform a second disk error recovery process;

wherein the second permission is requested concurrently with the first permission; and

granting, by the arbiter module, only the first permission.

8. A method for disk error recovery management, the method comprises:

executing by a first control node of a storage system, a first disk error recovery process;

wherein the first disk error recovery process involves halting access, of the first control node, to a first group of storage devices;

receiving a request to perform, by a second control node of the storage system, a second disk error recovery process that involves halting access, of the second control node, to at least one storage device of the first group; and

denying the request to perform the second disk error recovery process, at least until a completion of the first disk error recovery process.

9. The method according to claim 8 wherein the denying of the request is executed by an arbiter of the storage system.

10. The method according to claim 9 wherein the arbiter is coupled to the first and second control node but does not belong to either one of the first and second control nodes.

11. The method according to claim 9 wherein the arbiter belongs to at least one of the first and second control nodes.

12. The method according to claim 8, comprising:

receiving by the first control node, during the execution of the first disk error recovery process, a request to access a given storage device of the first group;

directing the request to access the given storage device to the second control node of the storage system; and

accessing the given storage device by the second control node, during the execution of the first disk error recovery process.

13. The method according to claim 8 comprising:

requesting by the first control node, from an arbiter of the storage system and prior to the execution of the first disk error recovery process, a permission to perform the first disk error recovery process; and

granting by the arbiter module the permission to perform the first disk error recovery process.

14. The method according to claim 13, wherein the receiving and the denying are performed by the arbiter;

wherein an execution, in parallel, of the first and second disk error recovery processes will cause the first and second control nodes to concurrently halt access to a same storage device;

wherein the method further comprising;

preventing the concurrent halt access to the same storage device by granting only the permission to perform the first disk error recovery process.

15. A computer program product that stores instructions that once executed by a storage system causes the storage system to:

execute by a first control node of a storage system, a first disk error recovery process;

wherein the first disk error recovery process involves halting access, of the first control node, to a first group of storage devices;

receive by the first control node, during the execution of the first disk error recovery process, a request to access a given storage device of the first group;

direct the request to access the given storage device to a second control node of the storage system if the second control node is not currently involved in an error recovery process that involves halting access to the given storage device; and

access the given storage device by the second control node, during the execution of the first disk error recovery process.

16. The computer program product according to claim 15, that stores instructions for denying the request to access the given storage device if the second control node is currently involved in the error recovery process that involves halting access to the given storage device.

17. The computer program product according to claim 15, that stores instructions for relaying the request to access the given storage device to a third control node if (i) the third control node is not currently involved in the error recovery process that involves halting access to the given storage device, and (ii) if the second control node is currently involved in the error recovery process that involves halting access to the given storage device.

18. The computer program product according to claim 15, that stores instructions for:

requesting by the first control node, from an arbiter module of the storage system, a permission to perform the first disk error recovery process; and

executing the first disk error recovery process, when the arbiter module grants the permission to perform the first disk error recovery process.

19. The computer program product according to claim 15, that stores instructions for:

requesting by the first control node, from an arbiter module of the storage system, a first permission to perform the first disk error recovery process;

requesting by the second control node, from the arbiter module, a second permission to perform a second disk error recovery process;

wherein the second permission is requested concurrently with the first permission; and

granting, by the arbiter module, only the first permission.