Prioritized repair of data storage failures

• US 9,305,666 B2
• Filed: 07/06/2015
• Issued: 04/05/2016
• Est. Priority Date: 05/07/2014
• Status: Active Grant

• Current Assignee: Rubrik, Inc.
• Original Assignee: Igneous Systems, Inc. (Rubrik, Inc.)
• Inventors: Rakitzis, Triantaphyllos Byron, Lemar, Eric Michael, Hughes, Jeffrey Douglas, Bhageshpur, Kiran V.
• Primary Examiner(s): GUYTON, PHILIP A
• Application Number:
  US14/792,539
• Publication Number:
  US 20150325315A1
• Time in Patent Office: 274 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/073 in a memory management context, e.g. virtual memory or cache management memory management G06F12/00; testing of static memory units G11C29/00
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/1088 Reconstruction on already foreseen single or plurality of spare disks
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 11/2069 Management of state, configuration or failover
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/3419 by assessing time
• CPC Class Codes: G11C 29/70 Masking faults in memories by using spares or by reconfiguring

Abstract

Embodiments are directed towards managing data storage that may experience a data failure. If a repair event is associated with a data storage failure, a new repair task may be generated and added to a task list. A priority value for each repair task in the task list may be determined based in part on the mean-time-to-data-loss (MTTDL) value associated with each repair task in the task list such that a lower MTTDL may indicate a higher priority value over a lower MTTDL. One or more repair tasks may be promoted to become active repair tasks based on the priority value the repair tasks such that the promoted repair tasks have a higher priority that than other repair tasks in the task list, if any. Each active repair task may be executed to repair one or more associated the storage failures.

Claims

1. A method for managing data storage over a network using a network computer that executes instructions that perform actions, comprising:

when one or more repair events are associated with one or more new storage failures on a storage unit or a repair symbol unit, generating one or more new repair tasks that are associated with the one or more new storage failures;

determining a resource budget based on a network bandwidth capacity for one or more different portions of the network, wherein the resource budget includes separate values for each of the different portions of the network;

determining a protection level that represents a maximum number of storage unit or repair symbol unit failures before particular data is irrevocably lost on the data storage;

determining a risk of irrevocable loss for each repair task;

promoting the one or more repair tasks to be one or more new active repair tasks when a priority value for the one or more repair tasks is higher than a priority value for one or more active repair tasks and enough of the resource budget is available to execute the one or more new active repair tasks when each active repair task is executing; and

executing each active repair task to repair the one or more storage failures that are associated with the one or more active repair tasks.

2. The method of claim 1, further comprising determining each priority value for each repair task in a task list for at least one current storage failure based in part on a comparison of one or more characteristic values that are associated with each repair task in the task list, wherein a lower characteristic value generates a higher priority value over a higher characteristic value.

3. The method of claim 1, further comprising promoting the one or more repair tasks to be one or more promoted new active repair tasks when each priority value for the one or more promoted new active repair tasks is higher than each priority value for the one or more active repair tasks, wherein the one or more active repair tasks are demoted to a repair task that has a lower priority value than the priority value for the one or more promoted new active repair tasks until enough of the resource budget is released to execute the one or more promoted new active repair tasks.

4. The method of claim 1, further comprising promoting the one or more repair tasks to be one or more promoted new active repair tasks when a remainder of the resource budget enables execution of the one or more repair tasks when the priority value of the one or more repair tasks is lower than the priority value of the one or more active repair tasks requiring other than the remainder to execute.

5. The method of claim 1, wherein determining the priority value for each repair task further comprises:

determining a duration for each repair task;

determining the priority value based on the duration for each repair task; and

prioritizing repair tasks based on either longer durations being prioritized over repair tasks with shorter durations or shorter durations being prioritized over repair tasks with longer durations.

6. The method of claim 1, wherein the resource budget is dynamically updated based on one or more of, compute processing, network bandwidth, local bandwidth, non-repair related storage task, higher priority file read, storage device throughput, type of erasure encoding, or type of actions to perform each repair task.

7. The method of claim 1, wherein executing the at least one active repair task, further comprises, employing one or more of a storage management application, a service or another application to repair at least one portion of a data stripe associated with the at least one storage failure based on erasure coding.

8. A system that is arranged for managing data storage over a network, comprising:

a network computer comprising;

a transceiver that is operative to communicate over the network;

a memory that is operative to store at least instructions; and

a processor device that is operative to execute instructions that enable actions, including;

when one or more repair events are associated with one or more new storage failures on a storage unit or a repair symbol unit, generating one or more new repair tasks that are associated with the one or more new storage failures;

determining a resource budget based on a network bandwidth capacity for one or more different portions of the network, wherein the resource budget includes separate values for each of the different portions of the network;

determining a protection level that represents a maximum number of storage unit or repair symbol unit failures before particular data is irrevocably lost on the data storage;

determining a risk of irrevocable loss for each repair task;

promoting the one or more repair tasks to be one or more new active repair tasks when a priority value for the one or more repair tasks is higher than a priority value for one or more active repair tasks and enough of the resource budget is available to execute the one or more new active repair tasks when each active repair task is executing; and

executing each active repair task to repair the one or more storage failures that are associated with the one or more active repair tasks; and

a client computer, comprising;

a transceiver that is operative to communicate over the network;

a memory that is operative to store at least instructions; and

a processor device that is operative to execute instructions that enable actions, including;

providing configuration information to the network computer.

9. The system of claim 8, further comprising determining each priority value for each repair task in a task list for at least one current storage failure based in part on a comparison of one or more characteristic values that are associated with each repair task in the task list, wherein a lower characteristic value generates a higher priority value over a higher characteristic value.

10. The system of claim 8, further comprising promoting the one or more repair tasks to be one or more promoted new active repair tasks when each priority value for the one or more promoted new active repair tasks is higher than each priority value for the one or more active repair tasks, wherein the one or more active repair tasks are demoted to a repair task that has a lower priority value than the priority value for the one or more promoted new active repair tasks until enough of the resource budget is released to execute the one or more promoted new active repair tasks.

11. The system of claim 8, further comprising promoting the one or more repair tasks to be one or more promoted new active repair tasks when a remainder of the resource budget enables execution of the one or more repair tasks when the priority value of the one or more repair tasks is lower than the priority value of the one or more active repair tasks requiring other than the remainder to execute.

12. The system of claim 8, wherein determining the priority value for each repair task further comprises:

determining a duration for each repair task;

determining the priority value based on the duration for each repair task; and

prioritizing repair tasks based on either longer durations being prioritized over repair tasks with shorter durations or shorter durations being prioritized over repair tasks with longer durations.

13. The system of claim 8, wherein the resource budget is dynamically updated based on one or more of, compute processing, network bandwidth, local bandwidth, non-repair related storage task, higher priority file read, storage device throughput, type of erasure encoding, or type of actions to perform each repair task.

14. The system of claim 8, wherein executing the at least one active repair task, further comprises, employing one or more of a storage management application, a service or another application to repair at least one portion of a data stripe associated with the at least one storage failure based on erasure coding.

15. A processor readable non-transitory storage media that includes instructions for managing data storage over a network, wherein execution of the instructions by a processor device enables actions, comprising:

when one or more repair events are associated with one or more new storage failures on a storage unit or a repair symbol unit, generating one or more new repair tasks that are associated with the one or more new storage failures;

determining a resource budget based on a network bandwidth capacity for one or more different portions of the network, wherein the resource budget includes separate values for each of the different portions of the network;

determining a protection level that represents a maximum number of storage unit or repair symbol unit failures before particular data is irrevocably lost on the data storage;

determining a risk of irrevocable loss for each repair task;

promoting the one or more repair tasks to be one or more new active repair tasks when a priority value for the one or more repair tasks is higher than a priority value for one or more active repair tasks and enough of the resource budget is available to execute the one or more new active repair tasks when each active repair task is executing; and

executing each active repair task to repair the one or more storage failures that are associated with the one or more active repair tasks.

16. The media of claim 15, further comprising determining each priority value for each repair task in a task list for at least one current storage failure based in part on a comparison of one or more characteristic values that are associated with each repair task in the task list, wherein a lower characteristic value generates a higher priority value over a higher characteristic value.

17. The media of claim 15, further comprising promoting the one or more repair tasks to be one or more promoted new active repair tasks when each priority value for the one or more promoted new active repair tasks is higher than each priority value for the one or more active repair tasks, wherein the one or more active repair tasks are demoted to a repair task that has a lower priority value than the priority value for the one or more promoted new active repair tasks until enough of the resource budget is released to execute the one or more promoted new active repair tasks.

18. The media of claim 15, further comprising promoting the one or more repair tasks to be one or more promoted new active repair tasks when a remainder of the resource budget enables execution of the one or more repair tasks when the priority value of the one or more repair tasks is lower than the priority value of the one or more active repair tasks requiring other than the remainder to execute.

19. The media of claim 15, wherein determining the priority value for each repair task further comprises:

determining a duration for each repair task;

determining the priority value based on the duration for each repair task; and

prioritizing repair tasks based on either longer durations being prioritized over repair tasks with shorter durations or shorter durations being prioritized over repair tasks with longer durations.

20. The media of claim 15, wherein executing the at least one active repair task, further comprises, employing one or more of a storage management application, a service or another application to repair at least one portion of a data stripe associated with the at least one storage failure based on erasure coding.

21. A network computer that is operative for managing data storage over a network, comprising:

a transceiver that is operative to communicate over a network;

a memory that is operative to store at least instructions; and

a processor device that is operative to execute instructions that enable actions, including;

when one or more repair events are associated with one or more new storage failures on a storage unit or a repair symbol unit, generating one or more new repair tasks that are associated with the one or more new storage failures;

determining a resource budget based on a network bandwidth capacity for one or more different portions of the network, wherein the resource budget includes separate values for each of the different portions of the network;

determining a protection level that represents a maximum number of storage unit or repair symbol unit failures before particular data is irrevocably lost on the data storage;

determining a risk of irrevocable loss for each repair task;

promoting the one or more repair tasks to be one or more new active repair tasks when a priority value for the one or more repair tasks is higher than a priority value for one or more active repair tasks and enough of the resource budget is available to execute the one or more new active repair tasks when each active repair task is executing; and

executing each active repair task to repair the one or more storage failures that are associated with the one or more active repair tasks.

22. The network computer of claim 21, further comprising determining each priority value for each repair task in a task list for at least one current storage failure based in part on a comparison of one or more characteristic values that are associated with each repair task in the task list, wherein a lower characteristic value generates a higher priority value over a higher characteristic value.

23. The network computer of claim 21, further comprising promoting the one or more repair tasks to be one or more promoted new active repair tasks when each priority value for the one or more promoted new active repair tasks is higher than each priority value for the one or more active repair tasks, wherein the one or more active repair tasks are demoted to a repair task that has a lower priority value than the priority value for the one or more promoted new active repair tasks until enough of the resource budget is released to execute the one or more promoted new active repair tasks.

24. The network computer of claim 21, further comprising promoting the one or more repair tasks to be one or more promoted new active repair tasks when a remainder of the resource budget enables execution of the one or more repair tasks when the priority value of the one or more repair tasks is lower than the priority value of the one or more active repair tasks requiring other than the remainder to execute.

25. The network computer of claim 21, wherein determining the priority value for each repair task further comprises:

determining a duration for each repair task;

determining the priority value based on the duration for each repair task; and

prioritizing repair tasks based on either longer durations being prioritized over repair tasks with shorter durations or shorter durations being prioritized over repair tasks with longer durations.

26. The network computer of claim 21, wherein the resource budget is dynamically updated based on one or more of, compute processing, network bandwidth, local bandwidth, non-repair related storage task, higher priority file read, storage device throughput, type of erasure encoding, or type of actions to perform each repair task.