Prioritized repair of data storage failures

• US 9,075,773 B1
• Filed: 05/07/2014
• Issued: 07/07/2015
• Est. Priority Date: 05/07/2014
• Status: Active Grant

• Current Assignee: Rubrik, Inc.
• Original Assignee: Igneous Systems, Inc. (Rubrik, Inc.)
• Inventors: Rakitzis, Triantaphyllos Byron, Lemar, Eric Michael, Hughes, Jeffrey Douglas, Bhageshpur, Kiran V.
• Primary Examiner(s): GUYTON, PHILIP A
• Application Number:
  US14/272,303
• Time in Patent Office: 426 Days
• Field of Search: 714/6.31, 714/6.1, 714/6.24, 714/6.22
• US Class Current: 1/1
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/073 in a memory management context, e.g. virtual memory or cache management memory management G06F12/00; testing of static memory units G11C29/00
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/1088 Reconstruction on already foreseen single or plurality of spare disks
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 11/2069 Management of state, configuration or failover
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/3419 by assessing time
• CPC Class Codes: G11C 29/70 Masking faults in memories by using spares or by reconfiguring

Abstract

Embodiments are directed towards managing data storage that may experience a data failure. If a repair event is associated with a data storage failure, a new repair task may be generated and added to a task list. A priority value for each repair task in the task list may be determined based in part on the mean-time-to-data-loss (MTTDL) value associated with each repair task in the task list such that a lower MTTDL may indicate a higher priority value over a lower MTTDL. One or more repair tasks may be promoted to become active repair tasks based on the priority value the repair tasks such that the promoted repair tasks have a higher priority that than other repair tasks in the task list, if any. Each active repair task may be executed to repair one or more associated the storage failures.

Claims

1. A method for managing data storage over a network using a network computer that executes instructions that perform actions, comprising:

if at least one repair event is associated with at least one new storage failure on a storage unit or a repair symbol unit, generating at least one new repair task that is associated with the at least one new storage failure, wherein the at least one new repair task is added to a task list that includes at least one other repair task for at least one current storage failure;

determining a priority value for each repair task in the task list based in part on a comparison of at least one characteristic value that is associated with each repair task in the task list, wherein a lower characteristic value generates a higher priority value over a higher characteristic value;

determining a resource budget based on a network bandwidth capacity for at least one different portion of the network, wherein the resource budget includes separate values for each of the different portions of the network;

promoting at least one repair task to be at least one new active repair task if the priority value for the at least one promoted repair task is higher than each other repair task and enough of the resource budget is available to execute the at least one new active repair task when each current working task is executing;

promoting at least one repair task to be the at least one new active repair task if a priority value for the at least one promoted task is higher than the priority level of at least one current active repair task, wherein each current active repair task is demoted to a repair task that has a lower priority value than the priority value for the at least one new active repair task until enough of the resource budget is released to execute the at least one new active repair task;

promoting the at least one repair task to the at least one new active repair task if a remainder of the resource budget enables execution of the at least one new active repair task when the priority value of the at least one new active repair task is lower than at least one other active repair task requiring other than the remainder to execute; and

executing each active repair task to repair at least one storage failure that is associated with the active repair task.

2. The method of claim 1, wherein determining the priority value for each repair task further comprises:

determining a duration for each repair task; and

determining the priority value based on the duration for each repair task, wherein repair tasks with longer durations are prioritized over repair tasks with shorter durations.

3. The method of claim 1, wherein determining the priority value for each repair task further comprises:

determining a protection level associated with each repair task; and

determining the priority value based on the protection level for each repair task, wherein repair tasks with a lower protection level are prioritized over repair tasks associated with higher protection levels.

4. The method of claim 1, wherein determining the priority value for each repair task further comprises:

determining a duration for each repair task; and

determining the priority value based on the duration for each repair task, wherein repair tasks with shorter durations are prioritized over repair tasks with longer durations.

5. The method of claim 1, further comprising, if the at least one repair event indicates a completion of the at least one active repair task, removing the corresponding at least one repair task from the task list and re-prioritizing each repair task that remains in the task list.

6. The method of claim 1, wherein executing the at least one active repair task, further comprises, repairing at least one portion of a data stripe associated with the at least one storage failure based on erasure coding.

7. The method of claim 1, further comprising, determining the resource budget based on at least one of, compute processing, network bandwidth, local bandwidth, or storage device throughput.

8. A system that is arranged for managing data storage over a network, comprising:

a network computer comprising;

a transceiver that is operative to communicate over the network;

a memory that is operative to store at least instructions; and

a processor device that is operative to execute instructions that enable actions, including;

if at least one repair event is associated with at least one new storage failure on a storage unit or a repair symbol unit, generating at least one new repair task that is associated with the at least one new storage failure, wherein the at least one new repair task is added to a task list that includes at least one other repair task for at least one current storage failure;

determining a priority value for each repair task in the task list based in part on a comparison of at least one characteristic value that is associated with each repair task in the task list, wherein a lower characteristic value generates a higher priority value over a higher characteristic value;

determining a resource budget based on a network bandwidth capacity for at least one different portion of the network, wherein the resource budget includes separate values for each of the different portions of the network;

promoting at least one repair task to be at least one new active repair task if the priority value for the at least one promoted repair task is higher than each other repair task and enough of the resource budget is available to execute the at least one new active repair task when each current active repair task is executing;

promoting at least one repair task to be the at least one new active repair task if a priority value for the at least one promoted task is higher than the priority level of at least one current active repair task, wherein each current active repair task is demoted to a repair task that has a lower priority value than the priority value for the at least one new active repair task until enough of the resource budget is released to execute the at least one new active repair task;

promoting the at least one repair task to the at least one new active repair task if a remainder of the resource budget enables execution of the at least one new active repair task when the priority value of the at least one new active repair task is lower than at least one other active repair task requiring other than the remainder to execute; and

executing each active repair task to repair at least one storage failure that is associated with the active repair task; and

a client computer, comprising;

a transceiver that is operative to communicate over the network;

a memory that is operative to store at least instructions; and

a processor device that is operative to execute instructions that enable actions, including;

providing configuration information to the network computer.

9. The system of claim 8, wherein determining the priority value for each repair task further comprises:

determining a duration for each repair task; and

determining the priority value based on the duration for each repair task, wherein repair tasks with longer durations are prioritized over repair tasks with shorter durations.

10. The system of claim 8, wherein determining the priority value for each repair task further comprises:

determining a protection level associated with each repair task; and

determining the priority value based on the protection level for each repair task, wherein repair tasks with a lower protection level are prioritized over repair tasks associated with higher protection levels.

11. The system of claim 8, wherein determining the priority value for each repair task further comprises:

determining a duration for each repair task; and

determining the priority value based on the duration for each repair task, wherein repair tasks with shorter durations are prioritized over repair tasks with longer durations.

12. The system of claim 8, wherein the network computer processor device is operative to enable actions, further comprising, if the at least one repair event indicates a completion of the at least one active repair task, removing the corresponding at least one repair task from the task list and re-prioritizing each repair task that remains in the task list.

13. The system of claim 8, wherein executing the at least one active repair task, further comprises, repairing at least one portion of a data stripe associated with the at least one storage failure based on erasure coding.

14. The system of claim 8, wherein the network computer processor device is operative to enable actions, further comprising, determining the resource budget based on at least one of, compute processing, network bandwidth, local bandwidth, or storage device throughput.

15. A processor readable non-transitory storage media that includes instructions for managing data storage over a network, wherein execution of the instructions by a processor device enables actions, comprising:

if at least one repair event is associated with at least one new storage failure on a storage unit or a repair symbol unit, generating at least one new repair task that is associated with the at least one new storage failure, wherein the at least one new repair task is added to a task list that includes at least one other repair task for at least one current storage failure;

determining a priority value for each repair task in the task list based in part on a comparison of at least one characteristic value that is associated with each repair task in the task list, wherein a lower characteristic value generates a higher priority value over a higher characteristic value;

determining the resource budget based on a network bandwidth capacity for at least one different portion of the network, wherein the resource budget includes separate values for each of the different portions of the network;

promoting at least one repair task to be at least one new active repair task if the priority value for the at least one promoted repair task is higher than each other repair task and enough of the resource budget is available to execute the at least one new active repair task when each current active repair task is executing;

promoting at least one repair task to be the at least one new active repair task if a priority value for the at least one promoted task is higher than the priority level of at least one current active repair task, wherein each current active repair task is demoted to a repair task that has a lower priority value than the priority value for the at least one new active repair task until enough of the resource budget is released to execute the at least one new active repair task;

promoting the at least one repair task to the at least one new active repair task if a remainder of the resource budget enables execution of the at least one new active repair task when the priority value of the at least one new active repair task is lower than at least one other active repair task requiring other than the remainder to execute; and

executing each active repair task to repair at least one storage failure that is associated with the active repair task.

16. The media of claim 15, wherein determining the priority value for each repair task further comprises:

determining a duration for each repair task; and

determining the priority value based on the duration for each repair task, wherein repair tasks with longer durations are prioritized over repair tasks with shorter durations.

17. The media of claim 15, wherein determining the priority value for each repair task further comprises:

determining a protection level associated with each repair task; and

determining the priority value based on the protection level for each repair task, wherein repair tasks with a lower protection level are prioritized over repair tasks associated with higher protection levels.

18. The media of claim 15, wherein determining the priority value for each repair task further comprises:

determining a duration for each repair task; and

determining the priority value based on the duration for each repair task, wherein repair tasks with shorter durations are prioritized over repair tasks with longer durations.

19. The media of claim 15, further comprising, if the at least one repair event indicates a completion of the at least one active repair task, removing the corresponding at least one repair task from the task list and re-prioritizing each repair task that remains in the task list.

20. The media of claim 15, wherein executing the at least one active repair task, further comprises, repairing at least one portion of a data stripe associated with the at least one storage failure based on erasure coding.

21. A network computer that is operative for managing data storage over a network, comprising:

a transceiver that is operative to communicate over a network;

a memory that is operative to store at least instructions; and

a processor device that is operative to execute instructions that enable actions, including;

if at least one repair event is associated with at least one new storage failure on a storage unit or a repair symbol unit, generating at least one new repair task that is associated with the at least one new storage failure, wherein the at least one new repair task is added to a task list that includes at least one other repair task for at least one current storage failure;

determining a priority value for each repair task in the task list based in part on a comparison of at least one characteristic value that is associated with each repair task in the task list, wherein a lower characteristic value generates a higher priority value over a higher characteristic value;

determining a resource budget based on a network bandwidth capacity for at least one different portion of the network, wherein the resource budget includes separate values for each of the different portions of the network;

promoting at least one repair task to be at least one new active repair task if the priority value for the at least one promoted repair task is higher than each other repair task and enough of the resource budget is available to execute the at least one new active repair task when each current active repair task is executing;

promoting at least one repair task to be the at least one new active repair task if a priority value for the at least one promoted task is higher than the priority level of at least one current active repair task, wherein each current active repair task is demoted to a repair task that has a lower priority value than the priority value for the at least one new active repair task until enough of the resource budget is released to execute the at least one new active repair task;

promoting the at least one repair task to the at least one new active repair task if a remainder of the resource budget enables execution of the at least one new active repair task when the priority value of the at least one new active repair task is lower than at least one other active repair task requiring other than the remainder to execute; and

executing each active repair task to repair at least one storage failure that is associated with the active repair task.

22. The network computer of claim 21, wherein determining the priority value for each repair task further comprises:

determining a duration for each repair task; and

determining the priority value based on the duration for each repair task, wherein repair tasks with longer durations are prioritized over repair tasks with shorter durations.

23. The network computer of claim 21, wherein determining the priority value for each repair task further comprises:

determining a protection level associated with each repair task; and

determining the priority value based on the protection level for each repair task, wherein repair tasks with a lower protection level are prioritized over repair tasks associated with higher protection levels.

24. The network computer of claim 21, wherein determining the priority value for each repair task further comprises:

determining a duration for each repair task; and

determining the priority value based on the duration for each repair task, wherein repair tasks with shorter durations are prioritized over repair tasks with longer durations.

25. The network computer of claim 21, wherein the network computer processor device is operative to enable actions, further comprising, if the at least one repair event indicates a completion of the at least one active repair task, removing the corresponding at least one repair task from the task list and re-prioritizing each repair task that remains in the task list.

26. The network computer of claim 21, wherein executing the at least one active repair task, further comprises, repairing at least one portion of a data stripe associated with the at least one storage failure based on erasure coding.