System reliability by prioritizing recovery of objects

• US 10,275,302 B2
• Filed: 12/18/2015
• Issued: 04/30/2019
• Est. Priority Date: 12/18/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: D'Amato, Andrea, Kuznetsov, Vyacheslav, Shankar, Vinod, Mehra, Karan
• Primary Examiner(s): Guyton, Philip
• Application Number:
  US14/974,910
• Publication Number:
  US 20170177427A1
• Time in Patent Office: 1,229 Days
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0754 by exceeding limits
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/1088 Reconstruction on already foreseen single or plurality of spare disks

Abstract

Recovery requests are scheduled and prioritized according to priority valuations of the minimum time to next failure relative to the minimum time to next recovery for corresponding storage data sets. The prioritization can be performed by an upper layer recovery scheduler that dispatches requests to different storage units and/or by lower layers and individual storage unit schedulers. Prioritizations can be reflected in tagging on recovery requests, as determined by a first entity, and/or determined dynamically at the point of dispatch and processing.

Claims

1. A computing system comprising:

one or more processors;

one or more storage devices having stored computer-executable instructions which are executable by the one or more processors to cause the computing system to at least instantiate or manage a recovery scheduler;

a plurality of storage units storing one or more copies of each of a first data set and a second data set;

one or more data structures that include a minimum time to failure for the first data set and the second data set across the plurality of storage units, wherein the minimum time to failure for each respective data set is based, at least in part, on determining a total number of available next failure states for the respective data set, across the plurality of storage units, before it is in a total failure state; and

the recovery scheduler, wherein the recovery scheduler is configured to at least;

obtain a first recovery request corresponding to the first data set and a second recovery request corresponding to the second data set, each of the first recovery request and the second recovery request including an attached prioritization tag indicating an initial relative priority for executing the respective request;

obtain, from the one or more data structures, the minimum time to failure for each of the first data set and the second data set;

based on the obtained minimum time to failure for both the first data set and the second data set, schedule an ordering for dispatching the first and second recovery both the initial relative priority for the respective first and second recovery request and the obtained minimum time to failure received for the first data set and the second data set; and

dispatch the first and second recovery requests to the one or more storage units based on the priority valuation.

2. The computing system of claim 1, wherein the recovery scheduler is further configured to schedule the ordering for dispatching the first or second recovery requests to the one or more storage units in such a way as to reduce the minimum time to recovery relative to the determined minimum time to next failure for the first or second data sets.

3. The computing system of claim 2, wherein the first and second data sets each comprise one or more data sets stored on the one or more storage units.

4. The computing system of claim 2, wherein the minimum time to next failure for the first or second data sets corresponds directly to a minimum time to next failure for the one or more storage units storing the first or second data sets.

5. The computing system of claim 4, wherein the one or more storage units comprises one or more of a hardware storage drive, a storage rack, or a hardware storage array.

6. The computing system of claim 4, wherein the one or more storage units comprises a virtual storage drive or virtual network storage array.

7. The computing system of claim 1, wherein the recovery scheduler is further configured to schedule the ordering for dispatching the first or second recovery requests to the one or more storage units by querying processing bandwidth of the one or more storage units.

8. The computing system of claim 1, wherein the recovery scheduler is further configured to schedule the ordering for dispatching the first or second recovery requests to the one or more storage units by generating processing priority information for at least the first or second recovery requests.

9. The computing system of claim 8, wherein the recovery scheduler is further configured to dispatch the one or more recovery requests by attaching the processing priority information to the one or more recovery requests with one or more tags or flags that reflect the processing priority information.

10. The computing system of claim 1, wherein the minimum to failure for the first data set is also included within the attached prioritization tags included with the first recovery request and the second recovery request.

11. The computing system of claim 1, wherein the priority valuation is further based on determining, from the one or more data structures, which of the first data set or the second data set has experienced a greater percentage of degradation relative to a starting position.

12. The computing system of claim 1, wherein, subsequent to dispatching the first and second recovery requests, the one or more data structures are updated to reflect the result of the respective recovery requests.

13. A method, executed at one or more processors of a computing system, for scheduling processing of recovery requests at a storage unit, the method comprising:

receiving, at a storage unit scheduler, a first recovery request corresponding to a first data set and a second recovery request corresponding to a second data set, each of the first data set and second data set being stored within one or more of a plurality of storage units, and each of the first recovery request and the second recovery request including an attached prioritization tag indicating an initial relative priority for executing the respective request;

obtaining, from within the attached prioritization tag a minimum time to failure for the first data set and the second data set across the plurality of storage units, wherein the minimum time to failure for each respective data set is based, at least in part on determining a total number of available next failure states for the respective data set, across the plurality of storage units, before the respective data set is in a total failure state;

generating, at the storage unit scheduler, an ordered sequence for processing the first and second recovery requests by the storage unit based on the initial relative priority and the obtained minimum time to failure for the first data set and the second data set obtained from the attached prioritization tag; and

causing, at the storage unit scheduler, the storage unit to process the first and second recovery requests based on the priority valuation.

14. The method of claim 13, wherein processing the first and second recovery requests in the ordered sequence includes causing a new copy of the first or second data sets to be written to the storage unit.

15. The method of claim 13, wherein processing the first and second recovery requests in the ordered sequence includes causing the first or second data sets to read from the storage unit to be copied to at least one of a different location in the storage unit or to a different storage unit.

16. The method of claim 13, wherein processing the first and second recovery requests in the ordered sequence includes causing a restoration of the first or second data sets from parity symbols contained within at least one of a different location in the storage unit or a different storage unit.

17. A computing system comprising:

one or more processors;

one or more storage devices having stored computer-executable instructions which are executable by the one or more processors to cause the computing system to at least instantiate or manage a recovery scheduler and a storage unit scheduler;

a plurality of storage units storing a plurality of data sets;

one or more data structures that include a minimum time to failure for the first data set and the second data set across the plurality of storage units, wherein the minimum time to failure for each respective data set is based, at least in part, on determining a total number of available next failure states for the respective data set, across the plurality of storage units, before it is in a critical failure state; and

the recovery scheduler that is configured to at least;

obtain a first recovery request corresponding to the first data set of the plurality of data sets and a second recovery request corresponding to the second data set of the plurality of data sets, each of the first recovery request and the second recovery request including a prioritization tag indicating an initial relative priority for executing the respective request;

determine, using the one or more data structures, that the first recovery request corresponds to a critical recovery request and that the second recovery requests corresponds to a non-critical recovery request;

schedule an ordering for dispatching the first and second recovery requests to the one or more storage units based on a priority valuation that places the critical recovery request ahead of the non-critical recovery request; and

dispatch the first and second recovery requests to the one or more storage units based on the priority valuation.