Assigning prioritized rebuild resources optimally

• US 10,127,112 B2
• Filed: 07/20/2016
• Issued: 11/13/2018
• Est. Priority Date: 07/20/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Baptist, Andrew D., Dhuse, Greg R., Resch, Jason K., Wozniak, Ethan S.
• Primary Examiner(s): Kerveros, James C
• Application Number:
  US15/214,872
• Publication Number:
  US 20180024885A1
• Time in Patent Office: 846 Days
• Field of Search: 714773
• CPC Class Codes: G06F 11/108 Parity data distribution in semiconductor storages, e.g. in SSD
• CPC Class Codes: G06F 11/1088 Reconstruction on already foreseen single or plurality of spare disks
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0665 at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD

Abstract

A method begins by determining to rebuild one or more encoded data slices to a dispersed storage network (DSN) memory unit. The method continues by determining a rebuild rate of the DSN memory unit. The method continues by determining, based on the rebuild rate, a rebuild rate status of the DSN memory unit. The method continues by when the rebuild rate status is a high rebuild rate status, reducing the rebuild rate to the DSN memory unit. The method continues by rebuilding, when the rebuild rate is not zero, the one or more encoded data slices in the DSN memory unit.

Claims

1. A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN), the method comprises:

determining, by a plurality of rebuild modules, to rebuild one or more encoded data slices that are stored, or to be stored, in a DSN memory unit of a set of DSN memory units, wherein a data segment of data is dispersed storage error encoded to produce a set of encoded data slices that is stored in the set of DSN memory units, wherein the set of encoded data slices includes at least one of the one or more encoded data slices, and wherein a decode threshold number of encoded data slices of the set of encoded data slices is required to reconstruct the data segment;

determining, by at least one of the plurality of rebuild modules, a rebuild rate of the DSN memory unit, wherein determining the rebuild rate includes determining a number of the plurality of rebuild modules concurrently rebuilding encoded data slices for the DSN memory unit and their respective rebuild rates;

determining, based on the rebuild rate, a rebuild rate status of a plurality of rebuild rate statuses of the DSN memory unit, wherein a high rebuild status of the plurality of rebuild rate statuses indicates that a collected traffic rate to network capacity rate of the DSN memory unit is above a first threshold and a low rebuild status of the plurality of rebuild rate statuses indicates that the collected traffic rate to network capacity rate of the DSN memory unit is below a second threshold, wherein the collected traffic rate includes at least the rebuild rate and a regular access traffic rate, and wherein the first threshold is greater than the second threshold; and

when the rebuild rate status is the high rebuild status, reducing the rebuild rate to the DSN memory unit such that the collected traffic rate to network capacity rate is below the first threshold, the reducing including one or more of;

reducing the number of the plurality of rebuild modules concurrently rebuilding encoded data slices to the DSN memory unit or reducing rebuild rates of one or more of the plurality of rebuild modules; and

rebuilding, when the rebuild rate is not reduced to zero, the one or more encoded data slices to produce one or more rebuilt encoded data slices; and

sending, the one or more rebuilt encoded data slices to the DSN memory unit for storage.

2. The method of claim 1 further comprising, when the rebuild rate status is the low rebuild rate status, increasing the rebuild rate to the DSN memory unit such that the collected traffic rate to network capacity rate is above the second threshold and below the first threshold.

3. The method of claim 1 further comprising, when the rebuild rate status is the high rebuild rate status, reducing the rebuild rate to zero and delaying the rebuilding of the one or more encoded data slices to the DSN memory unit.

4. The method of claim 1, wherein determining to rebuild one or more encoded data slices to the DSN memory unit is based on a prioritized rebuild queue.

5. The method of claim 4, wherein the prioritized rebuild queue is based on one or more risk factors that determine an order in which the DSN memory units are to be rebuilt.

6. The method of claim 5, wherein the one or more risk factors include at least one of:

hardware class of a site of each of the DSN memory units, a dispersed storage (DS) unit of the DSN memory units, a memory device where the encoded data slice(s) will be rebuilt within of each of the DSN memory units, age of a site of the DSN memory units, age of the DS unit, age of a memory device where the one or more encoded data slices will be rebuilt within the DSN memory units, whether a site DS unit of each of the DSN memory units is subject to frequent power outages, whether the DSN memory units are subject to frequent network connectivity outages, adverse weather conditions, natural disaster predictions, or instability.

7. The method of claim 1, wherein the rebuild rate is determined passively.

8. The method of claim 1, wherein the rebuild rate is determined by requesting data directly from the DSN memory unit.

9. The method of claim 1, wherein the DSN memory unit is a dispersed storage (DS) unit.

10. The method of claim 1, wherein the DSN memory unit is a memory device.

11. A computing device of a group of computing devices of a dispersed storage network (DSN), the computing device comprises:

an interface;

a local memory; and

a processing module operably coupled to the interface and the local memory, wherein the processing module functions to;

determine, by a plurality of rebuild modules, to rebuild one or more encoded data slices that are stored, or to be stored, in a DSN memory unit of a set of DSN memory units, wherein a data segment of data is dispersed storage error encoded to produce a set of encoded data slices that is stored in the set of DSN memory units, wherein the set of encoded data slices includes at least one of the one or more encoded data slices, and wherein a decode threshold number of encoded data slices of the set of encoded data slices is required to reconstruct the data segment;

determine, by at least one of the plurality of rebuild modules, a rebuild rate of the DSN memory unit, wherein the rebuild rate includes determining a number of the plurality of rebuild modules concurrently rebuilding encoded data slices for the DSN memory unit and their respective rebuild rates;

determine, based on the rebuild rate, a rebuild rate status of a plurality of rebuild rate statuses of the DSN memory unit, wherein a high rebuild status of the plurality of rebuild rate statuses indicates that an collected traffic rate to network capacity rate of the DSN memory unit is above a first threshold and a low rebuild status of the plurality of rebuild rate statuses indicates that the collected traffic rate to network capacity rate of the DSN memory unit is below a second threshold, wherein the collected traffic rate includes at least the rebuild rate and a regular access traffic rate, and wherein the first threshold is greater than the second threshold; and

when the rebuild rate status is the high rebuild status, reduce the rebuild rate to the DSN memory unit such that the collected traffic rate to network capacity rate is below the first threshold, the reduce the rebuild rate including one or more of;

reducing the number of the plurality of rebuild modules concurrently rebuilding encoded data slices to the DSN memory unit or reducing rebuild rates of one or more of the plurality of rebuild modules; and

rebuild, when the rebuild rate is not reduced to zero, the one or more encoded data slices to produce one or more rebuilt encoded data slices; and

send, via the interface, the one or more rebuilt encoded data slices to the DSN memory unit for storage.

12. The computing device of claim 11 further comprising the processing module functions, when the rebuild rate status is the low rebuild rate status, to increase the rebuild rate to the DSN memory unit such that the collected traffic rate to network capacity rate is above the second threshold and below the first threshold.

13. The computing device of claim 11 further comprising the processing module functions, when the rebuild rate status is the high rebuild rate status, to reduce the rebuild rate to zero and delaying the rebuilding of the one or more encoded data slices to the DSN memory unit.

14. The computing device of claim 11, wherein the determining to rebuild one or more encoded data slices to the DSN memory unit is based on a prioritized rebuild queue.

15. The computing device of claim 14, wherein the prioritized rebuild queue is based on one or more risk factors that determine an order in which the DSN memory units are to be rebuilt.

16. The computing device of claim 15, wherein the one or more risk factors include at least one of:

hardware class of a site of each of the DSN memory units, a dispersed storage (DS) unit of the DSN memory units, a memory device where the encoded data slice(s) will be rebuilt within of each of the DSN memory units, age of a site of the DSN memory units, age of the DS unit, age of a memory device where the one or more encoded data slices will be rebuilt within the DSN memory units, whether a site or DS unit of each of the DSN memory units is subject to frequent power outages, whether the DSN memory units are subject to frequent network connectivity outages, adverse weather conditions, natural disaster predictions, or instability.

17. The computing device of claim 11, wherein the rebuild rate is determined by requesting data directly from the DSN memory unit.

18. The computing device of claim 11, wherein the rebuild rate is determined passively.