Handling storage unit failure in a dispersed storage network

• US 10,241,865 B2
• Filed: 02/15/2017
• Issued: 03/26/2019
• Est. Priority Date: 02/15/2017
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Baptist, Andrew D., Hendrickson, Dustin M., Leggette, Wesley B., Motwani, Manish, Ober, Brian F., Resch, Jason K.
• Primary Examiner(s): Guyton, Philip
• Application Number:
  US15/433,311
• Publication Number:
  US 20180232276A1
• Time in Patent Office: 769 Days
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 11/2053 where persistent mass storage functionality or persistent mass storage control functionality is redundant error detection or correction in information storage based on relative movement between record carrier and transducer G11B20/18
• CPC Class Codes: G06F 16/214 Database migration support
• CPC Class Codes: H03M 13/1515 Reed-Solomon codes
• CPC Class Codes: H03M 13/616 Matrix operations, especially for generator matrices or check matrices, e.g. column or row permutations
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06

Abstract

A method for execution by dispersed storage failure detection unit of a dispersed storage network includes generating a storage unit weights and storage set weights based on the storage unit weights. One storage unit that meets a weight-adjustment trigger condition is identified, and a new storage unit weight for the identified storage unit is generated. A first data migration within the storage set that includes the identified storage unit is facilitated based on the storage unit weights. A failing set of storage units in a second storage set is identified. A new storage set weight for the second storage set is generated when the number of storage units in the failing set compares favorably to a failure threshold number. A second data migration that includes transferring data slices stored in the second storage unit to other storage sets is facilitated based on the storage set weights.

Claims

1. A method for execution by a dispersed storage DS failure detection unit of a dispersed storage network (DSN) that includes a processor, the method comprises:

generating a plurality of storage unit weights corresponding to a plurality of storage units;

generating a plurality of storage set weights corresponding to a plurality of storage sets, wherein each of the plurality of storage sets includes a subset of the plurality of storage units, and wherein each of the plurality of storage set weights is based on the storage unit weights of the corresponding subset of storage units;

identifying one of the plurality of storage units that meets a weight-adjustment trigger condition, and generating a new storage unit weight for the identified one of the plurality of storage units in response;

facilitating a first data migration within a first one of the plurality of storage sets that includes the identified one of the plurality of storage units based on the plurality of storage unit weights, wherein the first data migration includes transferring data slices stored in the identified one of the plurality of storage units to at least one other storage unit in the first one of the plurality of storage sets;

identifying a failing set of storage units in a second one of the plurality of storage sets that meet the weight-adjustment trigger condition;

determining that a number of the identified storage units in the failing set compares favorably to a failure threshold number, and generating a new storage set weight for the second one of the plurality of storage sets in response; and

facilitating a second data migration based on the plurality of storage set weights, wherein the second data migration includes transferring data slices stored in the subset of storage units of the second one of the plurality of storage sets to storage units in at least one other storage set.

2. The method of claim 1, wherein generating the plurality of storage set weights includes at least one of:

calculating a sum of storage unit weights of the corresponding subset of storage units or calculating an average of storage unit weights of the corresponding subset of storage units.

3. The method of claim 1, wherein the plurality of storage unit weights of the storage units of a storage set are proportional to a storage capacity of each storage unit in the storage set.

4. The method of claim 1, further comprising:

updating a namespace of the DSN based on the new storage unit weight and the new storage set weight;

wherein a computing device of the DSN determines at least one destination storage unit for new data object in a write request based on the updated namespace.

5. The method of claim 1, further comprising:

generating a new storage set weight for the first one of the plurality of storage sets based on the new storage unit weight for the identified one of the plurality of storage units.

6. The method of claim 1, wherein the new storage unit weight for the identified one of the plurality of storage units is set to zero, wherein the first data migration includes transferring every data slice stored in the identified one of the plurality of storage units to at least one other storage unit in the first one of the plurality of storage sets, further comprising:

removing the identified one of the plurality of storage units from a registry of the DSN.

7. The method of claim 1, wherein the failure threshold number is based on at least one of:

a decode threshold, a read threshold, or a write threshold of an information dispersal algorithm associated with storage of data objects in the DSN.

8. The method of claim 1, further comprising:

identifying an active storage unit in the second one of the plurality of storage sets that compares unfavorably to the weight-adjustment trigger condition; and

reassigning the identified active storage unit to a different one of the plurality of storage sets.

9. The method of claim 8, further comprising:

generating a new storage set weight for the different one of the plurality of storage sets in response to the reassignment based on a storage unit weight corresponding to the active storage unit.

10. The method of claim 8, wherein every storage that is included in the subset of storage units of the second one of the plurality of storage sets and is not included in the failing set of storage units is reassigned to a different one of the plurality of storage sets.

11. A processing system of a dispersed storage (DS) failure detection unit comprises:

at least one processor;

a memory that stores operational instructions, that when executed by the at least one processor cause the processing system to;

generate a plurality of storage unit weights corresponding to a plurality of storage units;

generate a plurality of storage set weights corresponding to a plurality of storage sets, wherein each of the plurality of storage sets includes a subset of the plurality of storage units, and wherein each of the plurality of storage set weights is based on the storage unit weights of the corresponding subset of storage units;

identify one of the plurality of storage units that meets a weight-adjustment trigger condition, and generate a new storage unit weight for the identified one of the plurality of storage units in response;

facilitate a first data migration within a first one of the plurality of storage sets that includes the identified one of the plurality of storage units based on the plurality of storage unit weights, wherein the first data migration includes transferring data slices stored in the identified one of the plurality of storage units to at least one other storage unit in the first one of the plurality of storage sets;

identify a failing set of storage units in a second one of the plurality of storage sets that meet the weight-adjustment trigger condition;

determine that a number of the identified storage units in the failing set compares favorably to a failure threshold number, and generate a new storage set weight for the second one of the plurality of storage sets in response; and

facilitate a second data migration based on the plurality of storage set weights, wherein the second data migration includes transferring data slices stored in the subset of storage units of the second one of the plurality of storage sets to storage units in at least one other storage set.

12. The dispersed storage (DS) failure detection unit of claim 11, wherein generating the plurality of storage set weights includes at least one of:

calculating a sum of storage unit weights of the corresponding subset of storage units or calculating an average of storage unit weights of the corresponding subset of storage units.

13. The dispersed storage (DS) failure detection unit of claim 11, wherein the operational instructions, when executed by the at least one processor, further cause the processing system to:

update a namespace based on the new storage unit weight and the new storage set weight;

wherein a computing device determines at least one destination storage unit for new data object in a write request based on the updated namespace.

14. The dispersed storage (DS) failure detection unit of claim 11, wherein the operational instructions, when executed by the at least one processor, further cause the processing system to:

generate a new storage set weight for the first one of the plurality of storage sets based on the new storage unit weight for the identified one of the plurality of storage units.

15. The dispersed storage (DS) failure detection unit of claim 11, wherein the new storage unit weight for the identified one of the plurality of storage units is set to zero, wherein the first data migration includes transferring every data slice stored in the identified one of the plurality of storage units to at least one other storage unit in the first one of the plurality of storage sets, and wherein the operational instructions, when executed by the at least one processor, further cause the processing system to:

remove the identified one of the plurality of storage units from a registry.

16. The dispersed storage (DS) failure detection unit of claim 11, wherein the failure threshold number is based on at least one of:

a decode threshold, a read threshold, or a write threshold of an information dispersal algorithm associated with storage of data objects.

17. The dispersed storage (DS) failure detection unit of claim 11, wherein the operational instructions, when executed by the at least one processor, further cause the processing system to:

identify an active storage unit in the second one of the plurality of storage sets that compares unfavorably to the weight-adjustment trigger condition; and

reassign the identified active storage unit to a different one of the plurality of storage sets.

18. The dispersed storage (DS) failure detection unit of claim 17, wherein the operational instructions, when executed by the at least one processor, further cause the processing system to:

generate a new storage set weight for the different one of the plurality of storage sets in response to the reassignment based on a storage unit weight corresponding to the active storage unit.

19. The dispersed storage (DS) failure detection unit of claim 17, wherein every storage that is included in the subset of storage units of the second one of the plurality of storage sets and is not included in the failing set of storage units is reassigned to a different one of the plurality of storage sets.

20. A non-transitory computer readable storage medium comprises:

at least one memory section that stores operational instructions that, when executed by a processing system of a dispersed storage network (DSN) that includes a processor and a memory, causes the processing system to;

generate a plurality of storage unit weights corresponding to a plurality of storage units;

generate a plurality of storage set weights corresponding to a plurality of storage sets, wherein each of the plurality of storage sets includes a subset of the plurality of storage units, and wherein each of the plurality of storage set weights is based on the storage unit weights of the corresponding subset of storage units;

identify one of the plurality of storage units that meets a weight-adjustment trigger condition, and generate a new storage unit weight for the identified one of the plurality of storage units in response;

facilitate a first data migration within a first one of the plurality of storage sets that includes the identified one of the plurality of storage units based on the plurality of storage unit weights, wherein the first data migration includes transferring data slices stored in the identified one of the plurality of storage units to at least one other storage unit in the first one of the plurality of storage sets;

identify a failing set of storage units in a second one of the plurality of storage sets that meet the weight-adjustment trigger condition;

determine that a number of the identified storage units in the failing set compares favorably to a failure threshold number, and generate a new storage set weight for the second one of the plurality of storage sets in response; and

facilitate a second data migration based on the plurality of storage set weights, wherein the second data migration includes transferring data slices stored in the subset of storage units of the second one of the plurality of storage sets to storage units in at least one other storage set.