Storing an encoded data slice as a set of sub-slices

• US 9,489,264 B2
• Filed: 07/10/2014
• Issued: 11/08/2016
• Est. Priority Date: 11/25/2009
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Gladwin, S. Christopher
• Primary Examiner(s): BRYAN, JASON B
• Application Number:
  US14/327,615
• Publication Number:
  US 20140325264A1
• Time in Patent Office: 852 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 11/1096 Parity calculation or recalculation after configuration or reconfiguration of the system
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G11B 5/09 Digital recording

Abstract

A method begins by storage units of a dispersed storage network (DSN) receiving a set of encoded data slices for storage. The method continues with a first storage unit determining whether to store a first encoded data slice as a first set of encoded data sub-slices. The method continues with the first storage unit encoding the first encoded data slice in accordance with a first dispersed storage error encoding function to produce the first set of encoded data sub-slices. The method continues by a second storage unit of the storage units determining whether to store a second encoded data slice as a second set of encoded data sub-slices. The method continues with the second storage unit encoding the second encoded data slice in accordance with a second dispersed storage error encoding function to produce the second set of encoded data sub-slices.

Claims

1. A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN), the method comprises:

receiving, by storage units of the DSN, a set of encoded data slices for storage, wherein a data segment of data is dispersed storage error encoded into the set of encoded data slices;

determining, by a first storage unit of the storage units, whether to store a first encoded data slice of the set of encoded data slices as the first encoded data slice or as a first set of encoded data sub-slices;

when the first encoded data slice is to be stored as the first set of encoded data sub-slices, encoding, by the first storage unit, the first encoded data slice in accordance with a first dispersed storage error encoding function to produce the first set of encoded data sub-slices;

determining, by a second storage unit of the storage units, whether to store a second encoded data slice of the set of encoded data slices as the second encoded data slice or as a second set of encoded data sub-slices; and

when the second encoded data slice is to be stored as the second set of encoded data sub-slices, encoding, by the second storage unit, the second encoded data slice in accordance with a second dispersed storage error encoding function to produce the second set of encoded data sub-slices.

2. The method of claim 1 further comprises:

identifying, by the first storage unit, a first set of memory devices within the first storage unit;

storing, by the first storage unit, the first set of encoded data sub-slices in the first set of memory devices;

identifying, by the second storage unit, a second set of memory devices within the second storage unit; and

storing, by the second storage unit, the second set of encoded data sub-slices in the second set of memory devices.

3. The method of claim 1 further comprises:

determining, by the first storage unit, to store an encoded data sub-slice of the first set of encoded data sub-slices in a memory device of another storage unit of the storage units, wherein the other storage unit is the second storage unit or a different one of the storage units;

sending, by the first storage unit, the encoded data sub-slice to the other storage unit; and

storing, by the first storage unit, remaining encoded data sub-slices of the first set of encoded data sub-slices in a set of memory devices within the first storage unit.

4. The method of claim 3 further comprises:

selecting the other storage unit based on one or more selection criteria from a list of selection criteria, wherein the list of selection criteria includes;

a query and response with the other storage unit;

a pre-established relationship;

an ad hoc established relationship; and

a determination that the other storage unit has a sufficient amount of available memory.

5. The method of claim 3 further comprises:

determining to the store the encoded data sub-slice in the other storage unit based on one or more determination criteria from a list of determination criteria, wherein the list of determination criteria includes;

determining that the first storage unit has less than a desired amount of available memory;

determining that the first storage unit has more than a desired number of memory device with an age issue; and

determining that the other storage unit has more available storage than the first storage unit.

6. The method of claim 1 further comprises:

determining, by the first storage unit, a desired redundancy level; and

when the desired redundancy level is based on assurances against memory device failure with the first storage unit, determining, by the first storage unit, the first dispersed storage error encoding function based on the desired redundancy level, wherein the first dispersed storage error encoding function includes a pillar width number of n, a decode threshold number of k, and where n−

k is less than n/2.

7. The method of claim 1 further comprises:

determining, by the first storage unit, a desired redundancy level;

when the desired redundancy level is based on assurances against failure of the first storage unit, determining, by the first storage unit, the first dispersed storage error encoding function based on the desired redundancy level, wherein the first dispersed storage error encoding function includes a pillar width number of n, a decode threshold number of k, and where n−

k is equal to or greater than n/2; and

sending, by the first storage unit, at least n/2 encoded data sub-slices of the first set of encoded data sub-slices to another storage unit, wherein the other storage unit is the second storage unit or a different one of the storage units.

8. A method for execution by one or more processing modules of a storage unit of a dispersed storage network (DSN), the method comprises:

receiving an encoded data slice of a set of encoded data slices for storage, wherein a data segment of data is dispersed storage error encoded into the set of encoded data slices;

determining whether to store the encoded data slice as the encoded data slice or as a set of encoded data sub-slices;

when the encoded data slice is to be stored as the set of encoded data sub-slices, encoding the encoded data slice in accordance with a dispersed storage error encoding function to produce the set of encoded data sub-slices;

determining whether to store an encoded data sub-slice of the set of encoded data sub-slices in a memory device of another storage unit;

when the encoded data sub-slice is to be stored in the memory device of the other storage unit, sending the encoded data sub-slice to the other storage unit;

determining a desired redundancy level; and

when the desired redundancy level is based on assurances against memory device failure with the storage unit, determining the dispersed storage error encoding function based on the desired redundancy level, wherein the dispersed storage error encoding function includes a pillar width number of n, a decode threshold number of k, and where n−

k is less than n/2.

9. The method of claim 8 further comprises:

when the encoded data slice is to be stored as the encoded data slice, selecting a memory device of the storage unit; and

storing the encoded data slice in the selected memory device.

10. The method of claim 8 further comprises:

storing remaining encoded data sub-slices of the set of encoded data sub-slices in a set of memory devices of the storage unit.

11. The method of claim 8 further comprises:

when the encoded data sub-slice is not to be stored in the memory device of the other storage unit, storing the set of encoded data sub-slices in a set of memory devices of the storage unit.

12. A non-transitory computer readable storage medium comprises:

at least one memory section that stores operational instructions that, when executed by one or more processing modules of one or more computing devices of a dispersed storage network (DSN), causes the one or more computing devices to;

receive, by storage units of the DSN, a set of encoded data slices for storage, wherein a data segment of data is dispersed storage error encoded into the set of encoded data slices;

determine, by a first storage unit of the storage units, whether to store a first encoded data slice of the set of encoded data slices as the first encoded data slice or as a first set of encoded data sub-slices;

when the first encoded data slice is to be stored as the first set of encoded data sub-slices, encode, by the first storage unit, the first encoded data slice in accordance with a first dispersed storage error encoding function to produce the first set of encoded data sub-slices;

determine, by a second storage unit of the storage units, whether to store a second encoded data slice of the set of encoded data slices as the second encoded data slice or as a second set of encoded data sub-slices; and

when the second encoded data slice is to be stored as the second set of encoded data sub-slices, encode, by the second storage unit, the second encoded data slice in accordance with a second dispersed storage error encoding function to produce the second set of encoded data sub-slices.

13. The non-transitory computer readable storage medium of claim 12 further comprises:

the at least one memory section stores further operational instructs that, when executed by the one or more processing modules, causes the one or more computing devices of the DSN to;

identify, by the first storage unit, a first set of memory devices within the first storage unit;

store, by the first storage unit, the first set of encoded data sub-slices in the first set of memory devices;

identify, by the second storage unit, a second set of memory devices within the second storage unit; and

store, by the second storage unit, the second set of encoded data sub-slices in the second set of memory devices.

14. The non-transitory computer readable storage medium of claim 12 further comprises:

the at least one memory section stores further operational instructs that, when executed by the one or more processing modules, causes the one or more computing devices of the DSN to;

determine, by the first storage unit, to store an encoded data sub-slice of the first set of encoded data sub-slices in a memory device of another storage unit of the storage units, wherein the other storage unit is the second storage unit or a different one of the storage units;

send, by the first storage unit, the encoded data sub-slice to the other storage unit; and

store, by the first storage unit, remaining encoded data sub-slices of the first set of encoded data sub-slices in a set of memory devices within the first storage unit.

15. The non-transitory computer readable storage medium of claim 14 further comprises:

the at least one memory section stores further operational instructs that, when executed by the one or more processing modules, causes the one or more computing devices of the DSN to;

select the other storage unit based on one or more selection criteria from a list of selection criteria, wherein the list of selection criteria includes;

a query and response with the other storage unit;

a pre-established relationship;

an ad hoc established relationship; and

a determination that the other storage unit has a sufficient amount of available memory.

16. The non-transitory computer readable storage medium of claim 14 further comprises:

the at least one memory section stores further operational instructs that, when executed by the one or more processing modules, causes the one or more computing devices of the DSN to;

determine to the store the encoded data sub-slice in the other storage unit based on one or more determination criteria from a list of determination criteria, wherein the list of determination criteria includes;

determining that the first storage unit has less than a desired amount of available memory;

determining that the first storage unit has more than a desired number of memory device with an age issue; and

determining that the other storage unit has more available storage than the first storage unit.

17. The non-transitory computer readable storage medium of claim 12 further comprises:

the at least one memory section stores further operational instructs that, when executed by the one or more processing modules, causes the one or more computing devices of the DSN to;

determine, by the first storage unit, a desired redundancy level; and

when the desired redundancy level is based on assurances against memory device failure with the first storage unit, determine, by the first storage unit, the first dispersed storage error encoding function based on the desired redundancy level, wherein the first dispersed storage error encoding function includes a pillar width number of n, a decode threshold number of k, and where n−

k is less than n/2.

18. The non-transitory computer readable storage medium of claim 12 further comprises:

the at least one memory section stores further operational instructions that, when executed by the one or more processing modules, causes the one or more computing devices of the DSN to;

determine, by the first storage unit, a desired redundancy level;

when the desired redundancy level is based on assurances against failure of the first storage unit, determine, by the first storage unit, the first dispersed storage error encoding function based on the desired redundancy level, wherein the first dispersed storage error encoding function includes a pillar width number of n, a decode threshold number of k, and where n−

k is equal to or greater than n/2; and

send, by the first storage unit, at least n/2 encoded data sub-slices of the first set of encoded data sub-slices to another storage unit, wherein the other storage unit is the second storage unit or a different one of the storage units.

19. A storage unit of a dispersed storage network (DSN), the storage unit comprises:

an interface; and

a processing module operably coupled to the interface, wherein the processing module is operable to;

receive an encoded data slice of a set of encoded data slices for storage, wherein a data segment of data is dispersed storage error encoded into the set of encoded data slices;

determine whether to store the encoded data slice as the encoded data slice or as a set of encoded data sub-slices;

when the encoded data slice is to be stored as the set of encoded data sub-slices, encode the encoded data slice in accordance with a dispersed storage error encoding function to produce the set of encoded data sub-slices;

determine whether to store an encoded data sub-slice of the set of encoded data sub-slices in a memory device of another storage unit;

when the encoded data sub-slice is to be stored in the memory device of the other storage unit, send the encoded data sub-slice to the other storage unit;

determine a desired redundancy level; and

when the desired redundancy level is based on assurances against memory device failure with the storage unit, determine the dispersed storage error encoding function based on the desired redundancy level, wherein the dispersed storage error encoding function includes a pillar width number of n, a decode threshold number of k, and where n−

k is less than n/2.

20. The storage unit of claim 19 further comprises:

the processing module is further operable to;

when the encoded data slice is to be stored as the encoded data slice, select a memory device of the storage unit; and

store the encoded data slice in the selected memory device.

21. The storage unit of claim 19 further comprises:

the processing module is further operable to;

store remaining encoded data sub-slices of the set of encoded data sub-slices in a set of memory devices of the storage unit.

22. The storage unit of claim 19 further comprises:

the processing module is further operable to;

when the encoded data sub-slice is not to be stored in the memory device of the other storage unit, store the set of encoded data sub-slices in a set of memory devices of the storage unit.