Method and apparatus for nested dispersed storage

• US 20140281812A1
• Filed: 05/27/2014
• Published: 09/18/2014
• Est. Priority Date: 08/27/2009
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: Grube, Gary W., Markison, Timothy W.
• Granted Patent: US 8,977,931 B2
• Time in Patent Office: Days
• US Class Current: 714/764
• CPC Class Codes: G06F 11/1008 in individual solid state devices G06F11/1004 takes precedence
• CPC Class Codes: G06F 11/1088 Reconstruction on already foreseen single or plurality of spare disks
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 21/6272 by registering files or documents with a third party
• CPC Class Codes: H04L 63/126 the source of the received data

Abstract

A method begins by a DS processing module generating a plurality of encoded slices from a data segment using an error encoding function. The method continues with the DS processing module identifying a plurality of DS storage units for storing the plurality of encoded slices. The method continues with the DS processing module selecting an encoded slice of the plurality of encoded slices for sub-slicing using a sub-slicing encoding function to produce a selected encoded slice. The method continues with the DS processing module outputting the plurality of encoded slices to the plurality of DS storage units. The method continues with the DS processing module outputting a command to a DS storage unit of the plurality of DS storage units corresponding to the selected encoded slice, wherein the command includes an instruction to sub-slice the selected encoded slice.

Claims

1. A method for execution by a processing module in a distributed storage (DS) unit, the method comprising:

receiving a request to retrieve an error coded (EC) data slice from a user device, wherein data segment is dispersed error encoded to produce a plurality of EC data slices that includes the EC data slice;

determining whether the EC data slice is sub-sliced into a plurality of EC data sub-slices based on information stored within the DS unit;

when the EC data slice is determined to be sub-sliced;

identifying storage locations of the plurality of EC data sub-slices within other DS units;

transmitting retrieval commands for the plurality of EC data sub-slices to the other DS units;

receiving, in response to at least some of the retrieval commands, a minimum number of EC data sub-slices required to reconstruct the EC data slice; and

reconstructing the EC data slice from the minimum number of EC data sub-slices;

when the EC data slice is determined not to be sub-sliced, retrieving the EC data slice from local memory of the DS unit; and

outputting the EC data slice or the reconstructed EC data slice, via a network interface of the DS unit.

2. The method of claim 1 further comprising:

retrieving one EC data sub-slice of the plurality of EC data sub-slices from one of the identified storage locations within one DS unit of the other DS units, wherein the one EC data sub-slice is reconstructed from a minimum number of EC data sub-sub-slices.

3. The method of claim 1 further comprising:

when the EC data slice is determined to be sub-sliced, identifying the locations of the plurality of EC data sub-slices within the other DS units based on a virtual dispersed storage network (DSN) table maintained by the DSN unit.

4. The method of claim 1 further comprising:

authenticating the request to retrieve the EC data slice by verifying a user identification code (ID) associated with the request based on at least one of the information stored within memory of the DS unit and other information stored within another DS unit of the other DS units.

5. The method of claim 1, wherein the reconstructing the EC data slice from the minimum number of EC data sub-slices comprises:

processing the minimum number of EC data sub-slices using an forward error correction (FEC) decoding process that is an inverse process of an FEC encoding process that generates the plurality of EC data sub-slices based on the EC data slice.

6. A distributed storage (DS) unit comprises:

a network interface configured to communicate with other devices in a dispersed storage network (DSN);

a memory; and

a processing module configured to;

receive, via the network interface, a request to retrieve an error coded (EC) data slice from a user device, wherein data segment is dispersed error encoded to produce a plurality of EC data slices that includes the EC data slice;

determine whether the EC data slice is sub-sliced into a plurality of EC data sub-slices based on information stored within the memory;

when the EC data slice is determined to be sub-sliced;

identify storage locations of the plurality of EC data sub-slices within other DS units;

transmit retrieval commands for the plurality of EC data sub-slices to the other DS units;

receive, in response to at least some of the retrieval commands, a minimum number of EC data sub-slices required to reconstruct the EC data slice; and

reconstruct the EC data slice from the minimum number of EC data sub-slices;

when the EC data slice is determined not to be sub-sliced, retrieve the EC data slice from the memory of the DS unit; and

output the EC data slice or the reconstructed EC data slice, via the network interface.

7. The DS processing unit of claim 6, wherein the processing module is further configured to:

retrieve one EC data sub-slice of the plurality of EC data sub-slices from one of the identified storage locations within one DS unit of the other DS units, wherein the one EC data sub-slice is reconstructed from a minimum number of EC data sub-sub-slices.

8. The DS processing unit of claim 6, wherein the processing module is further configured to:

identify, when the EC data slice is determined to be sub-sliced, the locations of the plurality of EC data sub-slices within the other DS units based on a virtual dispersed storage network (DSN) table maintained by the DSN unit.

9. The DS processing unit of claim 6, wherein the processing module is further configured to:

authenticate the request to retrieve the EC data slice by verifying a user identification code (ID) associated with the request based on at least one of the information stored within the memory of the DS unit and other information stored within another DS unit of the other DS units.

10. The DS processing unit of claim 6, wherein the processing module is further configured to:

reconstruct the EC data slice from the minimum number of EC data sub-slices using an forward error correction (FEC) decoding process that is an inverse process of an FEC encoding process that generates the plurality of EC data sub-slices based on the EC data slice.

11. A method for execution by a processing module in a distributed storage (DS) unit, the method comprising:

receiving a request to retrieve an error coded (EC) data slice from a user device, wherein data segment is dispersed error encoded to produce a plurality of EC data slices that includes the EC data slice, and wherein the request indicates whether the EC data slice is sub-sliced into a plurality of EC data sub-slices;

when the EC data slice is sub-sliced as indicated by the request;

determining storage locations of the plurality of EC data sub-slices within other DS units;

transmitting retrieval commands for the plurality of EC data sub-slices to the other DS units;

receiving, in response to at least some of the retrieval commands, a minimum number of EC data sub-slices required to reconstruct the EC data slice; and

reconstructing the EC data slice from the minimum number of EC data sub-slices;

when the EC data slice is not sub-sliced as indicated by the request, retrieving the EC data slice from local memory of the DS unit; and

outputting the EC data slice or the reconstructed EC data slice via a network interface of the DS unit.

12. The method of claim 11 further comprising:

retrieving one EC data sub-slice of the plurality of EC data sub-slices from one of the identified storage locations within one DS unit of the other DS units, wherein the one EC data sub-slice is reconstructed from a minimum number of EC data sub-sub-slices.

13. The method of claim 11 further comprising:

when the EC data slice is determined to be sub-sliced as indicated by the request, identifying the locations of the plurality of EC data sub-slices within the other DS units based on a virtual dispersed storage network (DSN) table maintained by the DSN unit.

14. The method of claim 11 further comprising:

authenticating the request to retrieve the EC data slice by verifying a user identification code (ID) associated with the request based on at least one of the information stored within memory of the DS unit and other information stored within another DS unit of the other DS units.

15. The method of claim 11, wherein the reconstructing the EC data slice from the minimum number of EC data sub-slices comprises:

processing the minimum number of EC data sub-slices using an forward error correction (FEC) decoding process that is an inverse process of an FEC encoding process that generates the plurality of EC data sub-slices based on the EC data slice.

16. A distributed storage (DS) unit comprises:

a network interface configured to communicate with other devices in a dispersed storage network (DSN);

a memory; and

a processing module configured to;

receive, via the network interface, a request to retrieve an error coded (EC) data slice from a user device, wherein data segment is dispersed error encoded to produce a plurality of EC data slices that includes the EC data slice, and wherein the request indicates whether the EC data slice is sub-sliced into a plurality of EC data sub-slices;

when the EC data slice is sub-sliced as indicated by the request;

determine storage locations of the plurality of EC data sub-slices within other DS units;

transmit retrieval commands for the plurality of EC data sub-slices to the other DS units;

receive, in response to at least some of the retrieval commands, a minimum number of EC data sub-slices required to reconstruct the EC data slice; and

reconstruct the EC data slice from the minimum number of EC data sub-slices;

when the EC data slice is not sub-sliced as indicated by the request, retrieve the EC data slice from the memory of the DS unit; and

output the EC data slice or the reconstructed EC data slice via the network interface.

17. The DS processing unit of claim 16, wherein the processing module is further configured to:

retrieve one EC data sub-slice of the plurality of EC data sub-slices from one of the identified storage locations within one DS unit of the other DS units, wherein the one EC data sub-slice is reconstructed from a minimum number of EC data sub-sub-slices.

18. The DS processing unit of claim 16, wherein the processing module is further configured to:

identify, when the EC data slice is determined to be sub-sliced as indicated by the request, the locations of the plurality of EC data sub-slices within the other DS units based on a virtual dispersed storage network (DSN) table maintained by the DSN unit.

19. The DS processing unit of claim 16, wherein the processing module is further configured to:

authenticate the request to retrieve the EC data slice by verifying a user identification code (ID) associated with the request based on at least one of the information stored within the memory of the DS unit and other information stored within another DS unit of the other DS units.

20. The DS processing unit of claim 16, wherein the processing module is further configured to:

reconstruct the EC data slice from the minimum number of EC data sub-slices using an forward error correction (FEC) decoding process that is an inverse process of an FEC encoding process that generates the plurality of EC data sub-slices based on the EC data slice.