Partitioning data for storage in a dispersed storage network

• US 9,483,398 B2
• Filed: 11/07/2011
• Issued: 11/01/2016
• Est. Priority Date: 11/29/2010
• Status: Expired due to Fees

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Grube, Gary W., Markison, Timothy W.
• Primary Examiner(s): GIARDINO JR, MARK A
• Application Number:
  US13/291,030
• Publication Number:
  US 20120137095A1
• Time in Patent Office: 1,821 Days
• Field of Search: 711/5, 711/171, 707/652, 709/231
• US Class Current: 1/1
• CPC Class Codes: G06F 12/0638 Combination of memories, e.g. ROM and RAM such as to permit replacement or supplementing of words in one module by words in another module address formation of the next microinstruction G06F9/26; masking faults in memories by using spares or by reconfiguring G11C29/70
• CPC Class Codes: G06F 21/6218 to a system of files or objects, e.g. local or distributed file system or database
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0646 Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

A method begins by a processing module receiving data for storage and determining whether to partition the data in accordance with a data partitioning dispersed storage scheme. When the data is to be partitioned, the method continues with the processing module partitioning the data into a local data portion and a remaining data portion in accordance with the data partitioning dispersed storage scheme, dispersed storage encoding the local data portion to produce a plurality of local encoded data elements in accordance with dispersed storage encoding parameters, sending the plurality of local encoded data elements to an associated dispersed storage network (DSN) memory for storage therein, and sending the remaining data portion to another DS module.

Claims

1. A method for execution by a dispersed storage (DS) module, the method comprises:

receiving data for storage as determined by a first DS module;

determining whether to partition the data in accordance with a first data partitioning dispersed storage scheme by the first DS module;

when the data is not to be partitioned by the first DS module, process a data segment of the data to produce first manipulated data and send the first manipulated data for storage in a first set of storage units;

when the data is to be partitioned by the first DS module;

partitioning the data into a local data portion and a remaining data portion in accordance with the data partitioning dispersed storage scheme;

dispersed storage error encoding, in accordance with dispersed storage encoding parameters, a data segment of the local data portion to produce a plurality of encoded data elements in accordance with dispersed storage encoding parameters of the data partitioning dispersed storage scheme;

sending, in accordance with the dispersed storage encoding parameters, the plurality of encoded data elements to the first set of storage units, via a first set of network communication paths, for storage therein;

identifying a second DS module; and

sending the remaining data portion to the second DS module via a first network communication path;

receiving the remaining data portion for storage as determined by the second DS module;

determining whether to partition the remaining data portion in accordance with a second data partitioning dispersed storage scheme by the second DS module; and

when the remaining data portion is not to be partitioned by the second DS module, process a data segment of the remaining data portion to produce second manipulated data and send the second manipulated data for storage in a second set of storage units;

when the remaining data portion is to be partitioned by the second DS module;

partitioning the remaining data portion into another local data portion and another remaining data portion in accordance with the second data partitioning dispersed storage scheme;

dispersed storage error encoding, in accordance with other dispersed storage encoding parameters, another data segment of the another local data portion to produce another plurality of encoded data elements in accordance with the other dispersed storage encoding parameters of the data partitioning dispersed storage scheme;

sending, in accordance with the dispersed storage encoding parameters, the plurality of another encoded data elements to the second set of storage units, via a second set of network communication paths, for storage therein;

identifying a third DS module; and

sending the another remaining data portion to the third DS module via a second network communication path.

2. The method of claim 1, wherein receiving the data comprises:

receiving the data from a fourth DS module; and

receiving the data partitioning dispersed storage scheme from the fourth DS module.

3. The method of claim 1, wherein the determining whether to partition the data comprises:

interpreting data size of the data;

determining whether the data size is of a level that requires partitioning in accordance with the data partitioning dispersed storage scheme; and

when the data size is of a level that requires partitioning, indicating partitioning of the data.

4. The method of claim 1, wherein the partitioning comprises:

selecting a portion of the data per a data portion size indication of the data partitioning dispersed storage scheme to produce the local data portion.

5. The method of claim 1, wherein the partitioning comprises:

selecting the local data portion to have a partial overlap of data with the remaining data portion in accordance with the data partitioning dispersed storage scheme.

6. The method of claim 1, wherein the dispersed storage error encoding comprises one or more of:

dispersed storage error encoding;

slicing the local data portion into a set of slices and encrypting the set of slices;

slicing the local data portion into a set of slices; and

replicating the local data portion.

7. The method of claim 1, wherein the sending the remaining data portion to the second DS module comprises:

selecting the second DS module based at least one of;

the data partitioning dispersed storage scheme, a lookup, a query, or a message.

8. The method of claim 1 further comprises:

updating a location table to indicate that the plurality of encoded data elements is stored in the first set of storage units and that the another plurality of encoded data elements is stored in the second set of storage units.

9. A non-transitory computer readable memory device comprises:

a first memory section associated with a first computing device that stores operational instructions that, when executed by the first computing device, causes the first computing device to, determine whether to partition data in accordance with a data partitioning dispersed storage scheme, wherein the data is received for storage as determined by the first computing device;

a second memory section associated with the first computing device that stores operational instructions that, when executed by the first computing device, causes the first computing device to,

when the data is not to be partitioned, process a data segment of the data to produce first manipulated data and send the first manipulated data for storage in a first set of storage units;

when the data is to be partitioned;

partition the data into a local data portion and a remaining data portion in accordance with the data partitioning dispersed storage scheme;

dispersed storage error encode, in accordance with dispersed storage encoding parameters, a data segment of the local data portion to produce a plurality of encoded data elements in accordance with dispersed storage encoding parameters of the data partitioning dispersed storage scheme;

send, in accordance with the dispersed storage encoding parameters, the plurality of encoded data elements to the first set of storage units, via a first set of network communication paths, for storage therein; and

identify a second computing device; and

send the remaining data portion to the second computing device via a first network communication path;

a third memory section associated with the second computing device that stores operational instructions that, when executed by the second computing device, causes the second computing device to, determine whether to partition the remaining data portion in accordance with another data partitioning dispersed storage scheme, wherein the remaining data portion is received for storage as determined by the second computing device;

a fourth memory section associated with the second computing device that stores operational instructions that, when executed by the second computing device, causes the second computing device to,

when the remaining data portion is not to be partitioned, process a data segment of the remaining data portion to produce second manipulated data and send the second manipulated data for storage in a second set of storage units;

when the remaining data portion is to be partitioned;

partition the remaining data portion into another local data portion and another remaining data portion in accordance with the another data partitioning dispersed storage scheme;

dispersed storage error encode, in accordance with other dispersed storage encoding parameters, a data segment of the another local data portion to produce another plurality of encoded data elements in accordance with the other dispersed storage encoding parameters of the another data partitioning dispersed storage scheme;

send, in accordance with the other dispersed storage encoding parameters, the another plurality of encoded data elements to the second set of storage units, via a second set of network communication paths, for storage therein;

identify a third computing device; and

send the another remaining data portion to the third computing device via a second network communication path for storage as determined by the third computing device.

10. The non-transitory computer readable memory device of claim 9 further comprises:

a fifth memory section associated with the first computing device that stores operational instructions that, when executed by the first computing device, causes the first computing device to;

receive the data from a fourth computing device; and

receive the data partitioning dispersed storage scheme from the fourth computing device.

11. The non-transitory computer readable memory device of claim 9, wherein the first memory section associated with the first computing device further stores operational instructions that, when executed by the first computing device, causes the first computing device to determine whether to partition the data by:

interpreting data size of the data;

determining whether the data size is of a level that requires partitioning in accordance with the data partitioning dispersed storage scheme; and

when the data size is of a level that requires partitioning, indicating partitioning of the data.

12. The non-transitory computer readable memory device of claim 9, wherein the second memory section associated with the first computing device stores further operational instructions that, when executed by the first computing device, causes the first computing device to partition the data by:

selecting a portion of the data per a data portion size indication of the data partitioning dispersed storage scheme to produce the local data portion.

13. The non-transitory computer readable memory device of claim 9, wherein the second memory section associated with the first computing device stores further operational instructions that, when executed by the first computing device, causes the first computing device to partition the data by:

selecting the local data portion to have a partial overlap of data with the remaining data portion in accordance with the data partitioning dispersed storage scheme.

14. The non-transitory computer readable memory device of claim 9, wherein the dispersed storage error encode comprises one or more of:

dispersed storage error encode;

slice the local data portion into a set of slices and encrypt the set of slices;

slice the local data portion into a set of slices; and

replicate the local data portion.

15. The non-transitory computer readable memory device of claim 9, wherein the second memory section associated with the first computing device stores further operational instructions that, when executed by the first computing device, causes the first computing device to send the remaining data portion to the second computing device by:

selecting the second computing device based at least one of;

the data partitioning dispersed storage scheme, a lookup, a query, or a message.

16. The non-transitory computer readable memory device of claim 9 further comprises:

a fifth memory section associated with the first computing device that stores operational instructions that, when executed by the first computing device, causes the first computing device to update a location table to indicate that the plurality of encoded data elements is stored in the first set of storage units and that the another plurality of encoded data elements is stored in the second set of storage units.