Distributed storage network that processes data in either fixed or variable sizes

• US 8,468,137 B2
• Filed: 06/17/2010
• Issued: 06/18/2013
• Est. Priority Date: 10/30/2009
• Status: Expired due to Fees

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: Baptist, Andrew, Palthepu, Srinivas, Cilfone, Bart, Resch, Jason K., Volvovski, Ilya
• Primary Examiner(s): Lu, Kuen S
• Assistant Examiner(s): Nguyen, Loan T
• Application Number:
  US12/818,137
• Publication Number:
  US 20110106769A1
• Time in Patent Office: 1,097 Days
• Field of Search: None
• US Class Current: 707/652
• CPC Class Codes: G06F 11/1004 to protect a block of data words, e.g. CRC or checksum G06F11/1076 takes precedence; security arrangements for protecting computers or computer systems against unauthorized activity G06F21/00
• CPC Class Codes: G06F 12/1408 by using cryptography for digital transmission H04L9/00
• CPC Class Codes: G06F 21/6218 to a system of files or objects, e.g. local or distributed file system or database
• CPC Class Codes: G06F 2221/2107 File encryption
• CPC Class Codes: G06F 2221/2151 Time stamp
• CPC Class Codes: H04L 2209/34 Encoding or coding, e.g. Huffman coding or error correction
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 69/04 Protocols for data compression, e.g. ROHC
• CPC Class Codes: H04L 9/0819 Key transport or distribution, i.e. key establishment techniques where one party creates or otherwise obtains a secret value, and securely transfers it to the other(s) network architectures or network communication protocols for key distribution in a packet data network H04L63/062
• CPC Class Codes: H04L 9/0863 involving passwords or one-time passwords network architectures or network communication protocols for using one-time keys in a packet data network H04L63/067
• CPC Class Codes: H04L 9/0894 Escrow, recovery or storing of secret information, e.g. secret key escrow or cryptographic key storage

Abstract

A distributed storage network receives a data object and determines a size of the data object. The network then determines whether the data object should be segmented into fixed size segments or variable sizes segments based upon some variable-length algorithm. If the fixed approach is selected, the data object is segmented into two or more fixed size segments with attached header information and the information is further processed using error coding dispersal storage functions within the network. If the variable approach is selected, the data object is segmented into two or more variable-sized segments with attached header information and the information is further processed using error coding dispersal storage functions within the network. Both the variable and fixed approaches enable storage of error-correctable data slices in a distributed storage manner.

Claims

1. A method for execution by a computing core to store a data object within at least a portion of a distributed storage network, the method comprising:

receiving, via an interface of the computer core, the data object;

ascertaining an error coding dispersal storage function for the data object;

determining a segmenting approach of the error coding dispersal storage function, wherein the segmenting approach includes a fixed-sized segmenting or a variable sized segmenting;

when the segmenting approach is the variable sized segmenting;

determining a variable segmenting pattern; and

segmenting the data object into a plurality of segments in accordance with the variable segmenting pattern;

for a segment of the plurality of segments;

encoding the segment into a set of error coded data slices in accordance with an error coding function of the error coding dispersal storage function;

creating a set of slice names corresponding to the set of error coded data slices;

for each of the set of error coded data slices, appending a corresponding one of the set of slice names thereto to create a set of data slices; and

outputting, via the interface, the set of data slices for storage in storage units of the distributed storage network.

2. The method of claim 1 wherein the segmenting the data object further comprises:

generating a segment header to include one or more of;

data object size and the variable segmenting pattern; and

appending the segment header to the data object prior to segmenting the data object.

3. The method of claim 1, wherein the segmenting the data object further comprises:

generating a segment header to include one or more of;

data object size and the variable segmenting pattern; and

appending the segment header to at least one of the plurality of segments.

4. The method of claim 1 further comprises:

when the segmenting approach is the fixed-sized segmenting;

determining a fixed size;

generating a segment header to include one or more of;

data object size and the fixed size;

appending the segment header to the data object to produce an appended data object; and

segmenting the appended data object into a plurality of segments in accordance with the fixed size.

5. The method of claim 1 further comprises:

when the segmenting approach is the fixed-sized segmenting;

determining a fixed size;

generating a segment header to include one or more of;

data object size and the fixed size;

segmenting the data object into a plurality of segments in accordance with the fixed size; and

appending the segment header to one or more of the plurality of segments.

6. The method of claim 1 further comprises:

determining the segmenting approach based on at least one of;

one or more of the metadata, a system loading indicator, received information appended to the data object, a lookup data value, a command, a predetermined algorithm, inspection of the data object, and a user vault entry.

7. The method of claim 1 wherein determining the variable segmenting pattern comprises at least one of:

utilizing a random selection of sizes as the variable segmenting pattern;

utilizing a pseudo-random selection of sizes as the variable segmenting pattern;

utilizing a cyclical selection of sizes as the variable segmenting pattern;

utilizing a pattern of increasing sizes as the variable segmenting pattern;

utilizing a pattern of decreasing sizes as the variable segmenting pattern; and

utilizing a function over time to determines sizes as the variable segmenting pattern.

8. A computer core comprising:

an interface; and

a processing module, the processing module being coupled to the interface and being operable to;

receive, via the interface, the data object;

ascertain an error coding dispersal storage function for the data object;

determine a segmenting approach of the error coding dispersal storage function, wherein the segmenting approach includes a fixed-sized segmenting or a variable sized segmenting;

when the segmenting approach is the variable sized segmenting;

determine a variable segmenting pattern; and

segment the data object into a plurality of segments in accordance with the variable segmenting pattern; and

for a segment of the plurality of segments;

encode the segment into a set of error coded data slices in accordance with an error coding function of the error coding dispersal storage function;

create a set of slice names corresponding to the set of error coded data slices;

for each of the set of error coded data slices, append a corresponding one of the set of slice names thereto to create a set of data slices; and

output, via the interface, the set of data slices for storage in storage units of the distributed storage network.

9. The computer core of claim 8, wherein the processing module is further operable to segment the data object further by:

generating a segment header to include one or more of;

data object size and the variable segmenting pattern; and

appending the segment header to at least one of the plurality of segments.

10. The computer core of claim 8, wherein the processing module is further operable to:

when the segmenting approach is the fixed-sized segmenting;

determine a fixed size;

generate a segment header to include one or more of;

data object size and the fixed size;

append the segment header to the data object to produce an appended data object; and

segment the appended data object into a plurality of segments in accordance with the fixed size.

11. The computer core of claim 8, wherein the processing module is further operable to:

when the segmenting approach is the fixed-sized segmenting;

determine a fixed size;

generate a segment header to include one or more of;

data object size and the fixed size;

segment the data object into a plurality of segments in accordance with the fixed size; and

append the segment header to one or more of the plurality of segments.

12. The computer core of claim 8, wherein the processing module is further operable to:

determine the segmenting approach based on at least one of;

one or more of the metadata, a system loading indicator, received information appended to the data object, a lookup data value, a command, a predetermined algorithm, inspection of the data object, and a user vault entry.

13. The computer core of claim 8, wherein the processing module is further operable determine the variable segmenting pattern by at least one of:

utilizing a random selection of sizes as the variable segmenting pattern;

utilizing a pseudo-random selection of sizes as the variable segmenting pattern;

utilizing a cyclical selection of sizes as the variable segmenting pattern;

utilizing a pattern of increasing sizes as the variable segmenting pattern;

utilizing a pattern of decreasing sizes as the variable segmenting pattern; and

utilizing a function over time to determines sizes as the variable segmenting pattern.

14. The computer core of claim 8, wherein the processing module is further operable to segment the data object further by:

generating a segment header to include one or more of;

data object size and the variable segmenting pattern; and

appending the segment header to the data object prior to segmenting the data object.