Method and apparatus for dispersed storage memory device utilization

• US 8,478,937 B2
• Filed: 05/12/2010
• Issued: 07/02/2013
• Est. Priority Date: 09/30/2009
• Status: Expired due to Fees

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: Grube, Gary W., Markison, Timothy W.
• Primary Examiner(s): Dudek, Jr., Edward
• Assistant Examiner(s): Verderamo, III, Ralph A
• Application Number:
  US12/778,680
• Publication Number:
  US 20110078512A1
• Time in Patent Office: 1,147 Days
• Field of Search: None
• US Class Current: 711/114
• CPC Class Codes: G06F 11/10 Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/2053 where persistent mass storage functionality or persistent mass storage control functionality is redundant error detection or correction in information storage based on relative movement between record carrier and transducer G11B20/18
• CPC Class Codes: G06F 11/2089 Redundant storage control functionality
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/1827 Management specifically adapted to NAS management of storage area networks [SAN] G06F3/067
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0665 at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/068 Hybrid storage device
• CPC Class Codes: G06F 3/0685 Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays

Abstract

A method begins by a processing module receiving data for storage. The method continues with the processing module determining storage metadata regarding the data the method continues with the processing module. The method continues with the processing module determining memory device capabilities based on the storage metadata. The method continues with the processing module identifying memory devices based on the memory device capabilities to produce identified memory devices. The method continues with the processing module encoding the data into a plurality of data slices in accordance with an error coding dispersal function. The method continues with the processing module storing the plurality of data slices in the identified memory devices.

Claims

1. A method for dispersed memory storage comprises:

receiving data for storage;

determining storage metadata regarding storage requirements of the data;

determining memory device capabilities of a dispersed storage network (DSN) memory;

identifying memory devices of the DSN memory based on the memory device capabilities and the storage metadata to produce identified memory devices;

encoding the data into a plurality of data slices in accordance with an error coding dispersal storage function, wherein the encoding includes;

determining operational parameters of the error coding dispersal storage function based on the storage metadata, which further includes;

determining a first pillar width corresponding to an estimated time between retrievals requirement component of the storage metadata; and

determining a second pillar width corresponding to a storage reliability requirement component of the storage metadata, wherein the second pillar width is greater than the first pillar width, wherein the operational parameters includes at least one of;

a pillar width;

a read threshold;

an error coding algorithm;

an encryption algorithm;

a slicing parameter;

a compression algorithm;

an integrity check method;

a caching settings; and

a parallelism settings; and

storing the plurality of data slices in the identified memory devices.

2. The method of claim 1, wherein the storage metadata comprises at least one of:

a data type;

a data size;

a data priority;

a data security index;

an estimated storage time;

an estimated time between retrievals; and

a storage requirement.

3. The method of claim 1 further comprises:

identifying the memory devices based on a comparison of at least a portion of the storage metadata to one or more memory device capabilities that includes at least one of;

a memory device storage cost;

a memory device storage access speed;

a memory device storage reliability;

a memory device storage availability; and

a memory device storage capacity.

4. The method of claim 1, wherein the storing the plurality of data slices in the identified memory devices comprises:

storing at least a read threshold number of the plurality of data slices in a first set of the identified memory devices; and

storing remaining data slices of the plurality of data slices in a second set of the identified memory devices.

5. The method of claim 1 further comprises:

determining whether a condition has occurred to update the identifying the memory devices, wherein the condition includes at least one of;

updating of the storage metadata, a change of memory device characteristics, a change of available memory devices, and an occurrence of a triggering event; and

when the condition has occurred, re-identifying memory devices.

6. The method claim 1, wherein the determining the storage metadata further comprises:

segmenting the data in accordance with the error coding dispersal storage function to produce a plurality of data segments; and

determining the storage metadata regarding at least one of the plurality of data segments.

7. The method claim 1, wherein the determining the storage metadata regarding the data comprises:

partitioning the data based on customized data content and generic data content to produce a customized data partition and a generic data partition;

determining customized data partition storage metadata regarding the customized data partition;

determining generic data partition storage metadata regarding the generic data partition; and

aggregating the customized data partition storage metadata and the generic data partition storage metadata to produce the storage metadata.

8. A computer comprises:

a dispersal memory interface; and

a processing module operable to;

obtain data for storage;

determine storage metadata regarding storage requirements of the data;

determine memory device capabilities of a dispersed storage network (DSN) memory;

identify memory devices of the DSN memory based on the memory device capabilities and the storage metadata to produce identified memory devices;

encode the data into a plurality of data slices in accordance with an error coding dispersal function, wherein the encoding includes;

determining operational parameters of the error coding dispersal storage function based on the storage metadata, which further includes;

determining a first pillar width corresponding to an estimated time between retrievals requirement component of the storage metadata; and

determining a second pillar width corresponding to a storage reliability requirement component of the storage metadata, wherein the second pillar width is greater than the first pillar width, wherein the operational parameters includes at least one of;

a pillar width;

a read threshold;

an error coding algorithm;

an encryption algorithm;

a slicing parameter;

a compression algorithm;

an integrity check method;

a caching settings; and

a parallelism settings; and

output, via the dispersal memory interface, the plurality of data slices to the identified memory devices for storage therein.

9. The computer of claim 8 further comprises at least one of:

a main memory;

a local non-main memory; and

a non-local non-main memory.

10. The computer of claim 8, wherein the storage metadata comprises at least one of:

a data type;

a data size;

a data priority;

a data security index;

an estimated storage time;

an estimated time between retrievals; and

a storage requirement.

11. The computer of claim 8, wherein the processing module further functions to:

identify the memory devices based on a comparison of at least a portion of the storage metadata to one or more memory device capabilities that includes at least one of;

a memory device storage cost;

a memory device storage access speed;

a memory device storage reliability;

a memory device storage availability; and

a memory device storage capacity.

12. The computer of claim 8, wherein the processing module further functions to store the plurality of data slices by:

outputting, via the dispersal memory interface, at least a read threshold number of the plurality of data slices to a first set of the identified memory devices; and

outputting, via the dispersal memory interface, remaining data slices of the plurality of data slices to a second set of the identified memory devices.

13. The computer of claim 8, wherein the processing module further functions to:

determine whether a condition has occurred to update the identify the memory devices, wherein the condition includes at least one of;

updating of the storage metadata, a change of memory device characteristics, a change of available memory devices, and an occurrence of a triggering event; and

when the condition has occurred, re-identify memory devices.

14. The computer of claim 8, wherein the processing module further functions to determine storage metadata regarding the data by:

segmenting the data in accordance with the error coding dispersal function to produce a plurality of data segments; and

determining storage metadata regarding at least one of the plurality of data segments.

15. The computer of claim 8, wherein the processing module further functions to determine storage metadata by:

partitioning the data based on customized data content and generic data content to produce a customized data partition and a generic data partition;

determining customized data partition storage metadata regarding the customized data partition;

determining generic data partition storage metadata regarding the generic data partition; and

aggregating the customized data partition storage metadata and the generic data partition storage metadata to produce the storage metadata.