Method and apparatus for dispersed storage data transfer

• US 9,448,730 B2
• Filed: 05/12/2010
• Issued: 09/20/2016
• Est. Priority Date: 09/30/2009
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Grube, Gary W., Markison, Timothy W.
• Primary Examiner(s): SCHELL, JOSEPH O
• Application Number:
  US12/778,701
• Publication Number:
  US 20110078493A1
• Time in Patent Office: 2,323 Days
• Field of Search: 714/15, 707/687, 707/690, 707/697, 707/698, 707/809, 707/602
• US Class Current: 1/1
• CPC Class Codes: G06F 11/10 Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/2053 where persistent mass storage functionality or persistent mass storage control functionality is redundant error detection or correction in information storage based on relative movement between record carrier and transducer G11B20/18
• CPC Class Codes: G06F 11/2089 Redundant storage control functionality
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/1827 Management specifically adapted to NAS management of storage area networks [SAN] G06F3/067
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0665 at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/068 Hybrid storage device
• CPC Class Codes: G06F 3/0685 Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays

Abstract

The method begins with a processing module determining whether to reconstruct data corresponding to a plurality of data slices when the plurality of data slices is to be transferred from a first type of memory device to a second type of memory device. The method continues with the processing module retrieving the plurality of data slices from a first set of memory devices that are of the first type of memory, reconstructing at least a portion of the data from the plurality of data slice in accordance with a first error coding dispersal function to produce reconstructed data, encoding the reconstructed data in accordance with a second error coding dispersal function to produce a second plurality of data slices, and storing the second plurality of data slices in a second set of memory devices that are of the second type of memory when the data is to be reconstructed.

Claims

1. A method for dispersed memory storage comprises:

determining whether to move storage of a first plurality of data slices from a first memory device to a second memory device, wherein the first memory device is of a first memory type that includes one or more of a flash memory and a random access memory, wherein the second memory device is of a second memory type that includes one or more of a magnetic hard disk and an optical disk, wherein the first memory device has first memory device capabilities based on the first memory type and the second memory device has second memory device capabilities based on the second memory type and the determining to move the first plurality of data slices is based on a difference between the first and second memory device capabilities;

determining whether to move the first plurality of data slices from the first memory device to the second memory device by reconstructing data or to move the first plurality of data slices without reconstruction;

moving storage of the first plurality of data slices from the first memory device to the second memory device when the first plurality of data slices is to be moved without reconstruction; and

wherein when the data is to be reconstructed;

retrieving the first plurality of data slices from the first memory device;

reconstructing at least a portion of the data from the first plurality of data slices in accordance with a first error coding dispersal function to produce reconstructed data, in which the first error coding dispersal function corresponds to the memory device capabilities of the first memory device;

encoding the reconstructed data in accordance with a second error coding dispersal function to produce a second plurality of data slices, wherein the second error coding dispersal function corresponds to the memory device capabilities of the second memory device; and

storing the second plurality of data slices in the second memory device.

2. The method of claim 1, wherein the determining whether to move the first plurality of data slices is based on at least one of:

elapsed time period of storage in the first memory device;

elapsed time period since a last data slice retrieval from the first memory device;

a memory utilization indicator for the first memory device;

receiving a command; and

receiving a request.

3. The method of claim 1 further comprises:

when the data is to be moved without reconstruction;

retrieving the first plurality of data slices from the first memory device;

storing the first plurality of data slices in the second memory device; and

facilitating deletion of the first plurality of data slices from the first memory device.

4. The method of claim 1, wherein the first error coding dispersal function comprises:

an error coding type that includes at least one of an error coding algorithm, an encryption algorithm, and a compression algorithm; and

operational parameters that include two or more of a pillar width, a read threshold, a slicing parameter, an integrity check method, a caching settings, and a parallelism settings.

5. The method of claim 1, wherein the second error coding dispersal function comprises:

an error coding type that includes at least one of an error coding algorithm, an encryption algorithm, and a compression algorithm; and

operational parameters that include two or more of a pillar width, a read threshold, a slicing parameter, an integrity check method, a caching settings, and parallelism settings.

6. The method of claim 1, wherein the first plurality of data slices correspond to a data segment.

7. The method of claim 1, wherein the data is customized data or generic data.

8. A computer comprises:

a dispersal memory interface; and

a processing module operable to;

determine whether to move storage of a first plurality of data slices from a first memory device to a second memory device, wherein the first memory device is of a first memory type that includes one or more of a flash memory and a random access memory, wherein the second memory device is of a second memory type that includes one or more of a magnetic hard disk and an optical disk, wherein the first memory device has first memory device capabilities based on the first memory type and the second memory device has second memory device capabilities based on the second memory type and the determining to move the first plurality of data slices is based on a difference between the first and second memory device capabilities;

determine whether to move the first plurality of data slices from the first memory device to the second memory device by reconstructing data or to move the first plurality of data slices without reconstruction;

move storage of the first plurality of data slices from the first memory device to the second memory device when the first plurality of data slices is to be moved without reconstruction; and

when the data is to be reconstructed;

retrieve, via the dispersal memory interface, the first plurality of data slices from the first memory device;

reconstruct at least a portion of the data from the first plurality of data slices in accordance with a first error coding dispersal function to produce reconstructed data, in which the first error coding dispersal function corresponds to the memory device capabilities of the first memory device;

encode the reconstructed data in accordance with a second error coding dispersal function to produce a second plurality of data slices, wherein the second error coding dispersal function corresponds to the memory device capabilities of the second memory device; and

output, via the dispersal memory interface, the second plurality of data slices to the second memory device.

9. The computer of claim 8 further comprises:

a main memory and at least one of;

a local non-main memory; and

a non-local non-main memory.

10. The computer of claim 8, wherein the dispersal memory interface may comprise at least one of:

a multi-general purpose input output unit; and

a plurality of interfaces.

11. The computer of claim 8, wherein the processing module determines whether to transfer the first plurality of data slices based on at least one of:

elapsed time period of storage in the first memory device;

elapsed time period since a last data slice retrieval from the first memory device;

a memory utilization indicator for the first memory device;

obtaining a command; and

obtaining a request.

12. The computer of claim 8, wherein when the data is to be moved without reconstruction, the processing module further functions to:

retrieve, via the dispersal memory interface, the first plurality of data slices from the first memory device;

output, via the dispersal memory interface, the first plurality of data slices to be stored in the second memory device; and

facilitate deletion, via the dispersal memory interface, of the first plurality of data slices from the first memory device.

13. The computer of claim 8, wherein the first error coding dispersal function comprises:

an error coding type that includes at least one of an error coding algorithm, an encryption algorithm, and a compression algorithm; and

operational parameters that include two or more of a pillar width, a read threshold, a slicing parameter, an integrity check method, a caching settings, and a parallelism settings.

14. The computer of claim 8, wherein the second error coding dispersal function comprises:

an error coding type that includes at least one of an error coding algorithm, an encryption algorithm, and a compression algorithm; and

operational parameters that include two or more of a pillar width, a read threshold, a slicing parameter, an integrity check method, a caching settings, and a parallelism settings.

15. The computer of claim 8, wherein the first plurality of data slices correspond to a data segment.

16. The computer of claim 8, wherein the data is customized data or generic data.