Data migration in a dispersed storage network

• US 8,954,667 B2
• Filed: 11/10/2010
• Issued: 02/10/2015
• Est. Priority Date: 01/28/2010
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: Dhuse, Greg, Baptist, Andrew, Leggette, Wesley, Volvovski, Ilya, Resch, Jason K.
• Primary Examiner(s): Faal, Baboucarr
• Application Number:
  US12/943,826
• Publication Number:
  US 20110185141A1
• Time in Patent Office: 1,553 Days
• US Class Current: 711/114
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 3/0614 Improving the reliability of storage systems
• CPC Class Codes: G06F 3/0647 Migration mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method begins by processing module determining data to migrate, wherein the data is stored as a plurality of sets of encoded data slices in a first set of dispersed storage (DS) units. The method continues with the processing module retrieving at least a read threshold number of encoded data slices for each set of the plurality of sets of encoded data slices and dispersed storage error decoding the at least the read threshold number of encoded data slices in accordance with error coding dispersal storage function parameters to reproduce the data. The method continues with the processing module dispersed storage error encoding the data in accordance with second error coding dispersal storage function parameters to produce a plurality of sets of second encoded data slices and sending at least a write threshold number of second encoded data slices to a second set of DS units for storage therein.

Claims

1. A method comprises:

determining data to migrate, wherein the data is encoded and stored as a plurality of sets of encoded data slices in a first set of dispersed storage (DS) units, wherein the data was encoded by;

dividing the data into data segments;

for each data segment, dispersed storage error encoding a data segment in accordance with first error coding dispersal storage function parameters to produce a set of the plurality of sets of encoded data slices, wherein the set of encoded data slices includes a first number of encoded data slices and wherein at least a first decode threshold number of the first number of encoded data slices are needed to recover the data segment;

retrieving at least a read threshold number of encoded data slices for each set of the plurality of sets of encoded data slices, wherein the read threshold number is less than a total number of encoded data slices in a set of encoded data slices of the plurality of sets of encoded data slices;

dispersed storage error decoding the at least the read threshold number of encoded data slices for each set of the plurality of sets of encoded data slices in accordance with the first error coding dispersal storage function parameters to reproduce the data;

dispersed storage error encoding the reproduced data in accordance with second error coding dispersal storage function parameters to produce a plurality of sets of second encoded data slices, wherein a set of the plurality of sets of second encoded data slices includes a second number of encoded data slices and wherein at least a second decode threshold number of the second number of encoded data slices are needed to recover a corresponding data segment; and

for each set of the plurality of sets of second encoded data slices, sending at least a write threshold number of second encoded data slices to a second set of DS units for storage therein;

wherein the write threshold number is less than the total number of encoded data slices in the set of encoded data slices and is greater than the read threshold number.

2. The method of claim 1 further comprises:

creating an entry in a virtual dispersed storage network (DSN) address to physical location table to indicate an association between the plurality of sets of second encoded data slices and the second set of DS units; and

maintaining an entry in the virtual dispersed storage network (DSN) address to physical location table to indicate an association between the plurality of sets of encoded data slices and the set of DS units;

or

deleting the entry in the virtual dispersed storage network (DSN) address to physical location table to indicate an association between the plurality of sets of encoded data slices and the set of DS units.

3. The method of claim 1 further comprises one of:

sending a delete encoded data slice message to each DS unit of the first set of DS units, wherein the delete encoded data slice message includes a request to delete the encoded data slices for each set of the plurality of sets of encoded data slices;

when receiving a store data slice confirmation message indicating that the plurality of sets of second encoded data slices are stored in the second set of DS units, sending the delete encoded data slice message to each DS unit of the first set of DS units;

when a wait time period has elapsed after sending the at least some of the plurality of sets of second encoded data slices to a second set of DS units, sending the delete encoded data slice message to each DS unit of the first set of DS units; and

when a storage space required indicator is active, sending the delete encoded data slice message to each DS unit of the first set of DS units.

4. The method of claim 1, wherein each of the error coding dispersal storage function parameters and the second error coding dispersal storage function parameters comprises at least one of:

a pillars list;

a segmenting protocol;

a pre-slice data manipulation function;

a forward error correction encoding function;

a slicing pillar width;

a post-slice data manipulation function;

a write threshold; and

a read threshold.

5. The method of claim 1, wherein the determining data to migrate is based on one or more of:

an amount of data to move indicator;

a data transferred indicator;

a data transfer continuation indicator;

a DS managing unit message;

a DS unit query;

a DS unit message;

a newly allocated DS unit detection;

a location;

a schedule;

a list;

a predetermination;

an error message; and

a command.

6. A computer comprises:

an interface; and

a processing module operable to;

determine data to migrate, wherein the data is encoded and stored as a plurality of sets of encoded data slices in a first set of dispersed storage (DS) units, wherein the data was encoded by;

dividing the data into data segments;

for each data segment, dispersed storage error encoding a data segment in accordance with first error coding dispersal storage function parameters to produce a set of the plurality of sets of encoded data slices, wherein the set of encoded data slices includes a first number of encoded data slices and wherein at least a first decode threshold number of the first number of encoded data slices are needed to recover the data segment;

retrieve, via the interface, at least a read threshold number of encoded data slices for each set of the plurality of sets of encoded data slices, wherein the read threshold number is less than a total number of encoded data slices in a set of encoded data slices of the plurality of sets of encoded data slices;

dispersed storage error decode the at least the read threshold number of encoded data slices for each set of the plurality of sets of encoded data slices in accordance with the first error coding dispersal storage function parameters to reproduce the data;

dispersed storage error encode the reproduced data in accordance with second error coding dispersal storage function parameters to produce a plurality of sets of second encoded data slices, wherein a set of the plurality of sets of second encoded data slices includes a second number of encoded data slices and wherein at least a second decode threshold number of the second number of encoded data slices are needed to recover a corresponding data segment; and

send, via the interface, at least a write threshold number of second encoded data slices to a second set of DS units for storage therein for each set of the plurality of sets of second encoded data slices, wherein the write threshold number is less than the total number of encoded data slices in the set of encoded data slices and is greater than the read threshold number.

7. The computer of claim 6, wherein the processing module further functions to:

create an entry in a virtual dispersed storage network (DSN) address to physical location table to indicate an association between the plurality of sets of second encoded data slices and the second set of DS units; and

maintain an entry in the virtual dispersed storage network (DSN) address to physical location table to indicate an association between the plurality of sets of encoded data slices and the set of DS units;

or

delete the entry in the virtual dispersed storage network (DSN) address to physical location table to indicate an association between the plurality of sets of encoded data slices and the set of DS units.

8. The computer of claim 6, wherein the processing module further functions to perform one of:

send, via the interface, a delete encoded data slice message to each DS unit of the first set of DS units, wherein the delete encoded data slice message includes a request to delete the encoded data slices for each set of the plurality of sets of encoded data slices;

send, via the interface, the delete encoded data slice message to each DS unit of the first set of DS units when receiving, via the interface, a store data slice confirmation message indicating that the plurality of sets of second encoded data slices are stored in the second set of DS units;

send, via the interface, the delete encoded data slice message to each DS unit of the first set of DS units when a wait time period has elapsed after sending, via the interface, the at least some of the plurality of sets of second encoded data slices to a second set of DS units; and

send, via the interface, the delete encoded data slice message to each DS unit of the first set of DS units when a storage space required indicator is active.

9. The computer of claim 6, wherein each of the error coding dispersal storage function parameters and the second error coding dispersal storage function parameters comprises at least one of:

a pillars list;

a segmenting protocol;

a pre-slice data manipulation function;

a forward error correction encoding function;

a slicing pillar width;

a post-slice data manipulation function;

a write threshold; and

a read threshold.

10. The computer of claim 6, wherein the processing module functions to determine data to migrate based on one or more of:

an amount of data to move indicator;

a data transferred indicator;

a data transfer continuation indicator;

a DS managing unit message;

a DS unit query;

a DS unit message;

a newly allocated DS unit detection;

a location;

a schedule;

a list;

a predetermination;

an error message; and

a command.