Migrating data in a distributed storage network

• US 10,089,036 B2
• Filed: 11/30/2017
• Issued: 10/02/2018
• Est. Priority Date: 07/31/2014
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Kazi, Asimuddin, Kaczmarek, Joseph M., Resch, Jason K.
• Primary Examiner(s): Nguyen, Phuoc
• Application Number:
  US15/827,073
• Publication Number:
  US 20180081586A1
• Time in Patent Office: 306 Days
• Field of Search: 709212, 709213, 709224, 709238
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0611 in relation to response time
• CPC Class Codes: G06F 3/0647 Migration mechanisms
• CPC Class Codes: G06F 3/0653 Monitoring storage devices or systems
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method begins by a processing modules of a storage unit (SU) of a dispersed storage network (DSN) determining to migrate a set of encoded data slices from the first storage format to a second storage format. The method continues with the SU selecting the second storage format for the set of EDSs and selecting a second DSN memory configured to store the set of EDSs based on the second storage format. The method continues with the SU determining a first migration rate and migrating a first subset of the set of EDSs from the first DSN memory to the second DSN memory. The method continues with the SU monitoring activity associated with the DSN memory, selecting a second migration rate based on the activity associated with the at least one of the first DSN memory or the second DSN memory, and migrating a second subset of the set of EDSs based on the second migration rate.

Claims

1. A method for execution by one or more processing modules of one or more storage units (SUs) of a dispersed storage network (DSN), the method comprises:

identifying, by a storage unit (SU) of the one or more SUs, a first storage format for a set of encoded data slices (EDSs) stored in a first DSN memory, wherein a data object is segmented into a plurality of data segments, and wherein a data segment of the plurality of data segments is dispersed error encoded in accordance with dispersed error encoding parameters to produce the set of encoded data slices (EDSs);

determining to migrate the set of EDSs from the first storage format to a second storage format;

selecting the second storage format for the set of EDSs;

selecting a second DSN memory for the set of EDSs, wherein the second DSN memory is configured to store the set of EDSs based on the second storage format;

determining a first migration rate to be used when migrating the set of EDSs from the first DSN memory to the second DSN memory;

migrating a first subset of the set of EDSs from the first DSN memory to the second DSN memory based on the first migration rate to store the set of EDSs within the second DSN memory based on the second storage format;

monitoring at least one of data access request activity or data access response activity associated with at least one of the first DSN memory or the second DSN memory;

selecting a second migration rate based on the at least one of data access request activity or data access response activity associated with the at least one of the first DSN memory or the second DSN memory; and

migrating a second subset of the set of EDSs from the first DSN memory to the second DSN memory based on the second migration rate.

2. The method of claim 1, wherein the determining to migrate the set of EDSs from the first storage format to the second storage format is based on at least one of interpreting a request, identifying a data type, receiving new storage configuration information, detecting a storage efficiency level that compares unfavorably to a threshold level, and detecting an unfavorable performance level.

3. The method of claim 1, wherein the first DSN memory and the second DSN memory are common to the same memory device.

4. The method of claim 1, wherein the at least one of data access request activity or data access response activity is associated with at least one of rebuilding encoded data slices, writing encoded data slices, retrieving encoded data slices, listing encoded data slices, performing SU maintenance tasks and performing SU balancing tasks.

5. The method of claim 1, wherein the determining to migrate the set of EDSs from the first storage format to the second storage format is based on at least one of detecting a storage inefficiency, receiving an error message, interpreting an error message and detecting an available storage level that compares favorably to an available storage threshold level.

6. The method of claim 1, wherein the selecting the second storage format for the set of EDSs comprises at least one of:

identifying one or more storage formats;

estimating a level of storage performance for each of the one or more storage formats; and

selecting the se storage format that corresponds to a favorable estimated storage performance level.

7. The method of claim 1, wherein the migrating the first subset of the set of EDSs from the first DSN memory to the second DSN memory includes retrieving the first subset of the set of EDSs from the first DSN memory using the first storage format and storing the first subset of the set of EDSs using the second storage format.

8. A computer readable storage medium comprises:

at least one memory section that stores operational instructions that, when executed by one or more processing modules of a storage unit of a dispersed storage network (DSN), causes the storage unit to;

identify a first storage format for a set of encoded data slices (EDSs) stored in a first DSN memory, wherein a data object is segmented into a plurality of data segments, and wherein a data segment of the plurality of data segments is dispersed error encoded in accordance with dispersed error encoding parameters to produce the set of encoded data slices (EDSs);

determine to migrate the set of EDSs from the first storage format to a second storage format;

select the second storage format for the set of EDSs;

select a second DSN memory for the set of EDSs, wherein the second DSN memory is configured to store the set of EDSs based on the second storage format;

determine a first migration rate to be used when migrating the set of EDSs from the first DSN memory to the second DSN memory;

migrate a first subset of the set of EDSs from the first DSN memory to the second DSN memory based on the first migration rate to store the set of EDSs within the second DSN memory based on the second storage format;

monitor at least one of data access request activity or data access response activity associated with at least one of the first DSN memory or the second DSN memory;

select a second migration rate based on the at least one of data access request activity or data access response activity associated with the at least one of the first DSN memory or the second DSN memory; and

migrate a second subset of the set of EDSs from the first DSN memory to the second DSN memory based on the second migration rate.

9. The computer readable storage medium of claim 8 further comprises:

the at least one memory section stores further operational instructions that, when executed by the one or more processing modules, causes the one or more storage units of the DSN to;

determine to migrate the set of EDSs from the first storage format to the second storage format based on at least one of interpreting a request, identifying a data type, receiving new storage configuration information, detecting a storage efficiency level that compares unfavorably to a threshold level, and detecting an unfavorable performance level.

10. The computer readable storage medium of claim 8, wherein the first DSN memory and the second DSN memory are common to the same memory device.

11. The computer readable storage medium of claim 8, wherein the at least one of data access request activity or data access response activity comprises at least one of rebuilding encoded data slices, writing encoded data slices, retrieving encoded data slices, listing encoded data slices, performing SU maintenance tasks and performing SU balancing tasks.

12. The computer readable storage medium of claim 8 further comprises:

the at least one memory section stores further operational instructions that, when executed by the one or more processing modules, causes the one or more storage units of the DSN to;

determine to migrate the set of EDSs from the first storage format to the second storage format based on at least one of detecting a storage inefficiency, receiving an error message, interpreting an error message and detecting an available storage level that compares favorably to an available storage threshold level.

13. The computer readable storage medium of claim 8 further comprises:

the at least one memory section stores further operational instructions that, when executed by the one or more processing modules, causes the one or more storage units of the DSN to;

select the second storage format for the set of EDSs based on at least one of;

identifying one or more storage formats;

estimating a level of storage performance for each of the one or more storage formats; and

selecting the se storage format that corresponds to a favorable estimated storage performance level.

14. The computer readable storage medium of claim 8 further comprises:

the at least one memory section stores further operational instructions that, when executed by the one or more processing modules, causes the one or more storage units of the DSN to;

migrate the first subset of the set of EDSs from the first DSN memory to the second DSN memory by retrieving the first subset of the set of EDSs from the first DSN memory using the first storage format; and

store the first subset of the set of EDSs using the second storage format.

15. A storage unit of a group of storage units of a dispersed storage network (DSN), the storage unit comprises:

an interface;

a local memory; and

a processing module operably coupled to the interface and the local memory, wherein the processing module functions to;

identify a first storage format for a set of encoded data slices (EDSs) stored in a first DSN memory, wherein a data object is segmented into a plurality of data segments, and wherein a data segment of the plurality of data segments is dispersed error encoded in accordance with dispersed error encoding parameters to produce the set of encoded data slices (EDSs);

determine to migrate the set of EDSs from the first storage format to a second storage format;

select the second storage format for the set of EDSs;

select a second DSN memory for the set of EDSs, wherein the second DSN memory is configured to store the set of EDSs based on the second storage format;

determine a first migration rate to be used when migrating the set of EDSs from the first DSN memory to the second DSN memory;

migrate a first subset of the set of EDSs from the first DSN memory to the second DSN memory based on the first migration rate to store the set of EDSs within the second DSN memory based on the second storage format;

monitor at least one of data access request activity or data access response activity associated with at least one of the first DSN memory or the second DSN memory;

select a second migration rate based on the at least one of data access request activity or data access response activity associated with the at least one of the first DSN memory or the second DSN memory; and

migrate a second subset of the set of EDSs from the first DSN memory to the second DSN memory based on the second migration rate.

16. The storage unit of claim 15, wherein the processing module further functions to:

determine to migrate the set of EDSs from the first storage format to the second storage format based on at least one of interpreting a request, identifying a data type, receiving new storage configuration information, detecting a storage efficiency level that compares unfavorably to a threshold level, and detecting an unfavorable performance level.

17. The storage unit of claim 15, wherein the first DSN memory and the second DSN memory are common to the same memory device.

18. The storage unit of claim 15, wherein the at least one of data access request activity or data access response activity comprises at least one of rebuilding encoded data slices, writing encoded data slices, retrieving encoded data slices, listing encoded data slices, performing SU maintenance tasks and performing SU balancing tasks.

19. The storage unit of claim 15, wherein the processing module further functions to:

determine to migrate the set of EDSs from the first storage format to the second storage format based on at least one of detecting a storage inefficiency, receiving an error message, interpreting an error message and detecting an available storage level that compares favorably to an available storage threshold level.

20. The storage unit of claim 15, wherein the processing module further functions to:

select the second storage format for the set of EDSs based on at least one of;

identifying one or more storage formats;

estimating a level of storage performance for each of the one or more storage formats; and

selecting the second storage format that corresponds to a favorable estimated storage performance level.