Distributed data migration using chunking

• US 9,715,502 B1
• Filed: 03/25/2015
• Issued: 07/25/2017
• Est. Priority Date: 03/25/2015
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Manukonda, Manoj Chakravarthi
• Primary Examiner(s): Stevens, Robert
• Application Number:
  US14/668,211
• Time in Patent Office: 853 Days
• Field of Search: 707827
• CPC Class Codes: G06F 11/14 Error detection or correction of the data by redundancy in operation G06F11/16 takes precedence
• CPC Class Codes: G06F 11/1448 Management of the data involved in backup or backup restore
• CPC Class Codes: G06F 16/119 Details of migration of file systems migration mechanisms in storage systems G06F3/0647
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/24549 Run-time optimisation

Abstract

Techniques are described for migrating data from source storage to destination storage using a chunking technique. The data to be migrated may be divided into a number of chunks according to a factor that is the number of migration modules to be employed in migrating the data. The chunks may be stored on intermediate storage and migrated from the intermediate storage to the destination storage by the migration modules executing in parallel. Because the number of chunks is based on a factor that is the number of migration modules, the utilization of the migration modules may be substantially equivalent during the migration process. The chunks are generated with a size to optimize input/output operations and prevent timeouts during read operations.

Claims

1. A computer-implemented method, comprising:

accessing data stored on a first data storage system;

determining a factor corresponding to a number of migration processes to be employed for copying the data from a second data storage system to a third data storage system;

iteratively dividing the data into data chunks, according to the factor, until a size of the data chunks is included in a size range from a minimum size to a maximum size, wherein;

the minimum size is determined to optimize input/output operations on the second data storage system; and

the maximum size is determined to prevent timeouts of read operations on the second data storage system;

storing the data chunks on the second data storage system; and

employing the migration processes executing in parallel to copy the data chunks from the second data storage system to the third data storage system.

2. The method of claim 1, wherein the factor further corresponds to a number of processors on which the migration processes execute in parallel.

3. The method of claim 1, wherein the data chunks are stored, on the second data storage system, in a storage format supported by the third data storage system.

4. The method of claim 1, wherein iteratively dividing the data into the data chunks further comprises:

during a first iteration, dividing the data into a first number of first data chunks, the first number being the factor;

determining that the size of the first data chunks is greater than the maximum size;

during one or more second iterations, dividing individual ones of the first data chunks into a second number of second data chunks, the second number being the factor to a power that is a number of the one or more second iterations;

determining that the size of the second data chunks is not greater than the maximum size; and

designating the second data chunks as the data chunks to be stored on the second data storage system.

5. A system, comprising:

at least one computing device configured to implement one or more services, the one or more services configured to;

determine a factor corresponding to a number of migration modules to be employed for copying data chunks from an intermediate data storage system to a destination data storage system;

access data stored on a source data storage system;

divide the data into the data chunks, wherein a number of the data chunks is a multiple of the factor, and wherein a size of the data chunks is not greater than a maximum size;

store the data chunks on the intermediate data storage system; and

employ at least a portion of the migration modules in parallel to copy the data chunks from the intermediate data storage system to the destination data storage system.

6. The system of claim 5, wherein the one or more services are further configured to:

determine the maximum size to prevent timeouts of read operations on the intermediate data storage system.

7. The system of claim 5, wherein the size of the data chunks is at least a minimum size that is determined to optimize input/output operations on the intermediate data storage system.

8. The system of claim 7, wherein:

the minimum size is 1 megabyte; and

the maximum size is 800 megabytes.

9. The system of claim 5, wherein dividing the data into the data chunks further comprises iteratively dividing the data into the data chunks according to the factor, until a size of the data chunks is not greater than the maximum size.

10. The system of claim 9, wherein the number of the data chunks is the factor to a power that is a number of iterations performed to divide the data into the data chunks.

11. The system of claim 5, wherein:

the migration modules are executed on one or more processors; and

the maximum size is based on one or more of a speed or a memory capacity of the one or more processors.

12. The system of claim 11, wherein the one or more processors are incorporated into the destination data storage system.

13. The system of claim 5, wherein the data chunks are stored, on the intermediate data storage system, in a storage format supported by the destination data storage system.

14. The system of claim 5, wherein the factor further corresponds to a number of processors on which the migration modules execute in parallel.

15. One or more computer-readable media storing instructions which, when executed by at least one processor, instruct the at least one processor to:

determine a factor corresponding to a number of migration modules to be employed for copying data chunks from an intermediate data storage system to a destination data storage system;

access data to be stored on the destination data storage system;

generate the data chunks by iteratively dividing the data, according to the factor, until a size of the data chunks is not greater than a maximum size;

store the data chunks on the intermediate data storage system; and

employ at least a portion of the migration modules in parallel to copy the data chunks from the intermediate data storage system to the destination data storage system.

16. The one or more computer-readable media of claim 15, wherein generating the data chunks further comprises:

during a first iteration, dividing the data into a first number of first data chunks, the first number being the factor;

determining that the size of the first data chunks is greater than the maximum size;

during a second iteration, dividing individual ones of the first data chunks into a second number of second data chunks, the second number being the factor;

determining that the size of the second data chunks is not greater than the maximum size; and

designating the second data chunks as the data chunks to be stored on the intermediate data storage system.

17. The one or more computer-readable media of claim 15, wherein the data chunks are stored, on the intermediate data storage system, in a storage format supported by the destination data storage system.

18. The one or more computer-readable media of claim 15, wherein the data chunks are stored in a compressed format on the intermediate data storage system.

19. The one or more computer-readable media of claim 15, wherein the instructions further instruct the at least one processor to:

determine the maximum size to prevent timeouts of read operations on the intermediate data storage system.

20. The one or more computer-readable media of claim 15, wherein the size of the data chunks is at least a minimum size that is determined to optimize input/output operations on the intermediate data storage system.