System and method for data migration using hybrid modes

• US 8,943,032 B1
• Filed: 09/30/2011
• Issued: 01/27/2015
• Est. Priority Date: 09/30/2011
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Xu, Teng, Hsu, Windsor W., Chin, Lan
• Primary Examiner(s): Chen, Susan
• Application Number:
  US13/250,684
• Time in Patent Office: 1,215 Days
• Field of Search: 707/203, 707/802, 707/899, 711/165, 711/154
• US Class Current: 707/698
• CPC Class Codes: G06F 16/119 Details of migration of file systems migration mechanisms in storage systems G06F3/0647

Abstract

Techniques for data migration of a storage system are described herein. According to one embodiment, a list of files stored in a source storage tier is identified as candidate files to be migrated to a target storage tier based on a migration policy. One of a regular data movement method and a bulk data movement method is selected based on a characteristic of the candidate files. The candidate files are then migrated from the source storage tier to the target storage tier using the selected data movement method.

Claims

1. A computer-implemented method, comprising:

identifying, by a processor, a list of files stored in a source storage tier as candidate files to be migrated to a target storage tier based on a migration policy;

determining by the processor a locality of segments of the candidate files in the source storage tier;

determining by the processor a data compression rate of the segments of the candidate files;

selecting a bulk data movement method in response to determining that at least one of the determined locality and the determined data compression rate of the candidate files satisfies a predetermined condition;

otherwise selecting a regular data movement method; and

migrating the candidate files from the source storage tier to the target storage tier using the selected data movement method, wherein when the bulk data movement method is selected, the segments of the candidate files are identified and transmitted from the source storage tier to the target storage tier as a bulk, and wherein when the regular data movement method is selected, segments of each candidate file are identified and transmitted from the source storage tier to the target storage tier on a file-by-file basis.

2. The method of claim 1, wherein the bulk data movement method is selected if a number of the candidate files is above a first predetermined threshold or a size of the candidate files is above a second predetermined threshold.

3. The method of claim 1, wherein the bulk data movement is selected if the locality of the candidate files indicates that a number of input and output (I/O) requests required to read the segments of the candidate files in a file order is above a first predetermined threshold or the data compression rate of the candidate files is below a second predetermined threshold.

4. The method of claim 1, wherein when the bulk data movement method is utilized, the method further comprises:

generating a candidate index to store information identifying segments of the candidate files; and

migrating segments of the one or more candidate files from the source storage tier to the target storage tier according to a sequential order based on the candidate index.

5. The method of claim 4, wherein all of the candidate files are identified from the source storage tier and indicated in the candidate index before starting migration of segments of the candidate files in a bulk manner.

6. The method of claim 4, wherein generating a candidate index comprises:

for each candidate file, identifying a plurality of segments associated with the candidate file;

for each of the segments of each candidate file, generating a fingerprint of the segment by performing a hash operation on the segment using a predetermined hash function; and

populating a field of the candidate index based on the fingerprint, wherein a value of the field indicates that the segment is to be migrated.

7. The method of claim 6, further comprising transmitting the fingerprint to the target storage tier to allow the target storage tier to determine based on the fingerprint whether the segment has been stored in the target storage tier, wherein the segment is transmitted to the target storage tier only if the segment has not been stored in the target storage tier.

8. The method of claim 6, further comprising estimating whether the target storage tier has enough storage space to store segments of the candidate file, wherein the candidate index is populated only if the target storage tier has enough storage space.

9. The method of claim 1, wherein when the regular data movement method is selected, the method further comprises:

for each candidate file, determining whether each of a plurality of segments of the candidate file has been stored in the target storage tier; and

transmitting one or more segments of the candidate file to the target storage tier that have not been stored in the target storage tier; and

deleting segments of the candidate file from the source storage tier.

10. The method of claim 9, wherein determining whether each of a plurality of segments of the candidate file has been stored in the target storage tier comprises:

transmitting fingerprints of segments of each candidate file to the target storage tier to allow the target storage tier to determine whether the segments have been stored in the target tier; and

receiving from the target storage tier fingerprints of the segments that have not been stored in the target storage tier.

11. A non-transitory computer-readable storage medium having instructions stored therein, which when executed by a computer, cause the computer to perform operations for data migration of a storage system, the operations comprising:

identifying a list of files stored in a source storage tier as candidate files to be migrated to a target storage tier based on a migration policy;

determining a locality of segments of the candidate files in the source storage tier;

determining a data compression rate of the segments of the candidate files;

selecting a bulk data movement method in response to determining that at least one of the determined locality and the determined data compression rate of the candidate files satisfies a predetermined condition;

otherwise selecting a regular data movement method; and

migrating the candidate files from the source storage tier to the target storage tier using the selected data movement method, wherein when the bulk data movement method is selected, the segments of the candidate files are identified and transmitted from the source storage tier to the target storage tier as a bulk, and wherein when the regular data movement method is selected, segments of each candidate file are identified and transmitted from the source storage tier to the target storage tier on a file-by-file basis.

12. The computer-readable storage medium of claim 11, wherein the bulk data movement method is selected if a number of the candidate files is above a first predetermined threshold or a size of the candidate files is above a second predetermined threshold.

13. The computer-readable storage medium of claim 11, wherein the bulk data movement method is selected if the locality of the candidate files indicates that a number of input and output (I/O) requests required to read the segments of the candidate files in file order is above a first predetermined threshold or the data compression rate of the candidate files is below a second predetermined threshold.

14. The computer-readable storage medium of claim 11, wherein when the bulk data movement method is utilized, the operations further comprise:

generating a candidate index to store information identifying segments of the candidate files; and

migrating segments of the one or more candidate files from the source storage tier to the target storage tier according to a sequential order based on the candidate index.

15. The computer-readable storage medium of claim 14, wherein all of the candidate files are identified from the source storage tier and indicated in the candidate index before starting migration of segments of the candidate files in a bulk manner.

16. The computer-readable storage medium of claim 14, wherein generating a candidate index comprises:

for each candidate file, identifying a plurality of segments associated with the candidate file;

for each of the segments of each candidate file, generating a fingerprint of the segment by performing a hash operation on the segment using a predetermined hash function; and

populating a field of the candidate index based on the fingerprint, wherein a value of the field indicates that the segment is to be migrated.

17. The computer-readable storage medium of claim 16, wherein the operations further comprise transmitting the fingerprint to the target storage tier to allow the target storage tier to determine based on the fingerprint whether the segment has been stored in the target storage tier, wherein the segment is transmitted to the target storage tier only if the segment has not been stored in the target storage tier.

18. The computer-readable storage medium of claim 16, wherein the operations further comprise estimating whether the target storage tier has enough storage space to store segments of the candidate file, wherein the candidate index is populated only if the target storage tier has enough storage space.

19. The computer-readable storage medium of claim 11, wherein when the regular data movement method is selected, the operations further comprise:

for each candidate file, determining whether each of a plurality of segments of the candidate file has been stored in the target storage tier; and

transmitting one or more segments of the candidate file to the target storage tier that have not been stored in the target storage tier; and

deleting segments of the candidate file from the source storage tier.

20. The computer-readable storage medium of claim 19, wherein determining whether each of a plurality of segments of the candidate file has been stored in the target storage tier comprises:

transmitting fingerprints of segments of each candidate file to the target storage tier to allow the target storage tier to determine whether the segments have been stored in the target tier; and

receiving from the target storage tier fingerprints of the segments that have not been stored in the target storage tier.

21. A storage system, comprising:

a processor; and

a memory to store instructions, which when executed from the memory, cause the processor to

identify a list of files stored in a source storage tier as candidate files to be migrated to a target storage tier based on a migration policy,

determine a locality of segments of the candidate files in the source storage tier,

determine a data compression rate of the segments of the candidate files,

select a bulk data movement method in response to determining that at least one of the determined locality and the determined data compression rate of the candidate files satisfies a predetermined condition,

otherwise select a regular data movement method, and

migrate the candidate files from the source storage tier to the target storage tier using the selected data movement method, wherein when the bulk data movement method is selected, the segments of the candidate files are identified and transmitted from the source storage tier to the target storage tier as a bulk, and wherein when the regular data movement method is selected, segments of each candidate file are identified and transmitted from the source storage tier to the target storage tier on a file-by-file basis.

22. The system of claim 21, wherein the bulk data movement method is selected if a number of the candidate files is above a first predetermined threshold or a size of the candidate files is above a second predetermined threshold.

23. The system of claim 21, wherein the bulk data movement method is selected if the locality of the candidate files indicates that a number of input and output (I/O) requests required to read the segments of the candidate files in file order is above a first predetermined threshold or the data compression rate of the candidate files is below a second predetermined threshold.

24. The system of claim 21, wherein when the bulk data movement method is utilized a candidate index is generated to store information identifying segments of the candidate files and segments of the one or more candidate files are migrated from the source storage tier to the target storage tier according to a sequential order based on the candidate index.

25. The system of claim 24, wherein all of the candidate files are identified from the source storage tier and indicated in the candidate index before starting migration of segments of the candidate files in a bulk manner.