Efficient storage tiering

• US 9,280,550 B1
• Filed: 12/31/2010
• Issued: 03/08/2016
• Est. Priority Date: 12/31/2010
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Hsu, Windsor W., Xu, Teng
• Primary Examiner(s): Trujillo, James
• Assistant Examiner(s): Le, Jessica N
• Application Number:
  US12/983,281
• Time in Patent Office: 1,894 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 16/1744 using compression, e.g. sparse files
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/185 Hierarchical storage management [HSM] systems, e.g. file migration or policies thereof details of archiving G06F16/11
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0649 Lifecycle management
• CPC Class Codes: G06F 3/0685 Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays

Abstract

A method, article of manufacture, and apparatus for transferring data from a source tier to a target tier. In some embodiments, this may include dividing an object stored in the source tier into a plurality of segments, determining if the target tier is missing at least one of the segments, writing the missing segments to the target tier, and deleting the object from the source tier. In some embodiments, the object'"'"'s metadata is updated to indicate that the object is in the target tier.

Claims

1. A method for transferring data from a source tier to a target tier in a data system, the method comprising:

storing a first object in the source tier, wherein storing the first object in the source tier includes dividing the first object into a first plurality of segments using a first segment technique and compressing the first plurality of segments using a compression technique of the source tier, and wherein the source tier deduplicates data based at least in part on the first segment technique;

determining, using a processor, whether to transfer the first object from the source tier to the target tier, wherein the target tier deduplicates data based at least in part on a second segment technique that is different from the first segment technique used at the source tier, wherein the determination to transfer the first object is based at least in part on one or more policies for determining whether to transfer an object and a tier including at least one of the source tier and the target tier to which to transfer the object;

decompressing the first plurality of segments using the compression technique of the source tier;

re-segmenting the segmented first object comprising the first plurality of segments into a second plurality of segments based at least in part on the second segment technique of the target tier;

transmitting to the target tier a representation of a first segment from the second plurality of segments;

receiving from the target tier a response indicating that the first segment is not already present at the target tier, wherein the target tier determines based at least in part on the representation of first segment whether the first segment is already present at the target tier;

compressing the first segment using a compression technique of the target tier;

transferring the compressed first segment to the target tier in the event that the first tier is determined to be missing at the target tier; and

deleting at least one of the first plurality of segments related to the first object from the source tier.

2. The method as recited in claim 1, further comprising updating metadata of the first object to indicate that the first object is in the target tier.

3. The method as recited in claim 1, further comprising verifying that the first object was written to the target tier.

4. The method as recited in claim 1, further comprising compressing the missing segments written to the target tier according to a target tier compression technique.

5. The method as recited in claim 1, wherein the one or more policies are based at least in part on an access frequency or access time associated with the first object, and a respective performance and storage economy of the source tier and the target tier.

6. The method as recited in claim 1, further comprising computing a representation of each of at least some of the second plurality of segments.

7. The method as recited in claim 1, wherein the representation of the first segment is a fingerprint of the first segment.

8. The method as recited in claim 7, wherein the fingerprint of the first segment is a hash value generated based at least in part on a content of the first segment.

9. The method as recited in claim 7, wherein the target tier determines whether it is missing the first segment at least in part by looking up a fingerprint index for the fingerprint of the first segment.

10. The method of claim 1, further comprising:

verifying whether the first object was properly written to the target tier,

wherein the deleting of the at least one of the plurality of segments related to the first object from the source tier comprises deleting the first object from the source tier in response to verification that the first object was properly written to the target tier.

11. A system for transferring data in a data system, the system comprising a source tier, a target tier, and a processor configured to:

store a first object in the source tier, wherein storing the first object in the source tier includes dividing the first object into a first plurality of segments using a first segment technique and compressing the first plurality of segments using a compression technique of the source tier, and wherein the source tier deduplicates data based at least in part on the first segment technique;

determine whether to transfer the first object from the source tier to the target tier, wherein the target tier deduplicates data based at least in part on a second segment technique that is different from the first segment technique used at the source tier, wherein the determination to transfer the first object is based at least in part on one or more policies for determining whether to transfer an object and a tier including at least one of the source tier and the target tier to which to transfer the object;

decompress the first plurality of segments using the compression technique of the source tier;

re-segmenting the segmented first object comprising the first plurality of segments into a second plurality of segments based at least in part on the second segment technique of the target tier;

transmit to the target tier a representation of a first segment from the second plurality of segments;

receive from the target tier a response indicating that the first segment is not already present at the target tier, wherein the target tier determines based at least in part on the representation of first segment whether the first segment is already present at the target tier;

compress the first segment using a compression technique of the target tier;

transfer the compressed first segment to the target tier in the event that it is determined to be missing at the target tier; and

delete at least one of the plurality of segments related to the first object from the source tier.

12. The system as recited in claim 11, the processor further configured update metadata of the first object to indicate that the first object is in the target tier.

13. The system as recited in claim 11, the processor further configured to verify that the first object was written to the target tier.

14. The system as recited in claim 11, the processor further configured to compress the missing segments written to the target tier according to a target tier compression technique.

15. The system as recited in claim 11, wherein the one or more policies are based at least in part on an access frequency or access time associated with the first object, and a respective performance and storage economy of the source tier and the target tier.

16. The system as recited in claim 11, further comprising computing a representation of each of at least some of the second plurality of segments.

17. The system as recited in claim 11, wherein the representation of the first segment is a fingerprint of the first segment.

18. The system as recited in claim 17, wherein the fingerprint of the first segment is a hash value generated based at least in part on a content of the first segment.

19. The system as recited in claim 17, wherein the target tier determines whether it is missing the first segment at least in part by looking up a fingerprint index for the fingerprint of the first segment.

20. The system of claim 11, wherein the processor is further configured to:

verify whether the first object was properly written to the target tier,

wherein the deleting of the first object from the source tier comprises deleting the first object from the source tier in response to verification that the first object was properly written to the target tier.

21. A computer program product for transferring data from a source tier to a target tier, comprising a non-transitory computer-readable storage medium having machine readable code embodied therein for:

storing a first object in the source tier, wherein storing the first object in the source tier includes dividing the first object into a first plurality of segments using a first segment technique and compressing the first plurality of segments using a compression technique of the source tier, and wherein the source tier deduplicates data based at least in part on a first segment technique;

determining whether to transfer the first object from the source tier to the target tier, wherein the target tier deduplicates data based at least in part on a second segment technique that is different from the first segment technique used at the source tier, wherein the determination to transfer the first object is based at least in part on one or more policies for determining whether to transfer an object and a tier including at least one of the source tier and the target tier to which to transfer the object;

decompressing the first plurality of segments using the compression technique of the source tier;

re-segmenting the segmented first object comprising the first plurality of segments into a second plurality of segments based at least in part on the second segment technique of the target tier;

transmitting to the target tier a representation of a first segment from the second plurality of segments;

receiving from the target tier a response indicating that the first segment is not already present at the target tier, wherein the target tier determines based at least in part on the representation of first segment whether the first segment is already present at the target tier;

compressing the first segment using a compression technique of the target tier;

transferring the compressed first segment to the target tier in the event that the first tier is determined to be missing at the target tier.

22. The computer program product as recited in claim 21, further comprising updating metadata of the first object to indicate that the first object is in the target tier.

23. The computer program product as recited in claim 21, further comprising verifying that the first object was written to the target tier.

24. The computer program product as recited in claim 21, further comprising compressing the missing segments written to the target tier according to a target tier compression technique.

25. The computer program product as recited in claim 21, wherein the one or more policies are based at least in part on an access frequency or access time associated with the first object, and a respective performance and storage economy of the source tier and the target tier.

26. The computer program product as recited in claim 21, further comprising computing a representation of each of at least some of the second plurality of segments.

27. The computer program product as recited in claim 21, wherein the representation of the first segment is a fingerprint of the first segment.

28. The computer program product as recited in claim 27, wherein the fingerprint of the first segment is a hash value generated based at least in part on a content of the first segment.

29. The computer program product as recited in claim 27, wherein the target tier determines whether it is missing the first segment at least in part by looking up a fingerprint index for the fingerprint of the first segment.