Selective deduplication

• US 9,348,538 B2
• Filed: 10/18/2012
• Issued: 05/24/2016
• Est. Priority Date: 10/18/2012
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Mallaiah, Damarugendra, Basak, Jayanta
• Primary Examiner(s): VU, BAI DUC
• Application Number:
  US13/655,263
• Publication Number:
  US 20140114932A1
• Time in Patent Office: 1,314 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/2358 Change logging, detection, and notification replication G06F16/27
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

Methods and apparatuses for performing selective deduplication in a storage system are introduced here. Techniques are provided for determining a probability of deduplication for a data object based on a characteristic of the data object and performing a deduplication operation on the data object in the storage system prior to the data object being stored in persistent storage of the storage system if the probability of deduplication for the data object has a specified relationship to a specified threshold.

Claims

1. A method comprising:

determining in a storage system a probability of deduplication for a data object, the probability of deduplication for the data object determined based on a characteristic of the data object, wherein the probability of deduplication for the data object is a statistical projection indicating a likelihood that the data object will provide a storage space benefit as a result of deduplication;

determining in the storage system a deduplication probability threshold, the deduplication probability threshold determined based on a performance metric of the storage system and adjusted based on availability of resources of the storage system and recent performance of the storage system relative to the performance metric;

determining in the storage system whether the probability of deduplication for the data object satisfies the deduplication probability threshold; and

performing a deduplication operation on the data object in the storage system prior to the data object being stored in a persistent storage of the storage system in an event it is determined that the probability of deduplication for the data object satisfies the deduplication probability threshold.

2. The method of claim 1 wherein adjusting the deduplication probability threshold includes reducing the deduplication probability threshold in an event a performance projection for the storage system indicates that the storage system will continue to satisfy the performance metric using the reduced deduplication probability threshold.

3. The method of claim 2 wherein the performance projection is based on the recent performance of the storage system.

4. The method of claim 1 wherein determining the probability of deduplication for the data object based on the characteristic includes determining the probability of deduplication for the data object based on statistical information from prior deduplication operations on other data objects having the characteristic.

5. The method of claim 4 wherein the statistical information indicates a percentage of the other data objects having the characteristic that were deduplication candidates in the prior deduplication operations, wherein a deduplication candidate includes data blocks that are duplicates of other data blocks in the storage system.

6. The method of claim 5 wherein the prior deduplication operations include a plurality of post-processing deduplication operations performed on data objects already stored in the persistent storage of the storage system.

7. The method of claim 4 wherein determining the probability of deduplication for the data object further includes determining a second characteristic of the data object, and determining the probability of deduplication for the data object is further based on statistical information associated with the second characteristic.

8. The method of claim 1 wherein the performance metric is a service level objective (SLO).

9. The method of claim 1 wherein the performance metric includes a response time of the storage system, a throughput of the storage system, a utilization of one or more networks associated with the storage system, or a combination thereof.

10. The method of claim 1 wherein the characteristic is a size of the data object, a type of the data object, an owner of the data object, a last modified date of the data object, an update frequency of the data object, or a combination thereof.

11. The method of claim 1 further comprising performing a post-processing deduplication operation on the data object after the data object is stored in the persistent storage of the storage system in an event it is determined that the probability of deduplication for the data object does not satisfy the deduplication probability threshold.

12. The method of claim 1 wherein the storage system is operated in a Network Attached Storage (NAS) environment or in a Storage Area Network (SAN).

13. A data storage system comprising:

a processor; and

a memory coupled with the processor and including a storage manager that directs the processor to;

determine a deduplication probability threshold, the deduplication probability threshold determined based on a performance metric of the data storage system and adjusted based on availability of resources of the data storage system and recent performance of the data storage system relative to the performance metric;

determine, prior to a data object being stored in a persistent storage, a probability of deduplication for the data object, the probability of deduplication for the data object determined based on a characteristic of the data object, wherein the probability of deduplication for the data object is a statistical projection indicating a likelihood that the data object will provide a storage space benefit as a result of deduplication;

determine whether the probability of deduplication for the data object satisfies the deduplication probability threshold; and

perform a deduplication operation on the data object prior to the data object being stored in the persistent storage in an event it is determined that the probability of deduplication for the data object satisfies the deduplication probability threshold.

14. The data storage system of claim 13 wherein to adjust the deduplication probability threshold includes to reduce the deduplication probability threshold in an event a performance projection for the data storage system based on historical performance information indicates that the data storage system will continue to meet the performance metric using the reduced deduplication probability threshold.

15. The data storage system of claim 13 wherein to determine the probability of deduplication for the data object based on the characteristic includes to:

determine the probability of deduplication for the data object based on statistical information associated with the characteristic, the statistical information aggregated from prior deduplication operations.

16. The data storage system of claim 15 wherein the statistical information associated with the characteristic indicates a percentage of other data objects having the characteristic which were found to include data blocks that were duplicates of other data blocks during the prior deduplication operations.

17. The data storage system of claim 15 wherein:

to determine the probability of deduplication for the data object based on the characteristic further includes to determine a second characteristic of the data object; and

to determine the probability of deduplication for the data object is further based on statistical information associated with the second characteristic.

18. The data storage system of claim 13 wherein the performance metric is a service level objective (SLO).

19. The data storage system of claim 13 wherein the performance metric includes a response time of the data storage system, a throughput of the data storage system, a utilization of a network associated with the data storage system, or a combination thereof.

20. The data storage system of claim 13 wherein the characteristic is a size of the data object, a type of the data object, an owner of the data object, a last modified date of the data object, an update frequency of the data object, or a combination thereof.

21. The data storage system of claim 13 wherein the storage manager further directs the processor to perform a post-processing deduplication operation on the data object after it is stored in the persistent storage in an event it is determined that the probability of deduplication for the data object does not satisfy the deduplication probability threshold.

22. A method of performing selective deduplication comprising:

adaptively adjusting a deduplication probability threshold based on availability of resources in a data storage system and recent performance of the data storage system relative to a performance metric for the data storage system, the deduplication probability threshold determined based on the performance metric;

determining a probability of deduplication for a data object based on-a one or more characteristics of the data object, wherein the probability of deduplication for the data object is a statistical projection indicating a likelihood that the data object will provide a storage space benefit as a result of deduplication;

determining whether the probability of deduplication for the data object exceeds the deduplication probability threshold;

performing a deduplication operation on the data object prior to storing the data object in persistent storage of the data storage system in an event it is determined that the deduplication probability for the data object exceeds the deduplication probability threshold; and

performing the deduplication operation on the data object after storing the data object in the persistent storage in an event it is determined that the deduplication probability for the data object does not exceed the deduplication probability threshold.

23. The method of claim 22 wherein adjusting the deduplication probability threshold includes increasing the deduplication probability threshold in an event the data storage system is not meeting the performance metric.

24. The method of claim 22 wherein determining the probability of deduplication for the data object based on the one or more characteristics includes determining a percentage of other data objects having the one or more characteristics whose physical storage space requirements were reduced through prior deduplication operations.

25. The method of claim 22 wherein the performance metric includes a response time of the data storage system, a throughput of the data storage system, a utilization of one or more networks associated with the data storage system, or a combination thereof.

26. The method of claim 22 wherein the one ore more characteristics include a size of the data object, a type of the data object, an owner of the data object, a last modified date of the data object, an update frequency of the data object, or a combination thereof.

27. The method of claim 22 wherein the storage system is operated in a Network Attached Storage (NAS) environment or in a Storage Area Network (SAN).