SELECTIVE DEDUPLICATION

• US 20140114932A1
• Filed: 10/18/2012
• Published: 04/24/2014
• Est. Priority Date: 10/18/2012
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Mallaiah, Damarugendra, Basak, Jayanta
• Granted Patent: US 9,348,538 B2
• Time in Patent Office: Days
• US Class Current: 707/692
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/2358 Change logging, detection, and notification replication G06F16/27
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

Methods and apparatuses for performing selective deduplication in a storage system are introduced here. Techniques are provided for determining a probability of deduplication for a data object based on a characteristic of the data object and performing a deduplication operation on the data object in the storage system prior to the data object being stored in persistent storage of the storage system if the probability of deduplication for the data object has a specified relationship to a specified threshold.

Claims

1. A method comprising:

determining in a storage system a probability of deduplication for a data object based on a characteristic of the data object; and

performing a deduplication operation on the data object in the storage system prior to the data object being stored in persistent storage of the storage system in response to the probability of deduplication for the data object having a specified relationship to a specified threshold.

2. The method of claim 1 wherein the specified threshold is a deduplication probability threshold that is determined based on a performance metric of the storage system.

3. The method of claim 2 wherein determining the deduplication probability threshold based on the performance metric includes repeatedly adjusting the deduplication probability threshold based on availability of resources of the storage system and recent performance of the storage system relative to the performance metric.

4. The method of claim 3 wherein adjusting the deduplication probability threshold includes reducing the deduplication probability threshold if a performance projection for the storage system indicates that the storage system will continue to satisfy the performance metric using the reduced deduplication probability threshold.

5. The method of claim 4 wherein the performance projection is based on the recent performance of the storage system.

6. The method of claim 1 wherein determining the probability of deduplication for the data object based on the characteristic includes determining the probability of deduplication based on statistical information from a prior deduplication operation on other data objects having the characteristic.

7. The method of claim 6 wherein the statistical information indicates a percentage of the other data objects having the characteristic that were deduplication candidates in the prior deduplication operation, wherein a data object is a deduplication candidate if it includes data blocks that are duplicates of other data blocks in the storage system.

8. The method of claim 7 wherein the prior deduplication operation includes a plurality of post-processing deduplication operations performed on data objects already stored in the persistent storage of the storage system.

9. The method of claim 6 wherein determining the probability of deduplication for the data object further includes determining a second characteristic of the data object and determining the probability of deduplication is further based on statistical information associated with the second characteristic.

10. The method of claim 2 wherein the performance metric is a service level objective (SLO).

11. The method of claim 2 wherein the performance metric includes one or more of:

a response time of the storage system, a throughput of the storage system, or a utilization of one or more networks associated with the storage system.

12. The method of claim 1 wherein the characteristic is one or more of:

a size of the data object, a type of the data object, an owner of the data object, a last modified date of the data object, or an update frequency of the data object.

13. The method of claim 1 further comprising performing a post-processing deduplication operation on the data object after the data object is stored in the persistent storage of the storage system if the probability of deduplication for the data object does not have the specified relationship to the specified threshold.

14. The method of claim 1 wherein the storage system is operated in a Network Attached Storage (NAS) environment or in a Storage Area Network (SAN).

15. A data storage system comprising:

persistent storage;

a processor; and

a memory coupled with the processor and including a storage manager that directs the processor to;

determine a deduplication probability threshold based on a performance metric of the data storage system;

determine, prior to a data object being stored in the persistent storage, a probability of deduplication for the data object based on a factor associated with the data object;

perform a deduplication operation on the data object prior to the data object being stored in the persistent storage if the probability of deduplication for the data object exceeds the deduplication probability threshold; and

store the data object in the persistent storage.

16. The data storage system of claim 15 wherein to determine the deduplication probability threshold based on the performance metric includes to periodically adjust the deduplication probability threshold based on availability of resources of the data storage system and recent performance of the data storage system relative to the performance metric.

17. The data storage system of claim 16 wherein to adjust the deduplication probability threshold includes to reduce the deduplication probability threshold if a performance projection for the data storage system based on historical performance information indicates that the data storage system will continue to meet the performance metric using the reduced deduplication probability threshold.

18. The data storage system of claim 15 wherein to determine the probability of deduplication for the data object based on the factor includes to:

determine a characteristic of the data object associated with the factor; and

determine the probability of deduplication based on statistical information associated with the characteristic, the statistical information aggregated from a prior deduplication operation.

19. The data storage system of claim 18 wherein the statistical information associated with the characteristic indicates a percentage of other data objects having the characteristic which were found to include data blocks that were duplicates of other data blocks in the data storage system as determined by the prior deduplication operation.

20. The data storage system of claim 18 wherein:

to determine the probability of deduplication for the data object based on the factor further includes to determine a second characteristic of the data object; and

to determine the probability of deduplication is further based on statistical information associated with the second characteristic.

21. The data storage system of claim 15 wherein the performance metric is a service level objective (SLO).

22. The data storage system of claim 15 wherein the performance metric includes one or more of:

a response time of the data storage system, a throughput of the data storage system, or a utilization of a network associated with the data storage system.

23. The data storage system of claim 15 wherein the factor includes one or more of:

a size of the data object, a type of the data object, an owner of the data object, a last modified date of the data object, or an update frequency of the data object.

24. The data storage system of claim 15 wherein the storage manager further directs the processor to perform a post-processing deduplication operation on the data object after it is stored if the probability of deduplication for the data object does not exceed the deduplication probability threshold.

25. A method of performing selective deduplication comprising:

adaptively adjusting a deduplication probability threshold based on availability of resources in a data storage system and recent performance of the data storage system relative to a performance metric for the data storage system;

determining a deduplication probability for a data object based on a characteristic of the data object;

performing a deduplication operation on the data object prior to storing the data object in persistent storage of the data storage system if the deduplication probability for the data object exceeds the deduplication probability threshold; and

performing the deduplication operation on the data object after storing the data object in the persistent storage if the deduplication probability for the data object does not exceed the deduplication probability threshold.

26. The method of claim 25 wherein adjusting the deduplication probability threshold includes increasing the deduplication probability threshold if the data storage system is not meeting the performance metric.

27. The method of claim 25 wherein to determine the probability of deduplication for the data object based on the one or more characteristics includes determining a percentage of other data objects having the characteristics whose physical storage space requirements were reduced through one or more prior deduplication operations.

28. The method of claim 25 wherein the performance metric includes one or more of:

a response time of the data storage system, a throughput of the data storage system, or a utilization of one or more networks associated with the data storage system.

29. The method of claim 25 wherein the characteristic is associated with one or more of:

a size of the data object, a type of the data object, an owner of the data object, a last modified date of the data object, or an update frequency of the data object.

30. The method of claim 25 wherein the storage system is operated in a NAS environment or in a SAN.