Selective deduplication

• US 9,753,938 B2
• Filed: 05/23/2016
• Issued: 09/05/2017
• Est. Priority Date: 10/18/2012
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Mallaiah, Damarugendra, Basak, Jayanta
• Primary Examiner(s): VU, BAI DUC
• Application Number:
  US15/162,496
• Publication Number:
  US 20160267098A1
• Time in Patent Office: 470 Days
• Field of Search: None
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/2358 Change logging, detection, and notification replication G06F16/27
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

Methods and apparatuses for performing selective deduplication in a storage system are introduced herein. Techniques are provided for determining deduplication priorities of data objects based upon projected likelihoods that deduplication of such data objects will provide a storage space benefit. Inline deduplication and/or post processing deduplication is selectively provided on an individual data object basis based upon deduplication priorities assigned to respective data objects.

Claims

1. A method comprising:

determining, by a processor of a computing device, a first deduplication priority for a first data object based upon a first projected likelihood that deduplication of the first data object will provide a storage space benefit to reduce storage consumption of a storage device;

determining a second deduplication priority for a second data object based upon a second projected likelihood that deduplication of the second data object will provide the storage space benefit; and

performing inline deduplication for the first data object but not the second data object based upon the first deduplication priority exceeding the second deduplication priority that is indicative of inline deduplication of the first data object having a greater likelihood of achieving the storage space benefit than inline deduplication for the second data object.

2. The method of claim 1, comprising:

performing inline deduplication for the second data object but not the first data object based upon the second deduplication priority exceeding the first deduplication priority.

3. The method of claim 1, comprising:

performing post-processing deduplication for the second data object based upon the first deduplication priority exceeding the second deduplication priority.

4. The method of claim 2, comprising:

performing post-processing deduplication for the first data object based upon the second deduplication priority exceeding the first deduplication priority.

5. The method of claim 1, wherein the performing inline deduplication comprises:

storing the first data object within a temporary storage location; and

performing the inline deduplication upon the first data object to create a deduplicated first data object that is stored into persistent storage.

6. The method of claim 1, wherein the performing inline deduplication comprises:

performing the inline deduplication for the first data object based upon the first deduplication priority exceeding a deduplication probability threshold.

7. The method of claim 6, wherein the performing inline deduplication comprises:

performing post-processing deduplication, and not the inline deduplication, for the first data object based upon the first deduplication priority not exceeding the deduplication probability threshold.

8. The method of claim 3, wherein the performing post-processing deduplication comprises:

storing the second data object to persistent storage; and

performing the post-processing deduplication upon the second data object while stored within the persistent storage.

9. The method of claim 3, wherein the performing post-processing deduplication comprises:

performing the post-processing deduplication based upon a determination that a current system load demand is below a threshold.

10. The method of claim 3, wherein the performing post-processing deduplication comprises:

performing the post-processing deduplication as a background operation.

11. The method of claim 1, wherein the determining a first deduplication priority comprises:

determining the first deduplication priority before the first data object is stored within persistent storage.

12. The method of claim 1, wherein the determining a first deduplication priority comprises:

determining the first deduplication priority based upon a size characteristic of the first data object.

13. The method of claim 1, wherein the determining a first deduplication priority comprises:

determining the first deduplication priority based upon a data object type of the first data object.

14. The method of claim 1, wherein the determining a first deduplication priority comprises:

determining the first deduplication priority based upon a last modified timestamp of the first data object.

15. The method of claim 1, wherein the determining a first deduplication priority comprises:

determining the first deduplication priority based upon an update frequency of the first data object.

16. The method of claim 1, wherein the determining a first deduplication priority comprises:

determining the first deduplication priority based upon statistic information derived from a previous deduplication operation.

17. The method of claim 6, comprising:

determining the deduplication probability threshold based upon at least one of a read throughput, a write throughput, a read response time, a write response time, network utilization, and a performance characteristic.

18. The method of claim 1, comprising:

triggering a comparison of deduplication priorities of data objects for selectively performing inline deduplication based upon a number of data objects, not yet deduplicated, exceeding a threshold.

19. A computing device comprising:

a memory containing machine-readable storage media having stored thereon instructions for performing a method; and

a processor coupled to the memory, the processor configured to execute the instructions to cause the processor to;

determine a first deduplication probability threshold and a second deduplication probability threshold based upon a performance characteristic;

create a multi-level deduplication probability comprising a hierarchical relationship between the first deduplication probability threshold and the second deduplication probability threshold;

utilize the multi-level deduplication probability to assign deduplication priorities to data objects; and

selectively perform inline deduplication but not post processing deduplication for the data objects based upon the deduplication priorities indicating that inline deduplication has a greater likelihood of achieving a storage space benefit to reduce storage consumption of a storage device than post processing deduplication.

20. A non-transitory machine-readable storage media having stored thereon instructions, for performing a method, which causes a computing device to:

determine a first deduplication priority for a first data object based upon a first projected likelihood that deduplication of the first data object will provide a storage space benefit to reduce storage consumption of a storage device;

determine a second deduplication priority for a second data object based upon a second projected likelihood that deduplication of the second data object will provide the storage space benefit;

perform inline deduplication for the first data object and post-processing deduplication for the second data object based upon the first deduplication priority exceeding the second duplication priority that is indicative of inline deduplication of the first data object having a greater likelihood of achieving the storage space benefit than inline deduplication for the second data object; and

perform inline deduplication for the second data object and post-processing duplication for the first data object based upon the second deduplication priority exceeding the first deduplication priority that is indicative of inline deduplication of the second data object having a greater likelihood of achieving the storage space benefit than inline deduplication for the first data object.