Managing deduplication density

• US 8,583,607 B1
• Filed: 03/28/2012
• Issued: 11/12/2013
• Est. Priority Date: 03/28/2012
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Chen, Xiangping, Ruef, Richard P., Armangau, Philippe, Owen, Karl M., Ku, Mark K.
• Primary Examiner(s): Kerzhner, Aleksandr
• Application Number:
  US13/432,276
• Time in Patent Office: 594 Days
• Field of Search: 707/762, 707/999.101
• US Class Current: 707/692
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD

Abstract

A method is used in managing deduplication density. A deduplication density of a data object is determined. The deduplication density is indicated by a number of deduplicated data objects represented by the data object. Based on the deduplication density of the data object, a deduplicating technique is applied to the data object.

Claims

1. A method for use in managing deduplication density, the method comprising:

receiving a request to deduplicate a first data object;

determining a candidate data object for deduplicating the first data object;

based on the determination, evaluating a deduplication density of the candidate data object, wherein the deduplication density is indicated by a number of deduplicated data objects represented by the candidate data object, wherein metadata of the candidate data object includes the deduplication density for the candidate data object;

based on the deduplication density of the candidate data object, skip deduplicating the first data object to the candidate data object; and

based on the deduplication density of the candidate data object, using the first data object for deduplicating additional candidate data objects to the first data object, wherein the first data object is used in addition to the candidate data object for deduplicating the additional candidate data objects.

2. The method of claim 1, further comprising:

based on the deduplication density of the data object, skipping the data object for applying the deduplicating technique.

3. The method of claim 1, wherein a data object is selected from the group consisting of a deduplication domain, a storage extent, a LUN, a file and a data block, wherein the data block is a fixed size chunk of physical disk storage.

4. The method of claim 3, wherein the deduplication domain comprises a set of storage extents, wherein each storage extent of the set of storage extents comprises a set of LUNs, each LUN of the set of LUNs is a logical representation of a subset of physical disk storage.

5. The method of claim 1, wherein a digest is associated with a data object, wherein the digest is based on contents of the data object and uniquely identifies the data object.

6. The method of claim 1, wherein an index table comprises a set of index entries, wherein each index entry of the set of index entries includes a deduplication density of a data object and is associated with a digest of the data object.

7. The method of claim 1, further comprising:

comparing the deduplication density of the data object with a threshold value.

8. The method of claim 1, wherein applying the deduplicating technique further comprising:

computing a digest for a data object; and

processing the digest for the data object.

9. The method of claim 8, wherein the processing of digest for the data object further comprising:

comparing the digest with one or more index entries of an index table;

determining, based on whether the digest matches with an index entry of the index table, whether to apply the deduplicating technique to the data object;

based on the determination, comparing contents of a data object associated with the digest with contents of a data object associated with the matching index entry; and

based on the comparison, applying the deduplicating technique to the data object associated with the digest and the data object associated with the matching index entry.

10. The method of claim 9, further comprising:

based on whether the digest matches with an index entry of the index table, adding an index entry to the index table, wherein the index entry is associated with the digest.

11. A system for use in managing deduplication density, the system comprising:

a processor;

first logic receiving a request to deduplicate a first data object;

second logic determining a candidate data object for deduplicating the first data object;

third logic evaluating, based on the determination, a deduplication density of the candidate data object, wherein the deduplication density is indicated by a number of deduplicated data objects represented by the candidate data object, wherein metadata of the candidate data object includes the deduplication density for the candidate data object;

fourth logic skipping, based on the deduplication density of the candidate data object, deduplicating the first data object to the candidate data object; and

fifth logic using, based on the deduplication density of the candidate data object, the first data object for deduplicating additional candidate data objects to the first data object, wherein the first data object is used in addition to the candidate data object for deduplicating the additional candidate data objects.

12. The system of claim 11, further comprising:

sixth logic skipping, based on the deduplication density of the data object, the data object for applying the deduplicating technique.

13. The system of claim 11, wherein a data object is selected from the group consisting of a deduplication domain, a storage extent, a LUN, a file and a data block, wherein the data block is a fixed size chunk of physical disk storage.

14. The system of claim 13, wherein the deduplication domain comprises a set of storage extents, wherein each storage extent of the set of storage extents comprises a set of LUNs, each LUN of the set of LUNs is a logical representation of a subset of physical disk storage.

15. The system of claim 11, wherein a digest is associated with a data object, wherein the digest is based on contents of the data object and uniquely identifies the data object.

16. The system of claim 11, wherein an index table comprises a set of index entries, wherein each index entry of the set of index entries includes a deduplication density of a data object and is associated with a digest of the data object.

17. The system of claim 11, further comprising:

sixth logic comparing the deduplication density of the data object with a threshold value.

18. The system of claim 11, wherein applying the deduplicating technique further comprising:

sixth logic computing a digest for a data object; and

seventh logic processing the digest for the data object.

19. The system of claim 18, wherein the processing of digest for the data object further comprising:

eighth logic comparing the digest with one or more index entries of an index table;

ninth logic determining, based on whether the digest matches with an index entry of the index table, whether to apply the deduplicating technique to the data object;

tenth logic comparing, based on the determination, contents of a data object associated with the digest with contents of a data object associated with the matching index entry; and

eleventh logic applying, based on the comparison, the deduplicating technique to the data object associated with the digest and the data object associated with the matching index entry.

20. The system of claim 19, further comprising:

twelfth logic adding, based on whether the digest matches with an index entry of the index table, an index entry to the index table, wherein the index entry is associated with the digest.