Managing data deduplication in storage systems

• US 9,449,011 B1
• Filed: 12/28/2012
• Issued: 09/20/2016
• Est. Priority Date: 12/28/2012
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Suliman, Aron A., Chen, Xiangping, Ruef, Richard P., Mullis, Samuel L., Chaudhary, Monica
• Primary Examiner(s): Morrison, Jay
• Assistant Examiner(s): Hoang, Ken
• Application Number:
  US13/730,675
• Time in Patent Office: 1,362 Days
• Field of Search: 707/692, 707/E17.005, 707/E17.007, 707/679, 707/E17.002, 707/640, 707/827, 707/E17.009, 707/661, 707/E17.01, 707/E17.014, 707/609, 707/624, 707/674, 707/748, 707/755, 707/758, 707/E17.032, 707/E17.044, 707/E17.084, 707/610, 707/639, 707/649, 707/651, 707/652, 707/654, 707/667, 707/678, 707/696, 707/723, 707/752, 707/769, 707/781, 707/782, 707/785, 707/802, 707/803, 711/162, 711/E12.103, 711/103, 711/133, 711/154, 711/E12.001, 711/E12.017, 711/118, 711/E12.008, 711/E12.009, 711/E12.019, 711/E12.022, 711/E12.07, 711/E12.071, 711/E12.093, 711/111, 711/114, 711/130, 711/136, 711/156, 711/159, 711/160, 711/163, 711/165, 709/206, 709/204, 709/226, 709/203, 709/221, 709/223, 709/224
• US Class Current: 1/1
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641

Abstract

A method is used in managing data deduplication in storage systems. A data deduplication process is performed by applying a deduplicating technique to data of a deduplication domain. The data deduplication process is scheduled based on a priority. Characteristics of data deduplication performed by the data deduplication process are evaluated. Based on the evaluation, execution of the data deduplication process is effected.

Claims

1. A method for use in managing data deduplication in storage systems, the method comprising:

performing a data deduplication process by applying a deduplicating technique to data of a set of deduplication domains, wherein a set of data deduplication processes is scheduled for the set of deduplication domains, wherein each data deduplication process of the set of data deduplication processes iterates over data of a deduplication domain of the set of deduplication domains based on a respective priority associated with each deduplication domain, wherein each data deduplication process of the set of data deduplication processes iterates over data concurrently with other data deduplication processes, wherein a priority associated with a deduplication domain indicates an amount of time after which a next iteration is scheduled for the deduplication domain;

evaluating characteristics of data deduplication performed on each deduplication domain of the set of deduplication domains by the respective data deduplication process during a previous iteration, wherein evaluating the characteristics of data deduplication includes determining the rate at which data of each deduplication domain is deduplicated and a probability of determining duplicate data blocks in each deduplication domain during a next iteration; and

based on the evaluation, effecting execution of the respective data deduplication process for each deduplication domain, wherein effecting execution of the respective data deduplication process for each deduplication domain includes updating the respective priority associated with each deduplication domain for scheduling the next iteration, wherein updating a priority associated with a deduplication domain includes changing an amount of time after which a next iteration is scheduled for the deduplication domain.

2. The method of claim 1, further comprising:

evaluating the rate at which the data of a deduplication domain of the set of deduplication domains is deduplicated; and

based on the evaluation, adjusting the priority of the data deduplication process after applying the deduplicating technique to the data of the deduplication domain.

3. The method of claim 1, further comprising:

determining the amount of time elapsed since last time the deduplicating technique has been applied to a deduplication domain of the set of deduplication domains; and

based on the determination, adjusting the respective priority of the data deduplication process.

4. The method of claim 1, wherein the data includes a data block, wherein the data block is a fixed size chunk of physical disk storage.

5. The method of claim 1, wherein the deduplication domain comprises a set of storage extents, wherein each storage extent of the set of storage extents comprises a set of LUNs, each LUN of the set of LUNs is a logical representation of a subset of physical disk storage.

6. The method of claim 1, wherein a digest is associated with a data block, wherein the digest is based on contents of the data block and uniquely identifies the data block.

7. The method of claim 1, wherein the deduplicating technique uses an index table, wherein the index table comprises a set of index entries, wherein each index entry of the set of index entries is associated with a digest of a data block.

8. The method of claim 1, wherein applying the deduplicating technique further comprising:

computing a digest for a data block; and

processing the digest for the data block.

9. A system for use in managing data deduplication in storage systems, the system comprising:

a processor configured to;

perform a data deduplication process by applying a deduplicating technique to data of a set of deduplication domains, wherein a set of data deduplication processes is scheduled for the set of deduplication domains, wherein each data deduplication process of the set of data deduplication processes iterates over data of a deduplication domain of the set of deduplication domains based on a respective priority associated with each deduplication domain, wherein each data deduplication process of the set of data deduplication processes iterates over data concurrently with other data deduplication processes, wherein a priority associated with a deduplication domain indicates an amount of time after which a next iteration is scheduled for the deduplication domain;

evaluate characteristics of data deduplication performed on each deduplication domain of the set of deduplication domains by the respective data deduplication process during a previous iteration, wherein evaluating the characteristics of data deduplication includes determining the rate at which data of each deduplication domain is deduplicated and a probability of determining duplicate data blocks in each deduplication domain during a next iteration; and

effect, based on the evaluation, execution of the respective data deduplication process for each deduplication domain, wherein effecting execution of the respective data deduplication process for each deduplication domain includes updating the respective priority associated with each deduplication domain for scheduling the next iteration, wherein updating a priority associated with a deduplication domain includes changing an amount of time after which a next iteration is scheduled for the deduplication domain.

10. The system of claim 9, further comprising:

evaluate the rate at which the data of a deduplication domain of the set of deduplication domains is deduplicated; and

adjust, based on the evaluation, the priority of the data deduplication process after applying the deduplicating technique to the data of the deduplication domain.

11. The system of claim 9, further comprising:

determine the amount of time elapsed since last time the deduplicating technique has been applied to a deduplication domain of the set of deduplication domains; and

adjust, based on the determination, the respective priority of the data deduplication process.

12. The system of claim 9, wherein the data includes a data block, wherein the data block is a fixed size chunk of physical disk storage.

13. The system of claim 9, wherein the deduplication domain comprises a set of storage extents, wherein each storage extent of the set of storage extents comprises a set of LUNs, each LUN of the set of LUNs is a logical representation of a subset of physical disk storage.

14. The system of claim 9, wherein a digest is associated with a data block, wherein the digest is based on contents of the data block and uniquely identifies the data block.

15. The system of claim 9, wherein the deduplicating technique uses an index table, wherein the index table comprises a set of index entries, wherein each index entry of the set of index entries is associated with a digest of a data block.

16. The system of claim 9, wherein applying the deduplicating technique further comprising:

compute a digest for a data block; and

process the digest for the data block.