REDUCING DIGEST STORAGE CONSUMPTION IN A DATA DEDUPLICATION SYSTEM

• US 20140279953A1
• Filed: 03/15/2013
• Published: 09/18/2014
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: ARONOVICH, Lior
• Granted Patent: US 9,678,975 B2
• Time in Patent Office: Days
• US Class Current: 707/692
• CPC Class Codes: G06F 16/137 Hash-based content-based indexing of textual data G06F16/31
• CPC Class Codes: G06F 16/1752 based on file chunks

Abstract

For reducing digests storage consumption in a data deduplication system using a processor device in a computing environment, digest values are calculated for input data. The digest values are used to locate matches with data stored in a repository. The digest values are stored in the repository. The digest values of the data stored in the repository that is determined to be redundant with the input data are removed.

Claims

1. A method for reducing digests storage consumption in a data deduplication system using a processor device in a computing environment, comprising:

calculating digest values for input data;

using the digest values to locate matches with data stored in a repository;

storing the digest values in the repository; and

removing the digest values of the data stored in the repository that is determined to be redundant with the input data.

2. The method of claim 1, further including storing the digest values in the repository linearly in a sequence of occurrence of the digest values in the data.

3. The method of claim 1, further including storing the digest values in the repository in a form that is independent of the form by which the data that the digest values describe is stored.

4. The method of claim 1, further including determining the data stored in the repository, which is used to produce matches with the input data, to be redundant with the input data.

5. The method of claim 1, further including matching digest values of input data with digest values stored in the repository to locate matches with the data stored in the repository.

6. The method of claim 1, further including partitioning the input data into data chunks and grouping the data chunks into chunk sets.

7. The method of claim 6, further including storing the digest values in sets corresponding to the chunk sets, where the digest values sets can be efficiently accessed and removed.

8. The method of claim 1, further including removing digests of redundant repository data to make the digests storage consumption correlative to a factored size of the data stored in the repository rather than to a total data size in the repository.

9. A system for reducing digests storage consumption in a data deduplication system of a computing environment, the system comprising:

the data deduplication system;

a repository operating in the data deduplication system;

at least one processor device operable in the computing storage environment for controlling the data deduplication system, wherein the at least one processor device;

calculates digest values for input data,

uses the digest values to locate matches with data stored in a repository,

stores the digest values in the repository, and

removes the digest values of the data stored in the repository that is determined to be redundant with the input data.

10. The system of claim 9, wherein the at least one processor device stores the digest values in the repository linearly in a sequence of occurrence of the digest values in the data.

11. The system of claim 9, wherein the at least one processor device stores the digest values in the repository in a form that is independent of the form by which the data that the digest values describe is stored.

12. The system of claim 9, wherein the at least one processor device determines the data stored in the repository, which is used to produce matches with the input data, to be redundant with the input data.

13. The system of claim 9, wherein the at least one processor device matches digest values of input data with digest values stored in the repository to locate matches with the data stored in the repository.

14. The system of claim 9, wherein the at least one processor device partitions the input data into data chunks and grouping the data chunks into chunk sets.

15. The system of claim 14, wherein the at least one processor device stores the digest values in sets corresponding to the chunk sets, where the digest values sets can be efficiently accessed and removed

16. The system of claim 9, wherein the at least one processor device removes digests of redundant repository data to make the digests storage consumption correlative to a factored size of the data stored in the repository rather than to a total data size in the repository.

17. A computer program product for reducing digests storage consumption in a data deduplication system using a processor device in a computing environment, the computer program product comprising a computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising:

a first executable portion that calculates digest values for input data;

a second executable portion that uses the digest values to locate matches with data stored in a repository;

a third executable portion that stores the digest values in the repository; and

a fourth executable portion that removes the digest values of the data stored in the repository that is determined to be redundant with the input data.

18. The computer program product of claim 17, further including a fifth executable portion that stores the digest values in the repository linearly in a sequence of occurrence of the digest values in the data.

19. The computer program product of claim 17, further including a fifth executable portion that stores the digest values in the repository in a form that is independent of the form by which the data that the digest values describe is stored.

20. The computer program product of claim 17, further including a fifth executable portion that determines the data stored in the repository, which is used to produce matches with the input data, to be redundant with the input data.

21. The computer program product of claim 17, further including a fifth executable portion that matches digest values of input data with digest values stored in the repository to locate matches with the data stored in the repository.

22. The computer program product of claim 17, further including a fifth executable portion that partitions the input data into data chunks and grouping the data chunks into chunk sets.

23. The computer program product of claim 22, further including a sixth executable portion that stores the digest values in sets corresponding to the chunk sets, where the digest values sets can be efficiently accessed and removed.

24. The computer program product of claim 17, further including a fifth executable portion that removes digests of redundant repository data to make the digests storage consumption correlative to a factored size of the data stored in the repository rather than to a total data size in the repository.