Integrated Approach for Deduplicating Data in a Distributed Environment that Involves a Source and a Target

• US 20110016095A1
• Filed: 07/16/2009
• Published: 01/20/2011
• Est. Priority Date: 07/16/2009
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Van Hise, David G., Yakushev, Mark L., Anglin, Matthew J., Smith, James P., Hochberg, Avishai H., Cannon, David M.
• Granted Patent: US 9,058,298 B2
• Time in Patent Office: Days
• US Class Current: 707/692
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 16/1752 based on file chunks

Abstract

One aspect of the present invention includes a configuration of a storage management system that enables the performance of deduplication activities at both the client (source) and at the server (target) locations. The location of deduplication operations can then be optimized based on system conditions or predefined policies. In one embodiment, seamless switching of deduplication activities between the client and the server is enabled by utilizing uniform deduplication process algorithms and accessing the same deduplication index (containing information on the hashed data chunks). Additionally, any data transformations on the chunks are performed subsequent to identification of the data chunks. Accordingly, with use of this storage configuration, the storage system can find and utilize matching chunks generated with either client- or server-side deduplication.

Claims

1. A method for enabling deduplication of a data file at each of a source and a target location in a distributed storage management system, the storage management system containing a source computing system connected to a target computing system and a target data store located within the target computing system, the method comprising:

maintaining a shared index for tracking deduplicated data chunks stored within the target data store;

providing a deduplication process for deduplication of data chunks to be stored in deduplicated form within the target data store;

enabling execution of deduplication instructions by the target computing system and execution of deduplication instructions by the source computing system;

deduplicating a data file into a set of deduplicated data chunks with use of the deduplication process, the deduplication process comprising a set of deduplication instructions executed by either of the source computing system or the target computing system;

storing the set of deduplicated data chunks within the target data store; and

updating deduplication information for the set of deduplicated data chunks within the shared index.

2. The method of claim 1, wherein the deduplication instructions comprise a variable-size block hashing algorithm used for deduplication on each of the source computing system and the target computing system.

3. The method of claim 1, wherein the deduplication instructions comprise a first set of fingerprinting and hashing algorithms used for deduplication activities executed upon the source computing system, and a second set of fingerprinting and hashing algorithms used for deduplication activities executed on the target computing system.

4. The method of claim 1, further comprising restoring the set of deduplicated data chunks to the source.

5. The method of claim 1, further comprising performing data transformations after fingerprinting and hash identification, the fingerprinting and hash identification occurring within the deduplication process.

6. The method of claim 5, wherein the data transformations include one or more of compression and encryption.

7. The method of claim 5, further comprising undoing the data transformations by the source computing system for each chunk responsive to restoring the set of deduplicated data chunks to the source computing system.

8. The method of claim 5, further comprising undoing the transformations by the target computing system prior to restoring the set of deduplicated data chunks to the source computing system.

9. The method of claim 1, wherein the shared index is shared as a database within the storage management system accessible by each of the source computing system and the target computing system.

10. The method of claim 1, wherein the shared index is accessible via a client-server protocol that enables communication of data chunk information between the source and target.

11. A method of enabling deduplication of a data file at a selected source or target location in a distributed storage management system, the storage management system containing a source computing system connected to a target computing system and a target data store located within the target computing system, the method comprising:

tracking deduplication information for deduplicated data chunks stored within the target data store;

providing a deduplication process for deduplication of a data file to be stored within the target data store;

applying a selected policy from a plurality of defined policies to determine a location for execution of the deduplication process at either of the source computing system or the target computing system;

deduplicating the data file at the determined location by executing the deduplication process; and

updating the tracked deduplication information for the data file.

12. The method of claim 11, wherein the selected policy is selected based on one or more of time, system load, file characteristics, client node, and policy domain.

13. A storage management system, comprising:

a source computing system;

a target computing system connected to the source computing system;

a target data store located within the target computing system;

at least one processor within the storage management system; and

at least one memory within the storage management system storing instructions operable with the at least one processor for enabling deduplication of a data file at each of a source and a target location in the storage management system, the instructions being executed for;

maintaining a shared index for tracking deduplicated data chunks stored within the target data store;

providing a deduplication process for deduplication of data chunks to be stored in deduplicated form within the target data store;

enabling execution of deduplication instructions by the target computing system and execution of deduplication instructions by the source computing system;

deduplicating a data file into a set of deduplicated data chunks with use of the deduplication process, the deduplication process comprising a set of deduplication instructions executed by either of the source computing system or the target computing system;

storing the set of deduplicated data chunks within the target data store; and

updating deduplication information for the set of deduplicated data chunks within the shared index.

14. The storage management system of claim 13, wherein the deduplication instructions comprise a variable-size block hashing algorithm used for deduplication on each of the source computing system and the target computing system.

15. The storage management system of claim 13, wherein the deduplication instructions comprise a first set of fingerprinting and hashing algorithms used for deduplication activities executed upon the source computing system, and a second set of fingerprinting and hashing algorithms used for deduplication activities executed on the target computing system.

16. The storage management system of claim 13, further comprising restoring the set of deduplicated data chunks to the source.

17. The storage management system of claim 13, further comprising instructions being executed for performing data transformations after fingerprinting and hash identification, the fingerprinting and hash identification occurring within the deduplication process, wherein the data transformations include one or more of compression and encryption.

18. The storage management system of claim 13, wherein the shared index is shared as a database within the storage management system accessible by each of the source computing system and the target computing system.

19. A storage management system, comprising:

a source computing system;

a target computing system connected to the source computing system;

a target data store located within the target computing system;

at least one processor within the storage management system; and

at least one memory within the storage management system storing instructions operable with the at least one processor for enabling deduplication of a data file at a selected source or target location in the storage management system, the instructions being executed for;

tracking deduplication information for deduplicated data chunks stored within the target data store;

providing a deduplication process for deduplication of a data file to be stored within the target data store;

applying a selected policy from a plurality of defined policies to determine a location for execution of the deduplication process at either of the source computing system or the target computing system;

deduplicating the data file at the determined location by executing the deduplication process; and

updating the tracked deduplication information for the data file.

20. The storage management system of claim 19, wherein the selected policy is selected based on one or more of time, system load, file characteristics, client node, and policy domain.