REPLICATION OF DEDUPLICATED DATA

• US 20110040728A1
• Filed: 08/11/2009
• Published: 02/17/2011
• Est. Priority Date: 08/11/2009
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: ARONOVICH, Lior, ASHER, Ron, ISH-SHALOM, Ariel J., LENEMAN, Ofer, AKIRAV, Shay H., BACHAR, Yariv
• Granted Patent: US 8,356,017 B2
• Time in Patent Office: Days
• US Class Current: 707/634
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 16/275 Synchronous replication
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/065 Replication mechanisms

Abstract

Various embodiments for replicating deduplicated data using a processor device are provided. A block of the deduplicated data, created in a source repository, is assigned a global block identification (id) unique in a grid set inclusive of the source repository. The global block id is generated using at least one unique identification value of the block, a containing grid of the grid set, and the source repository. The global block id is transmitted from the source repository to a target repository. If the target repository determines the global block id is associated with an existing block of the deduplicated data located within the target repository, the block is not received by the target repository during a subsequent replication process.

Claims

1. A method for replicating deduplicated data using a processor device, comprising:

assigning a block of the deduplicated data, created in a source repository, a global block identification (id) unique in a grid set inclusive of the source repository, wherein the global block id is generated using at least one unique identification value of the block, a containing grid of the grid set, and the source repository; and

transmitting the global block id from the source repository to a target repository, wherein if the target repository determines the global block id is associated with an existing block of the deduplicated data located within the target repository, the block is not received by the target repository during a subsequent replication process.

2. The method of claim 1, further including, subsequent to the transmitting the global block id, transmitting, for the block, a representation enabling data deduplication from the source repository to the target repository, wherein the representation is inserted into a deduplication index within the target repository.

3. The method of claim 1, wherein the assigning the global id further includes performing at least one of:

assigning a grid identification (id), unique in the containing grid of the grid set, wherein the grid id is computed by performing one of selecting a previously nonexistent grid id and manually assigning the grid id,

upon assignment of the source repository to the containing grid of the grid set, assigning the source repository a repository identification (id), unique in the containing grid, wherein the repository id is computed by performing one of selecting a previously nonexistent repository id and manually assigning the repository id, and

assigning a block identification (id), unique the containing grid of the grid set, wherein the block id is computed by selecting a previously nonexistent block id.

4. The method of claim 3, wherein at least one of:

the selecting the previously nonexistent grid id includes incrementing a grid id variable corresponding to the grid set,

the selecting the previously nonexistent repository id includes incrementing a repository id variable corresponding to the containing grid, and

the selecting the previously nonexistent block id includes incrementing a block id variable corresponding to the source repository.

5. The method of claim 4, further including generating the global block id by combining the grid id, the repository id, and the block id.

6. The method of claim 1, further including, upon receipt of incoming data within the source repository:

deduplicating the incoming data with existing data in the source repository, partitioning the incoming data into a plurality of existing blocks and a plurality of new blocks of the deduplicated data, wherein the block is one of the plurality of the new blocks, each of the plurality of existing blocks having a reference count incremented to reflect the receipt of the incoming data,

in addition to the assigning the block the global block id, assigning a plurality of additional global block id s to each of a remaining plurality of new blocks, wherein each of the block, the global block id, the plurality of the new blocks, and the plurality of the additional global block id s are stored within the source repository, and

recording a mapping of sections of the incoming data to the plurality of the existing blocks and the plurality of the new blocks in the source repository.

7. The method of claim 1, further including, pursuant to the transmitting the global block id, recording, by the source repository, the at least one unique identification value within a global block id index.

8. The method of claim 1, further including by the target repository, pursuant to the transmitting the global block id, searching for an existence of the global block id using the deduplication index, wherein if the global block id is determined to be nonexistent within the target repository:

transmitting the global block id from the target repository to the source repository, and

transmitting data associated with the block from the source repository to the target repository.

9. The method of claim 8, further including by the target repository, subsequent to the transmitting the data associated with the block, storing the data associated with the block within the target repository, and incrementing a reference count associated with the block.

10. The method of claim 8, further including, if the global block id is determined to be existent within the target repository, incrementing a reference count associated with the block.

11. The method of claim 8, further including by the target repository, subsequent to the transmitting the global block id, performing at least one of:

recording the global block id within a global block id index, and

recording a mapping of sections of replicated data to the block and a plurality of additional blocks of the deduplicated data.

12. The method of claim 11, further including by the target repository, performing at least one of:

retrieving the mapping, and

loading at least one of the block and the plurality of the additional blocks into a memory location.

13. A system for replicating deduplicated data, comprising:

at least one processor device operable in a computing storage environment, the at least one processor in communication with a source repository and a target repository, wherein the at least one processor device is adapted for;

assigning a block of the deduplicated data, created in a source repository, a global block identification (id) unique in a grid set inclusive of the source repository, wherein the global block id is generated using at least one unique identification value of the block, a containing grid of the grid set, and the source repository, and

transmitting the global block id from the source repository to a target repository, wherein if the target repository determines the global block id is associated with an existing block of the deduplicated data located within the target repository, the block is not received by the target repository during a subsequent replication process.

14. The system of claim 13, wherein the at least one processor device is further adapted for, subsequent to the transmitting the global block id, transmitting, for the block, a representation enabling data deduplication from the source repository to the target repository, wherein the representation is inserted into a deduplication index within the target repository.

15. The system of claim 13, wherein the at least one processor device is further adapted for, pursuant to the assigning the global id, performing at least one of:

assigning a grid identification (id), unique in the containing grid of the grid set, wherein the grid id is computed by performing one of selecting a previously nonexistent grid id and manually assigning the grid id,

upon assignment of the source repository to the containing grid, assigning the source repository a repository identification (id), unique in the containing grid of the grid set, wherein the repository id is computed by performing one of selecting a previously nonexistent repository id and manually assigning the repository id, and

assigning a block identification (id), unique in the containing grid of the grid set, wherein the block id is computed by selecting a previously nonexistent block id.

16. The system of claim 15, wherein at least one of:

the selecting the previously nonexistent grid id includes incrementing a grid id variable corresponding to the grid set,

the selecting the previously nonexistent repository id includes incrementing a repository id variable corresponding to the containing grid, and

the selecting the previously nonexistent block id includes incrementing a block id variable corresponding to the source repository.

17. The system of claim 16, wherein the at least one processor device is further adapted for generating the global block id by combining the grid id, the repository id, and the block id.

18. The system of claim 14, wherein the at least one processor device is further adapted for, upon receipt of incoming data within the source repository:

deduplicating the incoming data with existing data in the source repository, partitioning the incoming data into a plurality of existing blocks and a plurality of new blocks of the deduplicated data, wherein the block is one of the plurality of the new blocks, each of the plurality of existing blocks having a reference count incremented to reflect the receipt of the incoming data,

in addition to the assigning the block the global block id, assigning a plurality of additional global block id s to each of a remaining plurality of new blocks, wherein each of the block, the global block id, the plurality of the new blocks, and the plurality of the additional global block id s are stored within the source repository, and

recording a mapping of sections of the incoming data to the plurality of the existing blocks and the plurality of the new blocks in the source repository.

19. The system of claim 14, wherein the source repository and the at least one processor device are further adapted for, pursuant to the transmitting the global block v, recording the at least one unique identification value within a global block id index.

20. The system of claim 14, wherein the target repository and the at least one processor device are further adapted for, pursuant to the transmitting the global block id, searching for an existence of the global block id using the deduplication index, wherein if the global block id is determined to be nonexistent within the target repository:

transmitting the global block id from the target repository to the source repository, and

transmitting data associated with the block from the source repository to the target repository.

21. The system of claim 20, wherein the target repository and the at least one processor device are further adapted for, subsequent to the transmitting the data associated with the block, storing the data associated with the block within the target repository, and incrementing a reference count associated with the block.

22. The system of claim 20, wherein the target repository and the at least one processor device are further adapted for, if the global block id is determined to be existent within the target repository, incrementing a reference count associated with the block.

23. The system of claim 20, wherein the target repository and the at least one processor device are further adapted for, subsequent to the transmitting the global block ID, performing at least one of:

recording the global block id within a global block id index, and

recording a mapping of sections of replicated data to the block and a plurality of additional blocks of the deduplicated data.

24. The system of claim 23, wherein the target repository and the at least one processor device are further adapted for, performing at least one of:

retrieving the mapping, and

loading at least one of the block and the plurality of the additional blocks into a memory location.

25. A computer program product for replicating deduplicated data using a processor device, the computer program product comprising a computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising:

a first executable portion for assigning a block of the deduplicated data, created in a source repository, a global block identification (id) unique in a grid set inclusive of the source repository, wherein the global block id is generated using at least one unique identification value of the block, a containing grid of the grid set, and the source repository; and

a second executable portion for transmitting the global block id from the source repository to a target repository, wherein if the target repository determines the global block id is associated with an existing block of the deduplicated data located within the target repository, the block is not received by the target repository during a subsequent replication process.

26. The computer program product of claim 25, further including a third executable portion for, subsequent to the transmitting the global block id, transmitting, for the block, a representation enabling data deduplication from the source repository to the target repository, wherein the representation is inserted into a deduplication index within the target repository.

27. The computer program product of claim 25, further including a third executable portion for, pursuant to the assigning the global id, performing at least one of:

assigning a grid identification (id), unique in the containing grid of the grid set, wherein the grid id is computed by performing one of selecting a previously nonexistent grid id and manually assigning the grid id,

upon assignment of the source repository to the containing grid, assigning the source repository a repository identification (id), unique in the containing grid, wherein the repository id is computed by performing one of selecting a previously nonexistent repository id and manually assigning the repository id, and

assigning a block identification (id), unique in the source repository, wherein the block id is computed by selecting a previously nonexistent block id.

28. The computer program product of claim 27, wherein at least one of:

the selecting the previously nonexistent grid id includes incrementing a grid id variable corresponding to the grid set,

the selecting the previously nonexistent repository id includes incrementing a repository id variable corresponding to the containing grid, and

the selecting the previously nonexistent block id includes incrementing a block id variable corresponding to the source repository.

29. The computer program product of claim 28 further including a fourth executable portion for generating the global block id by combining the grid id, the repository id, and the block id.

30. The computer program product of claim 26, further including a third executable portion for, upon receipt of incoming data within the source repository:

deduplicating the incoming data with existing data in the source repository, partitioning the incoming data into a plurality of existing blocks and a plurality of new blocks of the deduplicated data, wherein the block is one of the plurality of the new blocks, each of the plurality of existing blocks having a reference count incremented to reflect the receipt of the incoming data,

in addition to the assigning the block the global block id, assigning a plurality of additional global block ids to each of a remaining plurality of new blocks, wherein each of the block, the global block id, the plurality of the new blocks, and the plurality of the additional global block id s are stored within the source repository, and

recording a mapping of sections of the incoming data to the plurality of the existing blocks and the plurality of the new blocks in the source repository.

31. The computer program product of claim 26, further including a third executable portion for, pursuant to the transmitting the global block id, recording, by the source repository, the at least one unique identification value within a global block id index.

32. The computer program product of claim 26, further including a third executable portion for, by the target repository, pursuant to the transmitting the global block id, searching for an existence of the global block id using the deduplication index, wherein if the global block id is determined to be nonexistent within the target repository:

transmitting the global block id from the target repository to the source repository, and

transmitting data associated with the block from the source repository to the target repository.

33. The computer program product of claim 32, further including a fourth executable portion for, by the target repository, subsequent to the transmitting the data associated with the block, storing the data associated with the block within the target repository, and incrementing a reference count associated with the block.

34. The computer program product of claim 32, further including a fourth executable portion for, if the global block id is determined to be existent within the target repository, incrementing a reference count associated with the block.

35. The computer program product of claim 32, further including a fourth executable portion for, by the target repository, subsequent to the transmitting the global block id, performing at least one of:

recording the global block id within a global block id index, and

recording a mapping of sections of replicated data to the block and a plurality of additional blocks of the deduplicated data.

36. The computer program product of claim 35, further including a fifth executable portion for, by the target repository, performing at least one of:

retrieving the mapping, and

loading at least one of the block and the plurality of the additional blocks into a memory location.

37. A device, comprising:

a circuit customized for performing steps of a method for replicating deduplicated data, the steps including;

assigning a block of the deduplicated data, created in a source repository, a global block identification (id) unique in a grid set inclusive of the source repository, wherein the global block id is generated using at least one unique identification value of the block, a containing grid of the grid set, and the source repository, and

transmitting the global block id from the source repository to a target repository, wherein if the target repository determines the global block id is associated with an existing block of the deduplicated data located within the target repository, the block is not received by the target repository during a subsequent replication process.

38. The device of claim 37, wherein the steps further include, subsequent to the transmitting the global block id, transmitting, for the block, a representation enabling data deduplication from the source repository to the target repository, further wherein the representation is inserted into a deduplication index within the target repository.

39. The device of claim 37, wherein the circuit includes one of an application-specific integrated circuit (ASIC), system-on-chip (SoC), and a field programmable gate array (FPGA).

40. A method for replicating deduplicated data using a processor device, comprising:

performing at least one of;

assigning a grid identification (id), unique in a containing grid of a grid set, wherein the grid id is computed by performing one of selecting a previously nonexistent grid id and manually assigning the grid id,

upon assignment of the source repository to the containing grid of the grid set, assigning a source repository of the containing grid a repository identification (id), unique in the containing grid of the grid set, wherein the repository id is computed by performing one of selecting a previously nonexistent repository id and manually assigning the repository id, and

assigning a block identification (id), unique in the containing grid of the grid set, wherein the block id is computed by selecting a previously nonexistent block id; and

storing at least one of the grid id, the repository id, and the block id as metadata in an identification file.

41. The method of claim 40, further including storing at least one internet protocol (IP) address associated with at least one of the containing grid and the source repository as the metadata along with the at least one of the grid id, the repository id, and the block id in the identification file.

42. The method of claim 41, further including generating a global block identification (id), unique in the source repository and a target repository, by combining the grid id, the repository id, and the block id.

43. The method of claim 42, further including:

transmitting the global block id from the source repository to the target repository, wherein if the target repository determines the global block id is associated with an existing block of the deduplicated data located within the target repository, the block is not received by the target repository during a subsequent replication process.

44. The method of claim 43, further including, subsequent to the transmitting the global block id, transmitting, for a block associated with the block id, a representation enabling data deduplication from the source repository to the target repository, wherein the representation is inserted into a deduplication index within the target repository.