Multi source wire deduplication

• US 8,495,304 B1
• Filed: 12/23/2010
• Issued: 07/23/2013
• Est. Priority Date: 12/23/2010
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Natanzon, Assaf, Andreyev, Ivan
• Primary Examiner(s): NGUYEN, HIEP T
• Application Number:
  US12/977,651
• Time in Patent Office: 943 Days
• Field of Search: None
• US Class Current: 711/133
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/2074 Asynchronous techniques
• CPC Class Codes: G06F 2201/855 Details of asynchronous mirroring using a journal to transfer not-yet-mirrored changes

Abstract

A method, system, and program product for data deduplication in a replication environment, the replication environment having a production site, a splitter, and a replication site, wherein the replication site has a journal, comprising determining a digest for each chunk of data of a set of data chunks, determining for each chunk whether the digest is in an index on the production site, determining for which offsets are to be evicted from the cache on the replication site, replacing the chunks in set of chunks that are in the index with an offset, and transmitting the set of chunks, offsets and an eviction list to the replication site.

Claims

1. A method for data de-duplication in a replication environment, the replication environment having a production site, a splitter, and a replication site, wherein the replication site has a journal, the method comprising:

determining a digest for each chunk of data of a set of data chunks;

determining for each chunk of the set of data chunks whether the digest is in an index on the production site;

determining which offsets are to be evicted from a cache on the replication site;

replacing data chunks in the set of data chunks that are in the index with an offset; and

transmitting the set of data chunks, the offsets and an eviction list to the replication site.

2. The method of claim 1 further comprising:

determining if data belong to a consistency group (CG) being de-duplicated;

receiving a vector data at the replication site; and

freeing the offsets in the eviction list from the cache.

3. The method of claim 2 further comprising:

determining whether the replication site is full;

based on a positive determination, sending a de-acknowledgement to the production site.

4. The method of claim 3 further comprising:

based on a negative determination, adding the chunks to the cache on the replication site and returning a positive acknowledgement to the production site, including the offsets in which the chunks where added.

5. The method of claim 4 further comprising:

receiving a response at the production site from the replication site;

based on a response of an acknowledgement, adding a waiting list associated with the response to the index; and

releasing the data in the eviction list.

6. The method of claim 3 further comprising:

decrementing a reference counter; and

calculating an eviction list.

7. The method of claim 6 further comprising:

based on a response of a de-acknowledgement, removing all data in a waiting list;

adding old eviction data to a next list.

8. The method of claim 2, where determining whether CG is de-duplicated is determined by a user.

9. The method of claim 2, where determining whether CG is de-duplicated is determined by the replication environment.

10. The method of claim 9, where system determines whether to de-duplicate a CG based on the de-duplication ratio of the CG.

11. The method of claim 9, where system determines whether to de-duplicate a CG based on an application type of the CG.

12. A system for data deduplication in a replication environment, the system comprising:

a production site having an index,

a replication site having a data cache; and

computer-executable logic operating in memory, wherein the computer-executable program logic is configured for execution of the following;

determining a digest for each chunk of data of a set of data chunks;

determining for each chunk of the set of data chunks whether the digest is in an index on the production site;

determining for which offsets are to be evicted from the cache on the replication site;

replacing data chunks in the set of data chunks that are in the index with an offset; and

transmitting the set of data chunks, the offsets and an eviction list to the replication site.

13. The system of claim 12 where in the program logic is further configured for:

determining if data belong to a consistency group (CG) being de-duplicated receiving vector data at the replication site; and

freeing the offsets in the eviction list from the cache.

14. The system of claim 13 where in the program logic is further configured for:

determining whether the replication site is full;

based on a positive determination, sending a de-acknowledgement to the production site.

15. The system of claim 14 where in the program logic is further configured for:

based on a negative determination, adding the chunks to the cache on the replication site and returning a positive acknowledgement to the production site, including the offsets in which the chunks where added.

16. A computer program product comprising:

a non-transitory computer readable medium encoded with computer executable program code for data de-duplication in a replication environment, the replication environment having a production site, a splitter, and a replication site, wherein the replication site has a journal, the code enabling;

determining a digest for each chunk of data of a set of data chunks;

determining for each chunk of the set of data chunks whether the digest is in an index on the production site;

determining which offsets are to be evicted from a cache on the replication site;

replacing data chunks in the set of data chunks that are in the index with an offset; and

transmitting the set of data chunks, the offsets and an eviction list to the replication site.

17. The computer program product of claim 16 where in the code further enables:

determining if data belong to a consistency group (CG) being de-duplicated

receiving vector data at the replication site; and

freeing the offsets in the eviction list from the cache.

18. The computer program product of claim 17 where in the code further enables:

determining whether the replication site is full;

based on a positive determination, sending a de-acknowledgement to the production site.

19. The computer program product of claim 17 wherein determining whether CG is replicated is determined by a user.

20. The computer program product of claim 17 wherein determining whether CG is replicated is determined by the replication environment.