Techniques for global data deduplication

• US 8,577,850 B1
• Filed: 11/15/2010
• Issued: 11/05/2013
• Est. Priority Date: 11/15/2010
• Status: Active Grant

• Current Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Original Assignee: Symantec Corporation (NortonLifeLock Inc.)
• Inventors: Genda, Jon, Bromley, Graham, Angerer, Walter
• Primary Examiner(s): NGUYEN, CAM LINH T
• Application Number:
  US12/946,500
• Time in Patent Office: 1,086 Days
• Field of Search: 707/634, 707/661, 707/692
• US Class Current: 707/692
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

Techniques for data deduplication are disclosed. In one particular exemplary embodiment, the techniques may be realized as a method for data deduplication comprising receiving a write request for a unit of data, generating a fingerprint for the unit of data, transmitting at least a portion of the fingerprint to a plurality of deduplication monitors, performing the write request using an indicator indicating a location of a previously stored unit of data having a fingerprint matching the transmitted at least a portion of the fingerprint if a response indicating the previously stored unit of data having the fingerprint matching the transmitted at least a portion of the fingerprint is received from at least one of the plurality of deduplication monitors, and performing the write request using the unit of data if no response is received.

Claims

1. A method for data deduplication comprising:

receiving a write request for a block of data;

generating a fingerprint for the block of data;

transmitting, using at least one computer processor, a portion of the fingerprint to a plurality of deduplication monitors;

performing the write request using an indicator indicating a location of a previously stored block of data having a fingerprint matching the transmitted portion of the fingerprint if a response indicating the previously stored block of data having the fingerprint matching the transmitted portion of the fingerprint is received from at least one of the plurality of deduplication monitors; and

performing the write request using the block of data if no response is received.

2. The method of claim 1, wherein transmitting at least a portion of the fingerprint to a plurality of deduplication monitors comprises broadcasting at least a portion of the fingerprint to a plurality of deduplication monitors.

3. The method of claim 1, wherein transmitting at least a portion of the fingerprint to a plurality of deduplication monitors comprises sending at least a portion of the fingerprint to a control monitor and transmitting at least a portion of the fingerprint from a control monitor to a plurality of deduplication monitors.

4. The method of claim 1, wherein a plurality of responses are received from at least two of the plurality of deduplication monitors.

5. The method of claim 4, further comprising requesting a full fingerprint to verify the block of data is stored from the at least two of the plurality of deduplication monitors.

6. The method of claim 1, wherein the fingerprint comprises a hash key.

7. The method of claim 1, wherein each of the plurality of deduplication monitors comprises a monitor for a data domain and data deduplication is provided across a plurality of domains.

8. The method of claim 1, wherein the fingerprint is generated and transmitted from a client plug-in.

9. The method of claim 8, wherein the client comprises a media server.

10. The method of claim 1, wherein the indicator indicating a location of a previously stored block of data allows data written with the indicator to be migrated or restored without use of deduplication metadata.

11. The method of claim 1, wherein the indicator indicating a location of a previously stored block of data allows data written with the indicator to be migrated or restored without use of one or more of the plurality of deduplication monitors.

12. The method of claim 1, wherein implementation of deduplication is capable of being performed on storage to provide deduplication of data written to the storage after implementation without affecting existing data on the storage.

13. The method of claim 1, wherein the number of the plurality of deduplication monitors is configurable.

14. The method of claim 1, wherein loss of one or more deduplication components does not affect data protection.

15. The method of claim 1, wherein each of the plurality of deduplication monitors maintains a table of one or more fingerprints.

16. The method of claim 1, the write request comprises a portion of a stream of backup data.

17. The method of claim 1, wherein a counter is maintained for a unit of storage referenced by other storage and the unit of storage is deallocated only in the event the counter indicates no other storage is referencing the unit of storage.

18. At least one non-transitory processor readable storage medium for storing a computer program of instructions configured to be readable by at least one processor for instructing the at least one processor to execute a computer process for performing the method as recited in claim 1.

19. An article of manufacture for data deduplication, the article of manufacture comprising:

at least one non-transitory processor readable medium; and

instructions stored on the at least one medium;

wherein the instructions are configured to be readable from the at least one medium by at least one processor and thereby cause the at least one processor to operate so as to;

receive a write request for a block of data;

generate a fingerprint for the block of data;

transmit a portion of the fingerprint to a plurality of deduplication monitors;

perform the write request using an indicator indicating a location of a previously stored block of data having a fingerprint matching the transmitted portion of the fingerprint if a response indicating the previously stored block of data having the fingerprint matching the transmitted portion of the fingerprint is received from at least one of the plurality of deduplication monitors; and

perform the write request using the block of data if no response is received.

20. A system for data deduplication comprising:

one or more processors communicatively coupled to a network;

wherein the one or more processors are configured to;

receive a write request for a block of data;

generate a fingerprint for the block of data;

transmit a portion of the fingerprint to a plurality of deduplication monitors;

perform the write request using an indicator indicating a location of a previously stored block of data having a fingerprint matching the transmitted portion of the fingerprint if a response indicating the previously stored block of data having the fingerprint matching the transmitted portion of the fingerprint is received from at least one of the plurality of deduplication monitors; and

perform the write request using the block of data if no response is received.