Host side deduplication

• US 8,620,886 B1
• Filed: 09/20/2011
• Issued: 12/31/2013
• Est. Priority Date: 09/20/2011
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Lentini, James F., Condict, Michael N., Byan, Stephen M., Madan, Anshul
• Primary Examiner(s): CHANNAVAJJALA, SRIRAMA T
• Application Number:
  US13/237,417
• Time in Patent Office: 833 Days
• Field of Search: 711117-118, 711161-166, 709/203, 709/206, 709215-220, 707690-698, 707745-747, 707/822, 707/704, 707/770
• US Class Current: 707/698
• CPC Class Codes: G06F 11/3037 where the computing system component is a memory, e.g. virtual memory, cache accessing, addressing or allocating within memory systems or architectures G06F12/00; checking stores for correct operation G11C29/00
• CPC Class Codes: G06F 12/0891 using clearing, invalidating or resetting means
• CPC Class Codes: G06F 13/00 Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units interface circuits for specific input/output devices G06F3/00 ; multiprogram control therefor G06F9/46; multiprocessor systems G06F15/16
• CPC Class Codes: G06F 15/16 Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs coordinating program control therefor G06F9/52; in regulating and control system G05B
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/137 Hash-based content-based indexing of textual data G06F16/31
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/1756 based on delta files
• CPC Class Codes: G06F 16/181 providing write once read many [WORM] semantics
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 17/00 Digital computing or data processing equipment or methods, specially adapted for specific functions information retrieval, database structures or file system structures therefor G06F16/00
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0611 in relation to response time
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0679 Non-volatile semiconductor memory device, e.g. flash memory, one time programmable memory [OTP]
• CPC Class Codes: H04L 67/5651 Reducing the amount or size of exchanged application data
• CPC Class Codes: H04L 67/5682 Policies or rules for updating, deleting or replacing the stored data

Abstract

One or more techniques and/or systems are provided for performing host side deduplication. Host side deduplication may be performed upon writeable data within a write request received at a host computing device configured to access data stored by a storage server. The host side deduplication may be performed at the host computing device to determine whether the writeable data is already stored by the storage server based upon querying a host side cache comprising data stored by a storage server and/or a data structure comprising unique signatures of data stored by the storage server. If the writeable data is stored by the storage server, then a deduplication notification excluding the writeable data may be sent to the storage server, otherwise a write command comprising the writeable data may be sent. Accordingly, unnecessary network traffic of redundant data already stored by the storage server may be reduced.

Claims

1. A method for performing host side deduplication before accessing a storage server, comprising:

receiving a write request at a host computing device configured to access data stored by a storage server, the write request comprising a request to store writeable data to the storage server;

computing a signature for the writeable data, the signature uniquely identifying the writeable data;

querying a data structure to determine whether the signature is specified within the data structure without accessing the storage server, the data structure comprising one or more signatures uniquely identifying data stored by the storage server, the data structure hosted on the host computing device;

if the signature is specified within the data structure, then sending a deduplication notification to the storage server, the deduplication notification excluding the writeable data; and

if the signature is not specified within the data structure, then sending a write command to the storage server, the write command comprising the writeable data.

2. The method of claim 1, the computing a signature comprising performing a hash function upon the writeable data to compute the signature.

3. The method of claim 1, the deduplication notification comprising an instruction for the storage server to create a reference pointer within a destination location associated with the write request.

4. The method of claim 3, the reference pointer referencing a source location comprising an instance of the writeable data stored by the storage server.

5. The method of claim 1, the deduplication notification comprising an instruction for the storage server to copy the writeable data from a source location comprising an instance of the writeable data stored by the storage server to a destination location associated with the write request.

6. The method of claim 1, the write command comprising an instruction for the storage server to write the writeable data to a destination location associated with the write request.

7. The method of claim 1, comprising:

querying a host side cache to determine whether the writeable data is stored within the host side cache, the host side cache comprising one or more instances of data stored by the storage server.

8. The method of claim 7, comprising:

if the writeable data is stored within the host side cache, then sending the deduplication notification to the storage server; and

if the writeable data is not stored within the host side cache, then sending the write command to the storage server.

9. The method of claim 7, the host side cache comprising a non-volatile cache.

10. The method of claim 7, comprising:

maintaining cache consistency between the host side cache and the storage server.

11. The method of claim 7, the data structure comprising an index derived from the host side cache.

12. The method of claim 1, the data structure excluding at least one signature corresponding to at least one of:

data accessed by the host computing device outside a threshold time frame or data accessed by the host computing device below a threshold frequency.

13. The method of claim 1, comprising at least one of:

updating the data structure based upon sending the write command;

or

updating the data structure based upon sending the deduplication notification.

14. The method of claim 1, the signature comprising a first data block signature associated with a first data block of the writeable data, and the querying comprising:

comparing the first data block signature to one or more data block signatures specified within the data structure; and

if the first data block signature corresponds to a second data block signature specified within the data structure, then performing a byte-by-byte comparison between at least some of the first data block of the writeable data and at least some of an instance of a second data block associated with the second data block signature.

15. A computer readable medium comprising computer executable instructions that when executed by a processor implement a method for performing host side deduplication before accessing a storage server, comprising:

receiving a write request at a host computing device configured to access data stored by a storage server, the write request comprising a request to store writeable data to the storage server;

computing a signature for the writeable data, the signature uniquely identifying the writeable data;

querying a data structure to determine whether the signature is specified within the data structure without accessing the storage server, the data structure comprising one or more signatures uniquely identifying data stored by the storage server, the data structure hosted on the host computing device;

if the signature is specified within the data structure, then sending a deduplication notification to the storage server, the deduplication notification excluding the writeable data; and

if the signature is not specified within the data structure, then storing a write command to the storage server, the write command comprising the writeable data.

16. The computer readable medium of claim 15, the computing a signature comprising performing a hash function upon the writeable data to compute the signature.

17. The computer readable medium of claim 15, the deduplication notification comprising an instruction for the storage server to create a reference pointer within a destination location associated with the write request, the reference pointer referencing a source location comprising an instance of the writeable data stored by the storage server.

18. The computer readable medium of claim 15, the deduplication notification comprising an instruction for the storage server to copy the writeable data from a source location comprising an instance of the writeable date stored by the storage server to a destination location associated with the write request.

19. The computer readable medium of claim 15, the write command comprising an instruction for the storage server to write the writeable data to a destination location associated with the write request.

20. The computer readable medium of claim 15, the method comprising:

querying a host side cache to determine whether the writeable data is stored within the host side cache, the host side cache comprising one or more instances of data stored by the storage server.

21. The computer readable medium of claim 20, the method comprising:

maintaining cache consistency between the host side cache and the storage server, the data structure indexing data stored by the storage server.

22. The computer readable medium of claim 20, the-method comprising:

if the writeable data is stored within the host side cache, then sending the deduplication notification to the storage server; and

if the writeable data is not stored within the host side cache, then sending the write command to the storage server.

23. A system for performing host side deduplication before accessing a storage server, comprising:

a host deduplication component configured to;

receive a write request at a host computing device configured to access data stored by a storage server, the write request comprising a request to store writeable data to the storage server;

compute a signature for the writeable data, the signature uniquely identifying the writeable data;

query a data structure to determine whether the signature is specified within the data structure without accessing the storage server, the data structure comprising one or more signatures uniquely identifying data stored by the storage server, the data structure hosted on the host computing device;

if the signature is specified within the data structure, then send a deduplication notification to the storage server, the deduplication notification excluding the writeable data; and

if the signature is not specified within the data structure, then send a write command to the storage server, the write command comprising the writeable data.

24. The system of claim 23, the computing a signature comprising performing a hash function upon the writeable data to compute the signature.

25. The system of claim 23, the deduplication notification comprising an instruction for the storage server to create a reference pointer within a destination location associated with the write request, the reference pointer referencing a source location comprising an instance of the writeable data stored by the storage server.

26. The system of claim 23, the deduplication notification comprising an instruction for the storage server to copy the writeable data from a source location comprising an instance of the writeable data stored by the storage server to a destination location associated with the write request.

27. The system of claim 23, the write command comprising an instruction for the storage server to write the writeable data to a destination location associated with the write request.

28. The system of claim 23, the host deduplication component configured to:

query a host side cache to determine whether the writeable data is stored within the host side cache, the host side cache comprising one or more instances of data stored by the storage server;

if the writeable data is stored within the host side cache, then send the deduplication notification to the storage server; and

if the writeable data is not stored within the host side cache, then send the write command to the storage server.

29. The system of claim 28, the host deduplication component configured to:

maintain cache consistency between the host side cache and the storage server.

30. The system of claim 23, the host deduplication component configured to:

update the data structure based upon sending the write command.

31. The system of claim 23, the signature comprising a first data block signature associated with a first data block of the writeable data.

32. The system of claim 31, the host deduplication component configured to:

compare the first data block signature to one or more data block signatures specified within the data structure; and

if the first data block signature corresponds to a second data block signature specified within the data structure, then perform a byte-by-byte comparison between at least some of the first data block of the writeable data and at least some of an instance of a second data block associated with the second data block signature.