Technique selection in a deduplication aware client environment

• US 9,984,093 B2
• Filed: 08/06/2014
• Issued: 05/29/2018
• Est. Priority Date: 08/06/2014
• Status: Active Grant

• Current Assignee: Quest Software, Inc.
• Original Assignee: Quest Software, Inc.
• Inventors: Tripathy, Tarun K., Smith, Brian R., Dinkar, Abhijit S.
• Primary Examiner(s): HO, BINH VAN
• Application Number:
  US14/453,173
• Publication Number:
  US 20160042008A1
• Time in Patent Office: 1,392 Days
• Field of Search: 707692
• CPC Class Codes: G06F 11/00 Error detection; Error correction; Monitoring error detection, correction or monitoring in information storage based on relative movement between record carrier and transducer G11B20/18; monitoring, i.e. supervising the progress of recording or reproducing G11B27/36; in static stores G11C29/00
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 11/3017 where the computing system is implementing multitasking multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 16/1744 using compression, e.g. sparse files
• CPC Class Codes: G06F 16/1752 based on file chunks
• CPC Class Codes: G06F 16/183 Provision of network file services by network file servers, e.g. by using NFS, CIFS network file access protocols H04L67/1097
• CPC Class Codes: G06F 2201/81 Threshold
• CPC Class Codes: G06F 3/06 Digital input from, or digital output to, record carriers , e.g. RAID, emulated record carriers or networked record carriers
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 9/00 Arrangements for program control, e.g. control units program control for peripheral devices G06F13/10
• CPC Class Codes: H04L 65/1059 End-user terminal functionalities specially adapted for real-time communication
• CPC Class Codes: H04L 65/764 at the destination reformatting of additional data in video clients H04N21/4355
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Techniques and mechanisms described herein facilitate the transmission of a data stream to a networked storage system. According to various embodiments, a determination may be made as to whether an amount of available computing resources at a client device meets or exceeds a computing resource availability threshold at the client device. A processing operation on a data stream may be performed at the client device to produce a pre-processed data stream when the amount of available computing resources meets or exceeds the computing resource availability threshold. The pre-processed data stream may be transmitted to a networked storage system for storage via a network. The networked storage system may be operable to store deduplicated data for retrieval via the network.

Claims

1. A method comprising:

determining whether an amount of available computing resources at a client device comprising a processor and memory meets or exceeds a computing resource availability threshold at the client device;

performing a processing operation on a data stream at the client device to produce a pre-processed data stream when it is determined that the amount of available computing resources meets or exceeds the computing resource availability threshold, wherein the processing operating comprises parsing the data stream to identify one or more data chunks, and wherein the one or more data chunks are identified via a designated rolling hash parsing technique operable to identify at least some identical chunks when parsing different but overlapping data streams, wherein the one or more data chunks are identified by;

determining, using the rolling hash parsing technique, a hash value for a first chunk of the at least some identical chunks;

determining that the hash value for the first chunk qualifies as a chunk boundary; and

in response to determining that the hash value for the first chunk qualifies as the chunk boundary, determining that the data stream has reached a chunk boundary; and

transmitting the pre-processed data stream for storage to a networked storage system via a network, the networked storage system operable to store deduplicated data for retrieval via the network, the networked storage system operable to parse data streams via the designated rolling hash parsing technique.

2. The method recited in claim 1, wherein the processing operation comprises:

deduplicating the data stream.

3. The method recited in claim 1, wherein the processing operation comprises compressing the data stream.

4. The method recited in claim 1, wherein the processing operation comprises encrypting the data stream.

5. The method recited in claim 1, wherein the processing operation comprises:

for one or more of the identified data chunks, performing an operation selected from the group consisting of;

deduplicating the identified data chunk, compressing the identified data chunk, and encrypting the identified data chunk.

6. The method recited in claim 5, wherein the processing operation further comprises:

determining a respective fingerprint for each identified data chunk by applying a hash function to the identified data chunk.

7. The method recited in claim 6, wherein the processing operation further comprises:

determining whether each identified data chunk is stored at a networked storage system by transmitting each fingerprint to the networked storage system via the network.

8. The method recited in claim 7, wherein determining whether the identified chunk is stored at the networked storage system further comprises:

receiving a fingerprint status message from the networked storage system, the fingerprint status message indicating whether the chunk associated with the fingerprint is stored at the networked storage system.

9. The method recited in claim 5, wherein the processing operation further comprises:

transmitting a block map update request message to the networked storage system, the block map update request message including information for updating a block map at the networked storage system, the block map identifying a designated memory location at which each data encrypted chunk is stored at the networked storage system.

10. The method recited in claim 1, the method further comprising:

identifying an amount of available computing resources at the networked storage system.

11. The method recited in claim 1, wherein the data stream is generated at the client device via a network storage protocol selected from the group consisting of:

the Network File System (NFS) protocol, the Common Internet File System (CIFS) protocol, and the Open Storage (OST) protocol.

12. A device comprising:

memory operable to store a data stream;

a processor operable to determine whether an amount of available computing resources at a client device meets or exceeds a computing resource availability threshold at the client device and perform a processing operation on the data stream to produce a pre-processed data stream when it is determined that the amount of available computing resources meets or exceeds the computing resource availability threshold, wherein the processing operating comprises parsing the data stream to identify one or more data chunks, and wherein the one or more data chunks are identified via a designated rolling hash parsing technique operable to identify at least some identical chunks when parsing different but overlapping data streams, wherein the one or more data chunks are identified by;

determining, using the rolling hash parsing technique, a hash value for a first chunk of the at least some identical chunks;

determining that the hash value for the first chunk qualifies as a chunk boundary; and

in response to determining that the hash value for the first chunk qualifies as the chunk boundary, determining that the data stream has reached a chunk boundary; and

a communications interface operable to transmit the pre-processed data stream for storage to a networked storage system via a network, the networked storage system operable to store deduplicated data for retrieval via the network, the networked storage system operable to parse data streams via the designated rolling hash parsing technique.

13. The device recited in claim 12, wherein the processing operation is an operation selected from the group consisting of:

deduplicating the data stream, encrypting the data stream, and compressing the data stream.

14. The device recited in claim 12, wherein the processing operation further comprises:

determining a respective fingerprint for each identified data chunk by applying a hash function to the identified data chunk.

15. The device recited in claim 14, wherein the processing operation further comprises:

determining whether each identified data chunk is stored at a networked storage system by transmitting each fingerprint to the networked storage system via the network.

16. The device recited in claim 15, wherein determining whether the identified chunk is stored at the networked storage system further comprises:

receiving a fingerprint status message from the networked storage system, the fingerprint status message indicating whether the chunk associated with the fingerprint is stored at the networked storage system.

17. One or more non-transitory computer readable media having instructions stored thereon to perform operations comprising:

determining whether an amount of available computing resources at a client device comprising a processor and memory meets or exceeds a computing resource availability threshold at the client device;

performing a processing operation on a data stream at the client device to produce a pre-processed data stream when it is determined that the amount of available computing resources meets or exceeds the computing resource availability threshold, wherein the processing operating comprises parsing the data stream to identify one or more data chunks, and wherein the one or more data chunks are identified via a designated rolling hash parsing technique operable to identify at least some identical chunks when parsing different but overlapping data streams, wherein the one or more data chunks are identified by;

determining, using the rolling hash parsing technique, a hash value for a first chunk of the at least some identical chunks;

determining that the hash value for the first chunk qualifies as a chunk boundary; and

in response to determining that the hash value for the first chunk qualifies as the chunk boundary, determining that the data stream has reached a chunk boundary; and

transmitting the pre-processed data stream for storage to a networked storage system via a network, the networked storage system operable to store deduplicated data for retrieval via the network, the networked storage system operable to parse data streams via the designated rolling hash parsing technique.