FILE AWARE BLOCK LEVEL DEDUPLICATION

• US 20130297572A1
• Filed: 07/08/2013
• Published: 11/07/2013
• Est. Priority Date: 09/21/2009
• Status: Active Grant

• Current Assignee: Quest Software, Inc.
• Original Assignee: Dell Products LP (Dell Technologies Inc.)
• Inventors: Wilson, Mike, Munusamy, Parthiban, Bashyam, Murali, Jayaraman, Vinod, Rao, Goutham, George, Carter
• Granted Patent: US 9,753,937 B2
• Time in Patent Office: Days
• US Class Current: 707/692
• CPC Class Codes: G06F 11/1446 Point-in-time backing up or restoration of persistent data
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Abstract

A system provides file aware block level deduplication in a system having multiple clients connected to a storage subsystem over a network such as an Internet Protocol (IP) network. The system includes client components and storage subsystem components. Client components include a walker that traverses the namespace looking for files that meet the criteria for optimization, a file system daemon that rehydrates the files, and a filter driver that watches all operations going to the file system. Storage subsystem components include an optimizer resident on the nodes of the storage subsystem. The optimizer can use idle processor cycles to perform optimization. Sub-file compression can be performed at the storage subsystem.

Claims

1. A system comprising:

a communications interface operable to receive a message from a client device transmitted via a network, the message including a block list identifying a plurality of blocks in a plurality of candidate files;

an optimizer component operable to optimize at least a portion of the plurality of blocks identified in the block list, wherein optimizing includes deduplication and compression operations, and wherein the optimizer produces optimized data; and

a storage component operable to store at least a portion of the optimized data in a suitcase volume.

2. The system recited in claim 1, wherein the communications interface is further operable to transmit a notification message to the client device, the notification message indicating that the plurality of blocks have been processed by the optimizer

3. The system recited in claim 2, wherein the notification message includes optimized data location information.

4. The system recited in claim 2, wherein the message further includes a plurality of tokens corresponding with the plurality of candidate files, and wherein the notification message includes a designated token, the designated token being one of the plurality of tokens.

5. The system recited in claim 4, wherein the designated token is configured to be used to determine whether a designated file in the plurality of candidate files has changed during optimization.

6. The system recited in claim 1, wherein the message identifies a designated plurality of file blocks, the designated plurality of file blocks capable of being returned to a free space pool.

7. The system recited in claim 1, wherein the suitcase volume includes a designated plurality of data blocks, the designated plurality of data blocks being associated with deduplicated files, each of the designated plurality of data blocks being associated with a reference count, the reference count identifying a number of filemap entries referenced by the data block.

8. The system recited in claim 1, wherein the communications interface is further operable to receive a request for file information corresponding with a file from a file system daemon and transmit optimized data corresponding with the requested file to the file system daemon in response to the request, the optimized data being retrieved from the suitcase volume, the file system daemon being configured to rehydrate the optimized data to create the requested file.

9. The system recited in claim 1, wherein the plurality of candidate files are determined by a walker running on the client device, the walker being configured to traverse a namespace to search for one or more files that meet one or more criteria for optimization.

10. A method, comprising:

receiving at a storage subsystem a message from a client device transmitted via a network, the message including a block list identifying a plurality of blocks in a plurality of candidate files;

optimizing at least a portion of the plurality of blocks identified in the block list, wherein optimizing includes deduplication and compression operations, and wherein the optimizing produces optimized data; and

storing at least a portion of the optimized data in a suitcase volume.

11. The method recited in claim 10, the method further comprising transmitting a notification message to the client device, the notification message indicating that the plurality of blocks have been processed by the optimizer.

12. The method recited in claim 11, wherein the notification message includes optimized data location information.

13. The method recited in claim 11, wherein the message further includes a plurality of tokens corresponding with the plurality of candidate files, and wherein the notification message includes a designated token, the designated token being one of the plurality of tokens.

14. The method recited in claim 13, wherein the designated token is configured to be used to determine whether a designated file in the plurality of candidate files has changed during optimization.

15. The method recited in claim 10, wherein the message identifies a designated plurality of file blocks, the designated plurality of file blocks capable of being returned to a free space pool.

16. The method recited in claim 10, wherein the suitcase volume includes a designated plurality of data blocks, the designated plurality of data blocks being associated with deduplicated files, each of the designated plurality of data blocks being associated with a reference count, the reference count identifying a number of filemap entries referenced by the data block.

17. The method recited in claim 10, the method further comprising:

receiving a request for file information corresponding with a file from a file system daemon and transmit optimized data corresponding with the requested file to the file system daemon in response to the request, the optimized data being retrieved from the suitcase volume, the file system daemon being configured to rehydrate the optimized data to create the requested file.

18. The method recited in claim 10, wherein the plurality of candidate files are determined by a walker running on the client device, the walker being configured to traverse a namespace to search for one or more files that meet one or more criteria for optimization.

19. One or more computer readable media having instructions stored thereon for performing a method, the method comprising:

receiving at a storage subsystem a message from a client device transmitted via a network, the message including a block list identifying a plurality of blocks in a plurality of candidate files;

optimizing at least a portion of the plurality of blocks identified in the block list, wherein optimizing includes deduplication and compression operations, and wherein the optimizing produces optimized data; and

storing at least a portion of the optimized data in a suitcase volume.

20. The one or more computer readable media recited in claim 19, the method further comprising transmitting a notification message to the client device, the notification message indicating that the plurality of blocks have been processed by the optimizer.