File aware block level deduplication

• US 9,753,937 B2
• Filed: 07/08/2013
• Issued: 09/05/2017
• Est. Priority Date: 09/21/2009
• Status: Active Grant

• Current Assignee: Quest Software, Inc.
• Original Assignee: Quest Software, Inc.
• Inventors: Wilson, Micah, Munusamy, Parthiban, George, Carter, Bashyam, Murali, Jayaraman, Vinod, Rao, Goutham
• Primary Examiner(s): LE, HUNG D
• Application Number:
  US13/937,150
• Publication Number:
  US 20130297572A1
• Time in Patent Office: 1,520 Days
• Field of Search: None
• CPC Class Codes: G06F 11/1446 Point-in-time backing up or restoration of persistent data
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Abstract

A system provides file aware block level deduplication in a system having multiple clients connected to a storage subsystem over a network such as an Internet Protocol (IP) network. The system includes client components and storage subsystem components. Client components include a walker that traverses the namespace looking for files that meet the criteria for optimization, a file system daemon that rehydrates the files, and a filter driver that watches all operations going to the file system. Storage subsystem components include an optimizer resident on the nodes of the storage subsystem. The optimizer can use idle processor cycles to perform optimization. Sub-file compression can be performed at the storage subsystem.

Claims

1. A server comprising:

a communications interface operable to transmit to a client device file system traversal information via a network and receive a message from the client device transmitted via the network, the message including a block list identifying a plurality of blocks in a plurality of candidate files, the block list identified based on a traversal of a file system at the server by the client machine;

an optimizer component operable to optimize at least a portion of the plurality of blocks identified in the block list, wherein optimizing includes deduplication and compression operations, and wherein the optimizer produces optimized data; and

a storage component operable to store at least a portion of the optimized data in a suitcase volume, wherein the communications interface is further operable to receive a request for file information corresponding with a file from a file system daemon and transmit optimized data corresponding with the requested file to the file system daemon in response to the request, the optimized data being retrieved from the suitcase volume, the file system daemon being configured to rehydrate the optimized data to create the requested file.

2. The server recited in claim 1, wherein the communications interface is further operable to transmit a notification message to the client device, the notification message indicating that the plurality of blocks have been processed by the optimizer.

3. The server recited in claim 2, wherein the notification message includes optimized data location information.

4. The server recited in claim 2, wherein the message further includes a plurality of tokens corresponding with the plurality of candidate files, and wherein the notification message includes a designated token, the designated token being one of the plurality of tokens.

5. The server recited in claim 4, wherein the designated token is configured to be used to determine whether a designated file in the plurality of candidate files has changed during optimization.

6. The server recited in claim 1, wherein the message identifies a designated plurality of file blocks, the designated plurality of file blocks capable of being returned to a free space pool.

7. The server recited in claim 1, wherein the suitcase volume includes a designated plurality of data blocks, the designated plurality of data blocks being associated with deduplicated files, each of the designated plurality of data blocks being associated with a reference count, the reference count identifying a number of filemap entries referenced by the data block.

8. The server recited in claim 1, wherein the plurality of candidate files are determined by a walker running on the client device, the walker being configured to traverse a namespace to search for one or more files that meet one or more criteria for optimization.

9. A method, comprising:

transmitting to a client device file system traversal information via a network;

receiving at a storage subsystem a message from the client device transmitted via the network, the message including a block list identifying a plurality of blocks in a plurality of candidate files, the block list identified based on a traversal of a file system at the server by the client machine;

optimizing at least a portion of the plurality of blocks identified in the block list, wherein optimizing includes deduplication and compression operations, and wherein the optimizing produces optimized data;

storing at least a portion of the optimized data in a suitcase volume; and

receiving a request for file information corresponding with a file from a file system daemon and transmit optimized data corresponding with the requested file to the file system daemon in response to the request, the optimized data being retrieved from the suitcase volume, the file system daemon being configured to rehydrate the optimized data to create the requested file.

10. The method recited in claim 9, the method further comprising transmitting a notification message to the client device, the notification message indicating that the plurality of blocks have been processed by the optimizer.

11. The method recited in claim 10, wherein the notification message includes optimized data location information.

12. The method recited in claim 10, wherein the message further includes a plurality of tokens corresponding with the plurality of candidate files, and wherein the notification message includes a designated token, the designated token being one of the plurality of tokens.

13. The method recited in claim 12, wherein the designated token is configured to be used to determine whether a designated file in the plurality of candidate files has changed during optimization.

14. The method recited in claim 9, wherein the message identifies a designated plurality of file blocks, the designated plurality of file blocks capable of being returned to a free space pool.

15. The method recited in claim 9, wherein the suitcase volume includes a designated plurality of data blocks, the designated plurality of data blocks being associated with deduplicated files, each of the designated plurality of data blocks being associated with a reference count, the reference count identifying a number of filemap entries referenced by the data block.

16. The method recited in claim 9, wherein the plurality of candidate files are determined by a walker running on the client device, the walker being configured to traverse a namespace to search for one or more files that meet one or more criteria for optimization.

17. One or more non-transitory computer readable media having instructions stored thereon for performing a method, the method comprising:

transmitting to a client device file system traversal information via a network;

receiving at a storage subsystem a message from the client device transmitted via the network, the message including a block list identifying a plurality of blocks in a plurality of candidate files, the block list identified based on a traversal of a file system at the server by the client machine;

optimizing at least a portion of the plurality of blocks identified in the block list, wherein optimizing includes deduplication and compression operations, and wherein the optimizing produces optimized data; and

storing at least a portion of the optimized data in a suitcase volume, wherein the communications interface is further operable to receive a request for file information corresponding with a file from a file system daemon and transmit optimized data corresponding with the requested file to the file system daemon in response to the request, the optimized data being retrieved from the suitcase volume, the file system daemon being configured to rehydrate the optimized data to create the requested file.

18. The one or more computer readable media recited in claim 17, the method further comprising transmitting a notification message to the client device, the notification message indicating that the plurality of blocks have been processed by the optimizer.