File aware block level deduplication

• US 8,510,275 B2
• Filed: 08/17/2010
• Issued: 08/13/2013
• Est. Priority Date: 09/21/2009
• Status: Active Grant

• Current Assignee: Quest Software, Inc.
• Original Assignee: Dell Products LP (Dell Technologies Inc.)
• Inventors: Wilson, Mike, Munusamy, Parthiban, George, Carter, Jayaraman, Vinod, Rao, Goutham, Bashyam, Murli
• Primary Examiner(s): LE, HUNG D
• Application Number:
  US12/858,258
• Publication Number:
  US 20110071989A1
• Time in Patent Office: 1,092 Days
• Field of Search: None
• US Class Current: 707/692
• CPC Class Codes: G06F 11/1446 Point-in-time backing up or restoration of persistent data
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Abstract

A system provides file aware block level deduplication in a system having multiple clients connected to a storage subsystem over a network such as an Internet Protocol (IP) network. The system includes client components and storage subsystem components. Client components include a walker that traverses the namespace looking for files that meet the criteria for optimization, a file system daemon that rehydrates the files, and a filter driver that watches all operations going to the file system. Storage subsystem components include an optimizer resident on the nodes of the storage subsystem. The optimizer can use idle processor cycles to perform optimization. Sub-file compression can be performed at the storage subsystem.

Claims

1. A method comprising:

selecting candidate files for optimization by traversing a file system;

identifying a plurality of blocks in the plurality of candidate files;

sending a block list and a plurality of tokens to a storage sub-system for optimization, the blocks list corresponding to the plurality of blocks in the plurality of candidate files and the plurality of tokens corresponding to the plurality of candidate files, wherein optimization includes deduplication and compression;

receiving a notification that a file in the plurality of candidate files has been optimized, wherein the notification includes information on optimized data location, wherein upon receipt of the notification, a token in the plurality of tokens is used to determine if the file in tile plurality of candidate files has changed during optimization;

wherein if the file has not changed, a stub file including a filemap needed to find the optimized data is generated.

2. The method of claim 1, wherein the file stub comprises original metadata including file attributes so file attribute requests can be serviced directly from the stub file without accessing optimized data.

3. The method of claim 1, wherein if the file has changed, a stub file is noted generated.

4. The method of claim 1, wherein the block list, a file list, and volumes are sent to the storage sub-system for optimization.

5. The method of claim 1, wherein candidate files are selected using a plurality of criteria.

6. The method of claim 5, wherein the plurality of criteria for optimization includes file type and file size.

7. The method of claim 5, wherein the plurality of criteria for optimization includes file activity frequency.

8. A system comprising:

a processor operable to select candidate files for optimization by traversing a file system, wherein a plurality of blocks in the plurality of candidate files are identified;

an interface operable to send a block list and a plurality'"'"' of tokens to a storage sub-system for optimization, the blocks list corresponding to the plurality of blocks in the plurality of candidate files and the plurality of tokens corresponding to the plurality of candidate files, wherein optimization includes deduplication and compression, the interface further operable to receive a notification that a file in the plurality of candidate files has been optimized;

wherein the notification includes information on optimized data location, wherein upon receipt of the notification, a token in the plurality of tokens is used to determine if the file in the plurality of candidate files has changed during optimization;

wherein if the file has changed, a stub file is noted generated.

9. The system of claim 8, wherein if the file has not changed, a stub file including a filemap needed to find the optimized data is generated.

10. The system of claim 9, wherein the file stub comprises original metadata including file attributes so file attribute requests can be serviced directly from the stub file without accessing optimized data.

11. The system of claim 8, wherein the block list, a file list, and volumes are sent to the storage sub-system for optimization.

12. The system of claim 8, wherein candidate files are selected using a plurality of criteria.

13. The system of claim 12, wherein the plural of criteria for optimization includes file type and file size.

14. The system of claim 12, wherein the plurality of criteria for optimization includes file activity frequency.

15. A non-transitory computer readable medium comprising:

computer code for selecting candidate files for optimization by traversing a file system;

computer code for identifying a plurality of blocks in the plurality of candidate files;

computer code for sending a block list and a plurality of tokens to a storage sub-system for optimization, the blocks list corresponding to the plurality of blocks in the plurality of candidate files and the plurality of tokens corresponding to the plurality of candidate files, wherein optimization includes deduplication and compression;

computer code for receiving a notification that a file in the plurality of candidate files has been optimized, wherein the notification includes information on optimized data location, wherein upon receipt of the notification, a token in the plurality of tokens is used to determine if the file in the plurality of candidate files has changed during optimization;

wherein if the file has changed, a stub file is noted generated.

16. The non-transitory computer readable medium of claim 15, wherein if the file has not changed, a stub file including a filemap needed to find the optimized. data is generated.

17. The non-transitory computer readable medium of claim 16, wherein the file stub comprises original metadata including file attributes so file attribute requests can be serviced directly from the stub file without accessing optimized data.