FILE AWARE BLOCK LEVEL DEDUPLICATION

• US 20110071989A1
• Filed: 08/17/2010
• Published: 03/24/2011
• Est. Priority Date: 09/21/2009
• Status: Active Grant

• Current Assignee: Quest Software, Inc.
• Original Assignee: Ocarina Networks, Inc. (Dell Technologies Inc.)
• Inventors: Wilson, Mike, Rao, Goutham, George, Carter, Munusamy, Parthiban, Bashyam, Murali, Jayaraman, Vinod
• Granted Patent: US 8,510,275 B2
• Time in Patent Office: Days
• US Class Current: 707/692
• CPC Class Codes: G06F 11/1446 Point-in-time backing up or restoration of persistent data
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Abstract

A system provides file aware block level deduplication in a system having multiple clients connected to a storage subsystem over a network such as an Internet Protocol (IP) network. The system includes client components and storage subsystem components. Client components include a walker that traverses the namespace looking for files that meet the criteria for optimization, a file system daemon that rehydrates the files, and a filter driver that watches all operations going to the file system. Storage subsystem components include an optimizer resident on the nodes of the storage subsystem. The optimizer can use idle processor cycles to perform optimization. Sub-file compression can be performed at the storage subsystem.

Claims

1. A method, comprising:

determining candidate files for optimization at a client device;

identifying blocks associated with the candidate files;

sending a scatter gather block list from the client device over a network to a storage sub-system for optimization, wherein an optimizer running on the storage sub-system deduplicates and compresses the blocks from the scatter gather block list and stores optimized data in a suitcase volume.

2. The method of claim 1, wherein a notification from the optimizer is received indicating that the blocks from the block list have been processed.

3. The method of claim 1, wherein a stub file a stub file associated with the blocks from the block list is created.

4. The method of claim 1, wherein a message is sent to the storage sub-system identifying file blocks that can be returned to the free space pool.

5. The method of claim 1, wherein a plurality of portions in a first candidate for are analyzed for activity levels.

6. The method of claim 5, wherein the plurality of portions in the first candidate file as analyzed to determine hot and cold portions of the first candidate file.

7. The method of claim 6, wherein the plurality of portions in the first candidate file are analyzed using an activity threshold.

8. The method of claim 1, wherein the suitcase volume is maintained at a block storage sub-system.

9. The method of claim 1, wherein a walker running on the client is used to determine candidate files for optimization.

10. A method, comprising:

receiving a plurality of blocks associated candidate files for optimization, the plurality of blocks received from a client device;

deduplicating and compressing the plurality of blocks from the client device, wherein the plurality of blocks are determined to be cold portions of candidate files;

storing optimized data in a suitcase volume.

11. The method of claim 10, wherein a notification is sent to the client device indicating that the blocks have been processed.

12. The method of claim 10, wherein a stub file a stub file associated with the blocks from the block list is created.

13. The method of claim 10, wherein a message is received at the storage sub-system identifying file blocks that can be returned to the free space pool.

14. The method of claim 10, wherein a plurality of blocks analyzed for activity levels.

15. The method of claim 14, wherein the plurality of blocks are analyzed to determine hot and cold portions of candidate files.

16. The method of claim 15, wherein the plurality of blocks are analyzed using an activity threshold.

17. The method of claim 10, wherein the suitcase volume is maintained at a block storage sub-system.

18. The method of claim 10, wherein a walker running on the client is used to determine candidate files for optimization.

19. A system, comprising:

an interface configured to receive a plurality of blocks associated candidate files for optimization, the plurality of blocks received from a client device;

a processor configured to deduplicate and compress the plurality of blocks from the client device, wherein the plurality of blocks are determined to be cold portions of candidate files;

persistent storage configured to maintained optimized data in a suitcase volume.

20. The system of claim 19, wherein a notification from is sent indicating that the blocks from the block list have been processed.