Object deduplication and application aware snapshots

• US 9,613,043 B2
• Filed: 01/15/2016
• Issued: 04/04/2017
• Est. Priority Date: 10/07/2008
• Status: Active Grant

• Current Assignee: Quest Software, Inc.
• Original Assignee: Quest Software, Inc.
• Inventors: Rao, Goutham, Brueggemann, Eric, George, Carter
• Primary Examiner(s): Wong, Leslie
• Application Number:
  US14/996,608
• Publication Number:
  US 20160132524A1
• Time in Patent Office: 445 Days
• CPC Class Codes: G06F 16/128 Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion use of snapshots for error detection or correction G06F11/14, G06F11/16
• CPC Class Codes: G06F 16/1744 using compression, e.g. sparse files
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/183 Provision of network file services by network file servers, e.g. by using NFS, CIFS network file access protocols H04L67/1097
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 40/205 Parsing

Abstract

Embodiments deploy delayering techniques, and the relationships between successive versions of a rich-media file become apparent. With this, modified rich-media files suddenly present far smaller storage overhead as compared to traditional application-unaware snapshot and versioning implementations. Optimized file data is stored in suitcases. As a file is versioned, each new version of the file is placed in the same suitcase as the previous version, allowing embodiments to employ correlation techniques to enhance optimization savings.

Claims

1. A method comprising:

parsing a file to identify boundaries for a plurality of first level objects, including a first compound object, in order to determine whether each of the plurality of first level objects is compound or primitive, wherein a primitive object is the most basic representation of a discrete data structure in the file;

recursively parsing the first compound object until a plurality of lowest level primitive objects is identified;

correlating primitive objects within the file and across a plurality of files;

storing the identified boundaries for use in deduplication;

setting deduplication boundaries at boundaries of the plurality of lowest level primitive objects such that deduplication uses variable sized blocks instead of fixed sized blocks; and

optimizing the plurality of lowest level primitive objects with a plurality of object specific optimization algorithms.

2. The method of claim 1, wherein object based deduplication is applied to the plurality of lowest level primitive objects.

3. The method of claim 1, wherein compound objects include zip archives and PDF files.

4. The method of claim 1, wherein primitive object include JPEG images, TIFF images, and CAD drawings.

5. The method of claim 1, wherein deduplication is applied to the plurality of lowest level primitive objects.

6. The method of claim 1, wherein the first compound object comprises a plurality of second level objects.

7. The method of claim 1, wherein metadata for the plurality of lowest level primitive objects is stored in a separate file.

8. A system comprising:

a processor; and

memory comprising one or more programs, the one or more programs containing instructions for;

parsing a file to identify boundaries for a plurality of first level objects, including a first compound object, in order to determine whether each of the plurality of first level objects is compound or primitive, wherein a primitive object is the most basic representation of a discrete data structure in the file;

recursively parsing the first compound object until a plurality of lowest level primitive objects is identified;

correlating primitive objects within the file and across a plurality of files;

storing the identified boundaries for use in deduplication;

setting deduplication boundaries at boundaries of the plurality of lowest level primitive objects such that deduplication uses variable sized blocks instead of fixed sized blocks; and

optimizing the plurality of lowest level primitive objects with a plurality of object specific optimization algorithms.

9. The system of claim 8, wherein object based deduplication is applied to the plurality of lowest level primitive objects.

10. The system of claim 8, wherein compound objects include zip archives and PDF files.

11. The system of claim 8, wherein primitive object include JPEG images, TIFF images, and CAD drawings.

12. The system of claim 8, wherein deduplication is applied to the plurality of lowest level primitive objects.

13. The system of claim 8, wherein the first compound object comprises a plurality of second level objects.

14. The system of claim 8, wherein metadata for the plurality of lowest level primitive objects is stored in a separate file.

15. A non-transitory computer readable medium comprising one or more programs, the one or more programs containing instructions for:

parsing a file to identify boundaries for a plurality of first level objects, including a first compound object, in order to determine whether each of the plurality of first level objects is compound or primitive, wherein a primitive object is the most basic representation of a discrete data structure in the file;

recursively parsing the first compound object until a plurality of lowest level primitive objects is identified;

correlating primitive objects within the file and across a plurality of files;

storing the identified boundaries for use in deduplication;

setting deduplication boundaries at boundaries of the plurality of lowest level primitive objects such that deduplication uses variable sized blocks instead of fixed sized blocks; and

optimizing the plurality of lowest level primitive objects with a plurality of object specific optimization algorithms.

16. The non-transitory computer readable medium of claim 15, wherein object based deduplication is applied to the plurality of lowest level primitive objects.

17. The non-transitory computer readable medium of claim 15, wherein compound objects include zip archives and PDF files.

18. The non-transitory computer readable medium of claim 15, wherein primitive object include JPEG images, TIFF images, and CAD drawings.

19. The non-transitory computer readable medium of claim 15, wherein deduplication is applied to the plurality of lowest level primitive objects.

20. The non-transitory computer readable medium of claim 15, wherein the first compound object comprises a plurality of second level objects.