DE-DUPLICATION SYSTEMS AND METHODS FOR APPLICATION-SPECIFIC DATA

• US 20130290280A1
• Filed: 06/28/2013
• Published: 10/31/2013
• Est. Priority Date: 06/24/2008
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Prahlad, Anand, Vijayan, Manoj Kumar, Kottomtharayil, Rajiv, Gokhale, Parag
• Granted Patent: US 9,405,763 B2
• Time in Patent Office: Days
• US Class Current: 707/692
• CPC Class Codes: G06F 11/1448 Management of the data involved in backup or backup restore
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 16/162 Delete operations erasing in storage systems G06F3/0652
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/9574 of access to content, e.g. by caching
• CPC Class Codes: G06F 21/604 Tools and structures for managing or administering access control systems

Abstract

Content-aware systems and methods for improving de-duplication, or single instancing, in storage operations. In certain examples, backup agents on client devices parse application-specific data to identify data objects that are candidates for de-duplication. The backup agents can then insert markers or other indictors in the data that identify the location(s) of the particular data objects. Such markers can, in turn, assist a de-duplication manager to perform object-based de-duplication and increase the likelihood that like blocks within the data are identified and single instanced. In other examples, the agents can further determine if a data object of one file type can or should be single-instanced with a data object of a different file type. Such processing of data on the client side can provide for more efficient storage and back-end processing.

Claims

1. A system for managing application-generated data objects, the system comprising:

at least a first de-duplication database associated with first application-specific data, the first application-specific data comprising a plurality of data objects;

a first module comprising one or more computer processors configured to insert de-duplication indicators in the first application-specific data,

wherein the inserted de-duplication indicators are inserted in at least one header of the first application-specific data,

wherein one or more of the de-duplication indicators comprises an offset value identifying a beginning of at least one of the plurality of data objects within the first application-specific data, and

wherein the inserted de-duplication indicators further identify that the first de-duplication database is to be used in de-duplicating the first application-specific data;

a second module executing on one or more computer processors configured to perform block-level de-duplication, the second module further configured to;

use the inserted de-duplication indicators in the at least one header to identify the beginning of at least one of the plurality of data objects based on the offset value; and

determine if a duplicate copy of any of the blocks associated with the data objects exist in the first de-duplication database.

2. The system of claim 1, wherein the first module is configured to identify portions within the plurality of data objects to be considered for de-duplication.

3. The system of claim 1, further comprising a third module executing on one or more computer processors configured to:

parse second application-specific data, the second application-specific data comprising a plurality of second data objects;

identify portions within the plurality of second data objects to be considered for de-duplication; and

insert at least one de-duplication indicator in the second application-specific data that identifies at least one location of the identified portions in the second data objects to be considered for de-duplication.

4. The system of claim 3, wherein the second module is further configured to determine whether a duplicate copy of any of the blocks associated with the second data objects exist in a second de-duplication database.

5. The system of claim 3, wherein the inserted de-duplication indicators in the second application-specific data further identify that the second de-duplication database is to be used in de-duplicating the application-specific data.

6. The system of claim 3, wherein:

the first de-duplication database is configured to store unique blocks of the identified portions of the data objects associated with the first application-specific data; and

the second de-duplication database is configured to store unique blocks of the identified portions of the data objects associated with the second application-specific data, wherein the first de-duplication database is separate and different from the second de-duplication database.

7. The system of claim 3, wherein the de-duplication indicators indicate at least one of the first de-duplication database and the second de-duplication database.

8. The system of claim 1, wherein the first application-specific data is associated with an electronic mail server application.

9. A method for managing application-generated data objects, the method comprising:

storing a first de-duplication database associated with first application-specific data, the first application-specific data comprising a plurality of data objects;

inserting de-duplication indicators in at least one header of the first application-specific data, wherein one or more of the de-duplication indicators comprises an offset value identifying a beginning of at least one of the plurality of data objects within the first application-specific data, and wherein the inserted de-duplication indicators further identify that the first de-duplication database is to be used in de-duplicating the first application-specific data;

using the inserted de-duplication indicators in the at least one header to identify the beginning of at least one of the plurality of data objects based on the offset value; and

determining if a duplicate copy of any of the blocks associated with the data objects exist in the first de-duplication database.

10. The method of claim 9, further comprising identifying portions within the plurality of data objects to be considered for de-duplication.

11. The method of claim 9, further comprising:

parsing second application-specific data, the second application-specific data comprising a plurality of second data objects;

identifying portions within the plurality of second data objects to be considered for de-duplication; and

inserting at least one de-duplication indicator in the second application-specific data that identifies at least one location of the identified portions in the second data objects to be considered for de-duplication.

12. The method of claim 11, further comprising determining whether a duplicate copy of any of the blocks associated with the second data objects exist in a second de-duplication database.

13. The method of claim 11, wherein the inserted de-duplication indicators in the second application-specific data further identify that the second de-duplication database is to be used in de-duplicating the application-specific data.

14. The method of claim 11, wherein:

the first de-duplication database is configured to store unique blocks of the identified portions of the data objects associated with the first application-specific data; and

the second de-duplication database is configured to store unique blocks of the identified portions of the data objects associated with the second application-specific data, wherein the first de-duplication database is separate and different from the second de-duplication database.

15. The method of claim 11, wherein the de-duplication indicators indicate at least one of the first de-duplication database and the second de-duplication database.

16. The method of claim 9, wherein the first application-specific data is associated with an electronic mail server application.