DE-DUPLICATION SYSTEMS AND METHODS FOR APPLICATION-SPECIFIC DATA

• US 20110016091A1
• Filed: 03/15/2010
• Published: 01/20/2011
• Est. Priority Date: 06/24/2008
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Vijayan, Manoj Kumar, Gokhale, Parag, Prahlad, Anand, Kottomtharayil, Rajiv
• Granted Patent: US 8,484,162 B2
• Time in Patent Office: Days
• US Class Current: 707/654
• CPC Class Codes: G06F 11/1448 Management of the data involved in backup or backup restore
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 16/162 Delete operations erasing in storage systems G06F3/0652
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/9574 of access to content, e.g. by caching
• CPC Class Codes: G06F 21/604 Tools and structures for managing or administering access control systems

Abstract

Content-aware systems and methods for improving de-duplication, or single instancing, in storage operations. In certain examples, backup agents on client devices parse application-specific data to identify data objects that are candidates for de-duplication. The backup agents can then insert markers or other indictors in the data that identify the location(s) of the particular data objects. Such markers can, in turn, assist a de-duplication manager to perform object-based de-duplication and increase the likelihood that like blocks within the data are identified and single instanced. In other examples, the agents can further determine if a data object of one file type can or should be single-instanced with a data object of a different file type. Such processing of data on the client side can provide for more efficient storage and back-end processing.

Claims

1. A system for managing application-generated data objects, the system comprising:

a first backup agent executing on a first client device, the first backup agent being configured to, in response to a storage operation request,

parse first application-specific data of the first client device that is the subject of the storage operation request, the first application-specific data comprising a plurality of first data objects having different sizes,

identify, based at least on an application that generated the first application-specific data, one or more of the plurality of first data objects of the first application-specific data to be considered for de-duplication, wherein the one of more first data objects comprises less than the entire first application-specific data, and

insert at least one indicator in the first application-specific data that identifies at least one location of the one or more first data objects within the first application-specific data; and

a de-duplication module executing on a computing device, the de-duplication module being in communication with the first backup agent to receive the first application-specific data and to,

process each of the one or more data objects, and

based on said processing, determine if a duplicate copy of any of the one or more first data objects exists in a storage device.

2. The system of claim 1, wherein the first backup agent is configured to insert the at least one indicator by setting or clearing a bit in at least one header of the first application-specific data.

3. The system of claim 1, wherein the at least one indicator comprises an offset value identifying a beginning of the one or more first data objects within the first application-specific data.

4. The system of claim 1, further comprising a second backup agent executing on a second client device, the second backup agent being configured to, in response to a second storage operation request:

parse second application-specific data of the second client device that is the subject of the second storage operation request, the second application-specific data comprising a plurality of second data objects;

identify one or more of the plurality of second data objects to be considered for de-duplication; and

insert at least one second indicator in the second application-specific data that identifies at least one location of the one or more second data objects to be considered for de-duplication.

5. The system of claim 4, wherein the de-duplication module is configured to receive the second application-specific data from the second backup agent and to determine if a duplicate copy of the one or more second data objects exists in the storage device.

6. The system of claim 4, wherein the first application-specific data is received from a first operating system and the second application-specific data is received from a second operating system different than the first operating system.

7. The system of claim 4, further comprising:

a first de-duplication database configured to store unique ones of the one or more first data objects; and

a second de-duplication database configured to store unique ones of the one or more second data objects, wherein the first de-duplication database is separate and different from the second de-duplication database.

8. The system of claim 1, wherein the application comprises an electronic mail server application.

9. A method for managing application-generated data objects, the method comprising:

receiving a first storage operation request for first data generated by a first application executing on a first client device, the first data comprising a plurality of first data objects having non-uniform sizes;

parsing the first data to identify one or more of the plurality of first data objects to be considered for de-duplication, wherein the one of more first data objects comprises less than the entire first data;

inserting at least one indicator in the first data that identifies at least one location of the one or more first data objects within the first data;

processing each of the one or more first data objects to determine if a duplicate copy of the one or more first data objects exists in at least one storage device; and

for each of the one or more first data objects,

if a duplicate copy does not exist in the at least one storage device, storing the first data object in the at least one storage device,

otherwise, storing at least one of a stub file and a pointer in place of the first data object in the at least one storage device.

10. The method of claim 9, wherein said processing of each of the one or more first data objects further comprises:

generating a substantially unique identifier that represents the first data object; and

accessing a database separate from the at least one storage device that stores a plurality of substantially unique identifiers of other data objects stored in the at least one storage device.

11. The method of claim 10, wherein said generating a substantially unique identifier comprises generating a hash value of the corresponding first data object.

12. The method of claim 9, wherein the plurality of first data objects comprises a body of an electronic mail message and an attachment.

13. The method of claim 9, wherein each of the one or more first data objects comprises a file.

14. The method of claim 9, wherein the first data comprises a chunk file.

15. The method of claim 9, wherein said identifying is based at least in part on one or more file extensions associated with the one or more first data objects.

16. The method of claim 9, wherein said inserting the at least one indicator comprises inserting the at least one indicator in at least one header of the first data.

17. A method for managing application-generated data objects, the method comprising:

receiving a first storage operation request for first data generated by a first application executing on a first client device, the first data comprising a plurality of first data objects;

parsing the first data to identify one or more of the plurality of first data objects to be considered for de-duplication;

receiving a second storage operation request for second data generated by a second application, the second data comprising a plurality of second data objects, and the second data having a different file format than the first data;

parsing the second data to identify one or more of the plurality of second data objects to be considered for de-duplication; and

inserting in at least one of a copy of the first data and a copy of the second data an indicator that denotes, based on the first and second file formats, that the one or more first data objects and the one or more second data objects should not be considered together for de-duplication.

18. The method of claim 17, additionally comprising:

storing de-duplication data of the one or more first data objects in a first database; and

storing de-duplication data of the one or more second data objects in a second database.

19. A system for managing application-generated data objects, the system comprising:

means for receiving a first storage operation request for first data generated by a first application executing on a first client device, the first data comprising a plurality of first data objects having differing sizes;

means for parsing the first data to identify one or more of the plurality of first data objects to be considered for de-duplication, wherein the one of more first data objects comprises less than the entire first data;

means for inserting at least one indicator in the first data that identifies at least one location of the one or more first data objects within the first data;

means for processing each of the one or more first data objects to determine if a duplicate copy of the first data object exists in at least one storage device; and

means for (i) storing each of the one or more first data objects in the at least one storage device that does not have a duplicate copy and (ii) storing at least one of a stub and a pointer in place of each of the one or more first data objects that does have a duplicate copy in the at least one storage device.

20. The system of claim 19, wherein said processing means further comprises:

means for generating a substantially unique identifier that represents each first data object; and

means for accessing a database separate from the at least one storage device that stores a plurality of substantially unique identifiers of other data objects stored in the at least one storage device.