APPLICATION-AWARE AND REMOTE SINGLE INSTANCE DATA MANAGEMENT

• US 20090319585A1
• Filed: 06/24/2008
• Published: 12/24/2009
• Est. Priority Date: 06/24/2008
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Gokhale, Parag
• Granted Patent: US 8,219,524 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/9574 of access to content, e.g. by caching

Abstract

A method and system for reducing storage requirements and speeding up storage operations by reducing the storage of redundant data includes receiving a request that identifies one or more files or data objects to which to apply a storage operation. For each file or data object, the storage system determines if the file or data object contains data that matches another file or data object to which the storage operation was previously applied, based on awareness of the application that created the data object. If the data objects do not match, then the storage system performs the storage operation in a usual manner. However, if the data objects do match, then the storage system may avoid performing the storage operation with respect to the particular file or data object.

Claims

1. A method of storing application-specific data objects included within a file in a data storage system, the method comprising:

receiving a request to store data contained in a file generated by an application, wherein the data includes multiple discrete application-specific data objects having differing sizes;

determining the application that generated the file that includes the multiple discrete application-specific data objects;

based on the determination of the application, identifying at least some of the multiple discrete application-specific data objects within the data; and

for at least one of the identified multiple discrete application-specific data objects;

generating a substantially unique identifier that represents the discrete application-specific data object;

based on the generated substantially unique identifier, determining whether an instance of the discrete application-specific data object is already stored in a data storage system; and

if an instance of the discrete application-specific data object is not already stored in the data storage system, then storing the discrete application-specific data object in the data storage system.

2. The method of claim 1 wherein the file is a data file created by an electronic mail server application, and the multiple discrete application-specific data objects are electronic mail messages included within the data file.

3. The method of claim 1 wherein generating a substantially unique identifier includes using a cryptographic hash function to generate a hash of the discrete application-specific data object, and wherein determining whether an instance of the discrete application-specific data object is already stored in a data storage system includes comparing the generated hash to another hash stored by the data storage system.

4. The method of claim 1 wherein identifying the multiple discrete application-specific data objects within the file includes:

determining an application data format of the file; and

based on the determined application data format, parsing the file to identify the multiple discrete application-specific data objects within the file.

5. The method of claim 1 wherein identifying the multiple discrete application-specific data objects within the file includes:

determining an application that created the file; and

invoking the application to identify the multiple discrete application-specific data objects within the file.

6. The method of claim 1 wherein identifying the multiple discrete application-specific data objects within the file includes:

determining an application that created the file; and

invoking a module that is distinct from the application to identify the multiple discrete application-specific data objects within the file.

7. The method of claim 1 wherein generating a substantially unique identifier includes applying a cryptographic hash algorithm to only the portion of the file corresponding to the discrete application-specific data object to generate a hash value of the discrete application-specific data object.

8. The method of claim 1, further comprising:

if an instance of the discrete application-specific data object is already stored in the data storage system, incrementing a reference count corresponding to the discrete application-specific data object.

9. The method of claim 1, further comprising:

if an instance of the discrete application-specific data object is already stored in the data storage system;

identifying metadata associated with the discrete application-specific data object; and

storing the identified metadata.

10. The method of claim 1, further comprising:

for at least one of the identified multiple discrete application-specific data objects;

determining a first timestamp of the discrete application-specific data object;

determining a second timestamp of an instance of the discrete application-specific data object already stored in the data storage system;

comparing the first and second timestamps; and

if the first timestamp exceeds the second timestamp by a threshold amount, then storing the discrete application-specific data object in the data storage system.

11. The method of claim 10, further comprising if the first timestamp exceeds the second timestamp by a threshold amount, then removing the instance of the discrete application-specific data object from the data storage system.

12. The method of claim 1, further comprising:

for at least one of the identified multiple discrete application-specific data objects, wherein the discrete application-specific data is encrypted;

determining an encryption scheme of the encrypted discrete application-specific data object;

determining an encryption scheme of an encrypted instance of the discrete application-specific data object already stored in the data storage system;

comparing the two encryption schemes; and

if the two encryption schemes are identical, then storing the encrypted discrete application-specific data object in the data storage system.

13. A system for managing application-generated data objects, the system comprising:

a storage operation manager component configured to receive a request to perform a storage operation on a logical data container, wherein the logical data container includes data objects generated by one or more applications;

a data object identification component configured to identify the application-generated data objects included within the logical data container;

an application data extraction component configured to extract the identified application-generated data objects from the logical data container;

an identifier generation component configured to generate substantially unique identifiers for the extracted application-generated data objects;

an index configured to store substantially unique identifiers;

an identifier comparison component configured to determine whether the generated substantially unique identifiers are already stored in the index; and

a single instance data store configured to communicate with the identifier comparison component and store a subset of the extracted application-generated data objects, the subset including the extracted application-generated data objects whose substantially unique identifiers were not determined to be stored in the index, wherein only a single instance of an extracted application-generated data object is stored in the single instance data store.

14. The system of claim 13:

wherein the storage operation manager component is further configured to receive a request to perform a storage operation on a first logical data container and a second, different, logical data container, wherein the first and second logical data containers each include an instance of an identical data object generated by an application, and wherein the first and second logical data containers are files or databases;

wherein the identifier generation component is further configured to generate an identical substantially unique identifier for the instances of the identical data object; and

wherein the single instance data store is further configured to store only a single instance of the identical data object.

15. The system of claim 13 wherein the storage operation manager is further configured to receive requests to continuously replicate data from one or more client computer systems to a data storage system.

16. The system of claim 13 wherein:

the data object identification component is further configured to determine the application that created the logical data container; and

the application data extraction component is further configured to apply a data format specific to the application to parse the contents of the container.

17. The system of claim 13 wherein either the data object identification component or the application data extraction component is further configured to invoke the application or a module associated with the application that enables either the identification of application-generated data objects included within the logical data container or the extraction of application-generated data objects from the logical data container.

18. The system of claim 13 wherein:

the application data extraction component is further configured to extract metadata associated with the application-generated data objects; and

the single instance data store is further configured to store the extracted metadata.

19. A computer-readable storage medium whose contents cause a computer system to perform a method of storing application-specific data objects, the method comprising:

receiving a first file, the first file containing multiple data objects;

receiving a second file, the second file containing multiple data objects;

identifying the data objects within the first and second files;

generating substantially unique identifiers for the data objects within the first and second files;

determining whether the data objects in the first and second files are already stored in a single instance data store;

extracting the data objects in the first and second files that are not already stored in the single instance data store from the first and second files; and

storing the extracted data objects in the single instance data store.

20. The computer-readable storage medium of claim 19 wherein:

the first and second files each include an instance of an identical data object;

generating substantially unique identifiers for the instances of the identical data object includes generating an identical substantially unique identifier for the instances of the identical data object; and

storing the extracted data objects in the single instance data store includes storing a single instance of the identical data object in the single instance data store.

21. The computer-readable storage medium of claim 19 wherein the first and second files are first and second database files, and wherein:

identifying the data objects includes identifying entries within tables within the first and second database files;

generating substantially unique identifiers includes generating substantially unique identifiers for the data within the entries;

determining whether the data objects in the first and second files are already stored includes determining whether the data within the entries is already stored in the single instance data store;

extracting the data objects includes extracting the data within the entries from the first and second database files that is not already stored in the single instance data store; and

storing the extracted data objects includes storing the extracted data in the single instance data store.

22. The computer-readable storage medium of claim 19 wherein determining whether the data objects in the first and second files are already stored includes generating a digest of each data object and comparing the generated digest with one or more previously stored digests.

23. The computer-readable storage medium of claim 19 wherein the method further comprises:

extracting metadata from the extracted data objects; and

storing the extracted metadata in the single instance data store

24. The computer-readable storage medium of claim 19 wherein the method further comprises:

receiving an indication of a size threshold, wherein data objects whose size does not exceed the size threshold are to be stored in the single instance data store;

determining the sizes of the data objects within the first and second files; and

comparing the sizes of the data objects with the size threshold,

wherein determining whether the data objects in the first and second files are already stored in a single instance data store includes determining that the data objects within the first and second files whose size does not exceed the size threshold are not already stored in a single instance data store, thereby causing such data objects to be stored in the single instance data store.

25. The computer-readable storage medium of claim 19 wherein the method further comprises:

receiving an indication of a category or type, wherein data objects of the indicated category or type are not to be stored in the single instance data store;

determining the categories or types of the data objects within the first and second files; and

comparing the categories or types of the data objects with the indicated category or type,

wherein determining whether the data objects in the first and second files are already stored in a single instance data store includes determining that the data objects within the first and second files of the indicated category or type are already stored in a single instance data store.