APPLICATION-AWARE AND REMOTE SINGLE INSTANCE DATA MANAGEMENT

• US 20120271793A1
• Filed: 06/26/2012
• Published: 10/25/2012
• Est. Priority Date: 06/24/2008
• Status: Abandoned Application

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Gokhale, Parag
• Application Number:
  US13/532,877
• Publication Number:
  US 20120271793A1
• Time in Patent Office: Days
• US Class Current: 707/609
• CPC Class Codes: G06F 16/9574 of access to content, e.g. by caching

Abstract

A method and system for reducing storage requirements and speeding up storage operations by reducing the storage of redundant data includes receiving a request that identifies one or more files or data objects to which to apply a storage operation. For each file or data object, the storage system determines if the file or data object contains data that matches another file or data object to which the storage operation was previously applied, based on awareness of the application that created the data object. If the data objects do not match, then the storage system performs the storage operation in a usual manner. However, if the data objects do match, then the storage system may avoid performing the storage operation with respect to the particular file or data object.

Claims

1. A method for use with a data storage system, the method comprising:

determining an application that generated a file,

wherein the file includes multiple discrete application-specific data objects;

based on the determination of the application, identifying at least some of the multiple discrete application-specific data objects within the data,

wherein the identifying includes parsing the file using an already existing module particular to the application that generated the file, and

wherein the module does not expose the format of the file; and

for at least a first one of the identified multiple discrete application-specific data objects;

generating a substantially unique identifier that represents the first discrete application-specific data object;

based on the generated substantially unique identifier, determining whether an instance of the first discrete application-specific data object is already stored in a data storage system; and

if an instance of the first discrete application-specific data object is not already stored in the data storage system, then storing the first discrete application-specific data object in the data storage system.

2. The method of claim 1 wherein the file is a data file created by an electronic mail server application, and the multiple discrete application-specific data objects are electronic mail messages included within the data file.

3. The method of claim 1 wherein generating a substantially unique identifier includes using a cryptographic hash function to generate a hash of the first discrete application-specific data object, and wherein determining whether an instance of the first discrete application-specific data object is already stored in a data storage system includes comparing the generated hash to another hash stored by the data storage system.

4. The method of claim 1 wherein generating a substantially unique identifier includes applying a cryptographic hash algorithm to only the portion of the file corresponding to the first discrete application-specific data object to generate a hash value of the first discrete application-specific data object.

5. The method of claim 1, further comprising:

if an instance of the first discrete application-specific data object is already stored in the data storage system, incrementing a reference count corresponding to the first discrete application-specific data object.

6. The method of claim 1, further comprising:

if an instance of the first discrete application-specific data object is already stored in the data storage system;

identifying metadata associated with the first discrete application-specific data object; and

storing the identified metadata.

7. The method of claim 1, further comprising:

for the first discrete application-specific data object;

determining a first timestamp of the first discrete application-specific data object;

determining a second timestamp of an instance of the first discrete application-specific data object already stored in the data storage system;

comparing the first and second timestamps; and

if the first timestamp exceeds the second timestamp by a threshold amount, then storing the first discrete application-specific data object in the data storage system.

8. The method of claim 7, further comprising if the first timestamp exceeds the second timestamp by a threshold amount, then removing the instance of the first discrete application-specific data object from the data storage system.

9. The method of claim 1, wherein the first discrete application-specific data object is encrypted, and wherein the method further comprises:

determining an encryption scheme of the encrypted first discrete application-specific data object;

determining an encryption scheme of an encrypted instance of the first discrete application-specific data object already stored in the data storage system;

comparing the two encryption schemes; and

if the two encryption schemes are identical, then storing the encrypted first discrete application-specific data object in the data storage system.

10. A system for managing application-generated data objects, the system comprising:

a processor;

a data object identification component configured to identify application-generated data objects included within a logical data container;

an application data extraction component configured to extract the identified application-generated data objects from the logical data container;

an index configured to store substantially unique identifiers; and

,

a data store configured to store a subset of the application-generated data objects,

wherein the subset includes extracted application-generated data objects whose substantially unique identifiers were not determined to be stored in the index,

wherein only a single instance of an extracted application-generated data object is stored in the data store; and

wherein the data object identification component is further configured to determine the application that created the logical data container; and

wherein the data object identification component is further configured to utilize results of the determination made by the data object identification component to invoke an already existing module to parse the logical data container in order to identify the application-generated data objects included within the logical data container,

wherein the module is particular to the application that created the logical data container, and

wherein the module is configured to avoid exposing the format of the logical data container.

11. The system of claim 10, further comprising:

a storage operation manager component configured to receive a request to perform a storage operation on a first logical data container and a second, different, logical data container, wherein the first and second logical data containers each include an instance of an identical data object generated by an application, and wherein the first and second logical data containers are files or databases;

wherein the data store is further configured to store only a single instance of the identical data object.

12. The system of claim 11 wherein the storage operation manager is further configured to receive requests to continuously replicate data from one or more client computer systems to a data storage system.

13. The system of claim 11 wherein the application data extraction component is further configured to invoke the already-existing module particular to the application to parse the logical data container in order to extract the application-generated data objects from the logical data container.

14. The system of claim 11 wherein:

the application data extraction component is further configured to extract metadata associated with the application-generated data objects; and

the data store is further configured to store the extracted metadata.

15. A non-transitory computer-readable storage medium whose contents cause a computer system to perform an operation of storing application-specific data objects, the operation comprising:

determining a first application that created a first file;

based on the determination for the first file, selecting an object model that is particular to the first application;

determining a second application that created a second file;

based on the determination for the second file, selecting an already created module that is particular to the second application;

utilizing the object model that is particular to the first application to identify data objects within the first file;

utilizing the already created module that is particular to the second application to parse the second file in order to identify data objects within the second file, wherein the already created module differs from the object model;

determining whether the identified data objects in the first and second files are already stored in a data store;

utilizing the object model that is particular to the first application to extract the identified data objects in the first file that are not already stored in the data store;

utilizing the already created object model that is particular to the second application module to extract the identified data objects in the second file that are not already stored in the data store; and

causing the extracted data objects to be stored in the data store.

16. The non-transitory computer-readable storage medium of claim 15, further comprising:

generating an identical substantially unique identifier for instances of an identical data object

wherein the first and second files each include an instance of the identical data object; and

storing a single instance of the identical data object in the data store.

17. The non-transitory computer-readable storage medium of claim 15 wherein the first and second files are first and second database files, and wherein:

identifying the data objects includes identifying entries within tables within the first and second database files;

generating substantially unique identifiers for data within the identified entries;

determining whether the data within the identified entries is already stored in the data store;

extracting the data based on the identified entries from the first and second database files that is not already stored in the data store; and

storing the extracted data in the data store.

18. The non-transitory computer-readable storage medium of claim 15 wherein determining whether the data objects in the first and second files are already stored includes generating a digest of each data object and comparing the generated digest with one or more previously stored digests.

19. The non-transitory computer-readable storage medium of claim 15 wherein the operation further comprises:

extracting metadata from the extracted data objects; and

storing the extracted metadata in the data store

20. The non-transitory computer-readable storage medium of claim 15 wherein the operation further comprises:

receiving an indication of a size threshold, wherein data objects whose size does not exceed the size threshold are to be stored in the data store;

determining the sizes of the data objects within the first and second files; and

comparing the sizes of the data objects with the size threshold,

wherein determining whether the data objects in the first and second files are already stored in the data store includes determining that the data objects within the first and second files whose size does not exceed the size threshold are not already stored in a data store, thereby causing such data objects to be stored in the data store.

21. The non-transitory computer-readable storage medium of claim 15 wherein the operation further comprises:

receiving an indication of a category or type, wherein data objects of the indicated category or type are not to be stored in the data store;

determining the categories or types of the data objects within the first and second files; and

comparing the categories or types of the data objects with the indicated category or type,

wherein determining whether the data objects in the first and second files are already stored in a data store includes determining that the data objects within the first and second files of the indicated category or type are already stored in the data store.

22. The non-transitory computer-readable storage medium of claim 15 wherein the object model enumerates data objects within the first file.

23. The non-transitory computer-readable storage medium of claim 15 wherein the object model or a storage operation manager component understands a format used by the first application for the first file.

24. The non-transitory computer-readable storage medium of claim 15 wherein the operation further comprises:

receiving the first file that was created by the first application in a first format, wherein the first file contains multiple data objects;

receiving the second file that was created by the second, different application in a second format,

wherein the second file contains multiple data objects, and

wherein the second format differs from the first format;