Application-aware and remote single instance data management

• US 9,098,495 B2
• Filed: 06/24/2008
• Issued: 08/04/2015
• Est. Priority Date: 06/24/2008
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Gokhale, Parag
• Primary Examiner(s): Uddin, Mohammed R
• Application Number:
  US12/145,347
• Publication Number:
  US 20090319534A1
• Time in Patent Office: 2,597 Days
• Field of Search: 707/610, 707/655, 707/692, 707/699, 707/635, 707/999.003
• US Class Current: 1/1
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/162 Delete operations erasing in storage systems G06F3/0652
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/178 Techniques for file synchronisation in file systems

Abstract

A method and system for reducing storage requirements and speeding up storage operations by reducing the storage of redundant data includes receiving a request that identifies one or more files or data objects to which to apply a storage operation. For each file or data object, the storage system determines if the file or data object contains data that matches another file or data object to which the storage operation was previously applied, based on awareness of the application that created the data object. If the data objects do not match, then the storage system performs the storage operation in a usual manner. However, if the data objects do match, then the storage system may avoid performing the storage operation with respect to the particular file or data object.

Claims

1. A system for copying files from a computer system at a first location to a second location, the system comprising:

a processor;

a storage operation manager component, coupled to the processor, that is configured to receive a request to copy a file or data object from a computer system at a first location to a second location, wherein the first location and the second location are geographically remote from each other;

a file cache component at the first location configured to;

receive the file or data object to be copied from the computer system; and

store the file or data object before it is copied to the second location; and

a single instance database component at the first location configured to;

generate a substantially unique identifier for the file or data object;

extract metadata associated with the file or data object,

wherein the extracted metadata describes at least three of the following;

permissions for the file or data object,

a property of the file or data object,

an access control list for the file or data object,

an identifier for the file or data object,

a size of the file or data object,

a creation date for the file or data object, and

an access date for the file or data object;

query the second location to determine whether the file or data object is already stored at the second location, including sending the generated substantially unique identifier and extracted metadata to the second location;

in response to the query, receive a single response from the second location that indicates whether the file or data object is already stored at the second location and that indicates whether the generated substantially unique identifier and extracted metadata match a substantially unique identifier and extracted metadata from any files or data objects stored at the second location;

when the file or data object is not already stored at the second location, copy the file or data object from the file cache component to the second location; and

when the file or data object is already stored at the second location and the extracted metadata does not match extracted metadata from the stored file or data object, copy the extracted metadata to the second location, thereby resulting in storing two instances of metadata at the second location for a single stored instance for that file or data object;

wherein the single instance database component copies a file or data object from the file cache component to the second location as part of a continuous data replication operation that automatically saves copies of all changes made to the file or data object.

2. The system of claim 1 wherein the storage operation manager component is further configured to receive a second request to copy a second file or data object from a second computer system at a third location to the second location, wherein the second location and the third location are geographically remote, and further comprising:

a second file cache component at the third location configured to;

receive the second file or data object to be copied from the second computer system; and

store the second file or data object before it is copied to the second location; and

a second single instance database component at the third location configured to;

query the second location to determine whether the second file or data object is already stored at the second location;

receive a response from the second location that indicates whether the second file or data object is already stored at the second location; and

when the second file or data object is not already stored at the second location, copy the second file or data object from the second file cache component to the second location.

3. The system of claim 1 wherein the single instance database component is further configured to:

ascertain incremental changes in files or data objects stored at the file cache component; and

copy any incremental changes in files or data objects from the file cache component to the second location.

4. The system of claim 1 wherein the first location is associated with a satellite office and the second location is associated with a main office, wherein the file cache component is further configured to only store one copy of each file or data object, and wherein the single instance database component is further configured to copy the files or data objects that are unique to the first location and to the second location.

5. A method for copying files from a computer system at a first location to a second location, the method comprising:

receiving a request to copy a file or data object from a computer system at a first location to a second location, wherein the first location and the second location are geographically remote from each other;

at a file cache component at the first location;

receiving the file or data object to be copied from the computer system;

storing the file or data object before it is copied to the second location;

at a single instance database component at the first location;

generating a substantially unique identifier for the file or data object;

extracting metadata associated with the file or data object,

wherein the extracted metadata describes at least three of the following;

permissions for the file or data object, a property of the file or data object, an access control list for the file or data object, an identifier for the file or data object, a size of the file or data object, a creation date for the file or data object, and an access date for the file or data object;

querying the second location to determine whether the file or data object is already stored at the second location, including sending the generated substantially unique identifier and extracted metadata to the second location;

in response to the query, receiving a single response from the second location that indicates whether the file or data object is already stored at the second location and that indicates whether the generated substantially unique identifier and extracted metadata match a substantially unique identifier and extracted metadata from any files or data objects stored at the second location;

when the file or data object is not already stored at the second location, copying the file or data object from the file cache component to the second location as part of a continuous data replication operation that automatically saves copies of all changes made to the file or data object; and

when the file or data object is already stored at the second location and the extracted metadata does not match extracted metadata from the file or data object already stored at the second location, copying the extracted metadata to the second location, thereby resulting in storing two instances of metadata for a single stored instance for that file or data object.

6. The method of claim 5, further comprising:

receiving a second request to copy a second file or data object from a second computer system at a third location to the second location, wherein the second location and the third location are geographically remote;

at a second file cache component at the third location;

receiving the second file or data object to be copied from the second computer system; and

storing the second file or data object before it is copied to the second location; and

at a second single instance database component at the third location;

querying the second location to determine whether the second file or data object is already stored at the second location;

receiving a response from the second location that indicates whether the second file or data object is already stored at the second location; and

when the second file or data object is not already stored at the second location, copying the second file or data object from the second file cache component to the second location.

7. The method of claim 5 further comprising, at the single instance database component at the first location:

ascertaining incremental changes in files or data objects stored at the file cache component; and

copying any incremental changes in files or data objects from the file cache component to the second location.

8. The method of claim 5 wherein the first location is associated with a satellite office and the second location is associated with a main office, the method further comprising storing only one copy of each file or data object, and copying only the files or data objects that are unique to the first location and to the second location.

9. A non-transitory computer-readable storage medium containing instructions for controlling a computer system to copy files from a computer system at a first location to a second location, by a method comprising:

receiving a request to copy a file or data object from a computer system at a first location to a second location, wherein the first location and the second location are geographically remote from each other;

at a file cache component at the first location;

receiving the file or data object to be copied from the computer system;

storing the file or data object before it is copied to the second location;

at a single instance database component at the first location;

generating a substantially unique identifier for the file or data object;

extracting metadata associated with the file or data object,

wherein the extracted metadata describes at least three of;

permissions for the file or data object, a property of the file or data object, an access control list for the file or data object, an identifier for the file or data object, a size of the file or data object, a creation date for the file or data object, and an access date for the file or data object;

querying the second location to determine whether the file or data object is already stored at the second location, including sending the generated substantially unique identifier and extracted metadata to the second location;

in response to the query, receiving a single response from the second location that indicates whether the file or data object is already stored at the second location and that indicates whether the generated substantially unique identifier and extracted metadata match a substantially unique identifier and extracted metadata from any files or data objects stored at the second location;

when the file or data object is not already stored at the second location, copying the file or data object from the file cache component to the second location as part of a continuous data replication operation that automatically saves copies of all changes made to the file or data object; and

when the file or data object is already stored at the second location and the extracted metadata does not match extracted metadata from the file or data object already stored at the second location, copying the extracted metadata to the second location, thereby resulting in storing two instances of metadata for a single stored instance for that file or data object.

10. The computer-readable storage medium of claim 9, wherein the method further comprises:

receiving a second request to copy a second file or data object from a second computer system at a third location to the second location, wherein the second location and the third location are geographically remote;

at a second file cache component at the third location;

receiving the second file or data object to be copied from the second computer system; and

storing the second file or data object before it is copied to the second location; and

at a second single instance database component at the third location;

querying the second location to determine whether the second file or data object is already stored at the second location;

receiving a response from the second location that indicates whether the second file or data object is already stored at the second location; and

when the second file or data object is not already stored at the second location, copying the second file or data object from the second file cache component to the second location.

11. The computer-readable storage medium of claim 9, wherein the method further comprises, at the single instance database component at the first location:

ascertaining incremental changes in files or data objects stored at the file cache component; and

copying any incremental changes in files or data objects from the file cache component to the second location.

12. The computer-readable storage medium of claim 9, wherein the first location is associated with a satellite office and the second location is associated with a main office, and wherein the method further comprises storing only one copy of each file or data object, and copying only the files or data objects that are unique to the first location and to the second location.

13. A system for copying data objects from a computer system at a first location to a second location, the system comprising:

a processor;

a storage operation manager component, coupled to the processor, that is configured to receive a request to copy a data object from a computer system at a first location to a second location;

a file cache component at the first location configured to;

receive the data object to be copied from the computer system; and

store the data object before it is copied to the second location; and

a single instance database component at the first location configured to;

generate a hash for the data object;

extract metadata associated with the data object;

query the second location to determine whether the data object is already stored at the second location, including sending the hash and extracted metadata to the second location;

receive, in response to the query, a response from the second location that indicates whether the data object is already stored at the second location and that indicates whether the generated hash and extracted metadata match hashes and extracted metadata from any data objects stored at the second location;

copy, when the data object is not already stored at the second location, the data object from the file cache component to the second location; and

copy, when the data object is already stored at the second location and the extracted metadata does not match extracted metadata from the stored file or data object, the extracted metadata to the second location, thereby resulting in storing two instances of metadata at the second location for a single stored instance for that data object.

14. The system of claim 13 wherein the storage operation manager component is further configured to receive a second request to copy a second data object from a second computer system at a third location to the second location, and further comprising:

a second file cache component at the third location configured to;

receive the second data object to be copied from the second computer system; and

store the second data object before it is copied to the second location; and

a second single instance database component at the third location configured to;

query the second location to determine whether the second data object is already stored at the second location;

receive a response from the second location that indicates whether the second data object is already stored at the second location; and

when the second data object is not already stored at the second location, copy the second data object from the second file cache component to the second location.

15. The system of claim 13 wherein the single instance database component is further configured to:

ascertain incremental changes in data objects stored at the file cache component; and

copy any incremental changes in data objects from the file cache component to the second location.

16. The system of claim 13 wherein the first location is associated with a satellite office and the second location is associated with a main office, wherein the file cache component is further configured to only store one copy of each data object, and wherein the single instance database component is further configured to copy the data objects that are unique to the first location to the second location.

17. The system of claim 13, wherein the single instance database component at the first location configured to:

determine whether the data object is encrypted; and

decide whether or not to de-duplicate the data object based on a type of encryption used for encrypting the data object.