Application-aware and remote single instance data management

• US 9,971,784 B2
• Filed: 08/03/2015
• Issued: 05/15/2018
• Est. Priority Date: 06/24/2008
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Gokhale, Parag
• Primary Examiner(s): Uddin, Mohammed R
• Application Number:
  US14/816,687
• Publication Number:
  US 20150339315A1
• Time in Patent Office: 1,016 Days
• Field of Search: 707610, 707655, 707692, 707635, 707999204
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/162 Delete operations erasing in storage systems G06F3/0652
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/178 Techniques for file synchronisation in file systems

Abstract

A method and system for reducing storage requirements and speeding up storage operations by reducing the storage of redundant data includes receiving a request that identifies one or more files or data objects to which to apply a storage operation. For each file or data object, the storage system determines if the file or data object contains data that matches another file or data object to which the storage operation was previously applied, based on awareness of the application that created the data object. If the data objects do not match, then the storage system performs the storage operation in a usual manner. However, if the data objects do match, then the storage system may avoid performing the storage operation with respect to the particular file or data object.

Claims

1. A system for copying files or data objects from a computer system at a first location to a second location, including single-instancing the files or data objects with a plurality of differing associated metadata, the system comprising:

a processor; and

multiple hardware components, including;

a storage operation manager component coupled to the processor and configured to receive a request to copy a file or data object from a computer system at a first location to a second location, wherein the first location and the second location are geographically remote from each other;

a file cache component at the first location configured to;

receive the file or data object to be copied from the computer system, and

store the file or data object before it is copied to the second location;

a single instance database component at the first location configured to;

extract metadata associated with the file or data object,

query the second location to determine whether the file or data object is already stored at the second location,

wherein the query includes the extracted metadata, and

receive a response from the second location that indicates whether the file or data object is already stored at the second location, wherein the response is based on determining at the second location whether the extracted metadata matches metadata from any files or data objects stored at the second location; and

wherein the single instance database component at the first location is further configured to;

when the file or data object is not already stored at the second location, copy the file or data object from the file cache component at the first location to the second location, and

when the file or data object is already stored at the second location and the extracted metadata does not match metadata stored at the second location,

(a) single-instance the file or data object at the second location by declining to copy the file or data object thereto from the file cache component at the first location, and

(b) copy the extracted metadata to the second location and associate the extracted metadata with the already-stored file or data object at the second location, thereby storing for a single stored instance of the file or data object at the second location at least a first metadata version and a second metadata version that is different from the first metadata version.

2. The system of claim 1 wherein the storage operation manager component is further configured to receive a second request to copy a second file or data object from a second computer system at a third location to the second location, wherein the second location and the third location are geographically remote, and further comprising:

a second file cache component at the third location configured to;

receive the second file or data object to be copied from the second computer system, and

store the second file or data object before it is copied to the second location; and

a second single instance database component at the third location configured to;

query the second location to determine whether the second file or data object is already stored at the second location,

receive a response from the second location that indicates whether the second file or data object is already stored at the second location, and

when the second file or data object is not already stored at the second location, copy the second file or data object from the second file cache component to the second location.

3. The system of claim 1 wherein the single instance database component is further configured to:

ascertain incremental changes in files or data objects stored at the file cache component; and

copy any incremental changes in files or data objects from the file cache component to the second location.

4. The system of claim 1 wherein the first location is associated with a satellite office and the second location is associated with a main office, wherein the file cache component is further configured to only store one copy of each file or data object, and wherein the single instance database component is further configured to copy the files or data objects that are unique in the first location to the second location.

5. The system of claim 1 wherein the single instance database component is further configured to:

create a substantially unique identifier that represents the file or data object; and

send the substantially unique identifier to the second location as part of the query to determine whether the file or data object is already stored at the second location.

6. The system of claim 1, wherein the extracted metadata is associated with at least one of:

permissions for the file or data object, a property of the file or data object, an access control list for the file or data object, an identifier for the file or data object, a size of the file or data object, a creation date of the file or data object, a creation date for the file or data object, and an access date for the file or data object.

7. A non-transitory computer-readable storage medium encoded with instructions for controlling a computer system to transfer files from a computer system at a source location to a target location, by a method comprising:

receiving a request to transfer a file from a computer system at a source location to a target location, wherein the target location includes a single instance database, and wherein the source location and the target location are geographically remote from each other;

sending a request to the single instance database to determine whether the file matches any file already stored by the single instance database and wherein metadata extracted from the file to be transferred matches any metadata associated with any file already stored by the single instance database;

receiving a determination from the single instance database as to whether the file matches any file already stored by the single instance database;

when the file does not match any file already stored at the target location, storing the file from the computer system at the source location to the single instance database at the target location;

receiving a determination from the single instance database as to whether the extracted metadata matches any metadata associated with any file stored by the single instance database; and

when the file is already stored by the single instance database and the extracted metadata does not match metadata associated with the already-stored file;

(a) declining to store the file from the computer system to the single instance database at the target location, and (b) storing the extracted metadata from the computer system to the single instance database at the target location and associating the extracted metadata with the already-stored file, thereby storing for a single stored instance of the file at the single instance database at least a first metadata version and a second metadata version that is different from the first metadata version.

8. The computer-readable storage medium of claim 7 wherein the single instance database stores files received from computer systems at locations other than the source location and tracks such locations, and wherein the single instance database contains a reference count for each file indicating how many different tracked locations refer to the file.

9. The computer-readable storage medium of claim 7 wherein the request to store the file is received as part of a continuous data replication operation.

10. The computer-readable storage medium of claim 7 wherein sending a request to the single instance database includes:

generating a substantially unique identifier that represents the file; and

sending the substantially unique identifier to the single instance database.

11. The computer-readable storage medium of claim 7 wherein sending a request to the single instance database includes:

identifying data objects within the file;

generating substantially unique identifiers that represent each identified data object; and

sending the substantially unique identifiers to the single instance database.

12. The computer-readable storage medium of claim 7, wherein the method further comprises:

when the file matches a file already stored at the target location;

incrementing a reference count in an index that refers to the file; and

indicating to the requestor that the request to copy the file to the target location succeeded, without actually copying the file to the target location.

13. The computer-readable storage medium of claim 7 wherein sending a request to a single instance database at the target location includes:

determining the size of the file; and

sending the file size to the single instance database.

14. A method for transferring files from a computer system at a source location to a target location, the method comprising:

receiving a request to transfer a file from a computer system at a source location to a target location, wherein the target location includes a single instance database, and wherein the source location and the target location are geographically remote from each other;

sending a request to the single instance database to determine whether the file matches any file already stored by the single instance database and wherein metadata extracted from the file to be transferred matches any metadata associated with any file already stored by the single instance database;

receiving a determination from the single instance database as to whether the file matches any file already stored by the single instance database;

when the file does not match any file already stored at the target location, storing the file from the computer system at the source destination to the single instance database at the target location;

receiving a determination from the single instance database as to whether the extracted metadata matches any metadata associated with any file stored by the single instance database; and

when the file is already stored by the single instance database and the extracted metadata does not match metadata associated with the already-stored file;

(a) declining to store the file from the computer system at the source location to the single instance database at the target location, and (b) storing the extracted metadata from the computer system to the target location and associating the extracted metadata with the already-stored file, thereby storing for a single stored instance of the file at the single instance database at least a first metadata version and a second metadata version that is different from the first metadata version.

15. The method of claim 14, wherein the single instance database stores files received from computer systems at locations other than the source location and tracks such locations, and wherein the single instance database contains a reference count for each file indicating how many different tracked locations refer to the file.

16. The method of claim 14, wherein the request to store the file is received as part of a continuous data replication operation.

17. The method of claim 14, wherein sending a request to the single instance database includes:

generating a substantially unique identifier that represents the file; and

sending the substantially unique identifier to the single instance database.

18. The method of claim 14, wherein sending a request to the single instance database includes:

identifying data objects within the file;

generating substantially unique identifiers that represent each identified data object; and

sending the substantially unique identifiers to the single instance database.

19. The method of claim 14, wherein the method further comprises:

when the file matches a file already stored at the target location;

incrementing a reference count in an index that refers to the file; and

indicating to the requestor that the request to copy the file to the target location succeeded, without actually copying the file to the target location.