APPLICATION-AWARE AND REMOTE SINGLE INSTANCE DATA MANAGEMENT

• US 20150339315A1
• Filed: 08/03/2015
• Published: 11/26/2015
• Est. Priority Date: 06/24/2008
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Gokhale, Parag
• Granted Patent: US 9,971,784 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/162 Delete operations erasing in storage systems G06F3/0652
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/178 Techniques for file synchronisation in file systems

Abstract

A method and system for reducing storage requirements and speeding up storage operations by reducing the storage of redundant data includes receiving a request that identifies one or more files or data objects to which to apply a storage operation. For each file or data object, the storage system determines if the file or data object contains data that matches another file or data object to which the storage operation was previously applied, based on awareness of the application that created the data object. If the data objects do not match, then the storage system performs the storage operation in a usual manner. However, if the data objects do match, then the storage system may avoid performing the storage operation with respect to the particular file or data object.

Claims

1. A method for copying files or data objects from a first data storage system located at a first location to a second data storage system located at a second location that is geographically remote from the first location, the method comprising:

receiving at a first data storage system a file or data object from a first computing system, wherein the first data storage system and the first computing system are located at a first location;

determining whether an instance of the received file or data object is already stored in the first data storage system by performing a match of a substantially unique identifier that represents the received file or data object to substantially unique identifiers that represent files or data objects already stored in the first data system;

storing the received file or data object in the first data storage system when the substantially unique identifier that represents the received file or data object does not match one or more of the substantially unique identifiers that represent the files or data objects already stored in the first data system; and

transmitting the files and data objects stored in the first data storage system to the second data storage system that is located at the second location.

2. The method of claim 1, further comprising:

for at least some of the files and data objects transmitted from the first data storage system and at least some files and data objects transmitted from a third data storage system that is located at a third location;

generating a substantially unique identifier that represents the file or data object;

based on the generated substantially unique identifier, determining whether an instance of the file or data object is already stored in the second data storage system; and

if an instance of the file or data object is not already stored in the second data storage system, then storing the file or data object in the second data storage system.

3. The method of claim 1, further comprising:

determining the size of the received file or data object; and

if the size of the received file or data object does not exceed a threshold size, then storing the received file or data object in the first data storage system.

4. The method of claim 1 wherein receiving at a first data storage system a file or data object from a first computing system includes receiving the file or data object as part of a continuous data replication process between the first computing system and the first data storage system.

5. The method of claim 1 wherein transmitting the files and data objects stored in the first data storage system to a second data storage system includes transmitting the files and data objects as part of a continuous data replication process between the first data storage system and the second data storage system.

6. The method of claim 1, further comprising:

receiving at the first data storage system a second file or data object from a second computing system, wherein the second computing system is located at the first location;

generating a second substantially unique identifier that represents the received second file or data object;

based on the second generated substantially unique identifier, determining whether an instance of the received second file or data object is already stored in the first data storage system; and

if an instance of the received second file or data object is not already stored in the first data storage system, then storing the received second file or data object in the first data storage system.

7. A system for copying files from a computer system at a first location to a second location, the system comprising:

a processor;

a storage operation manager component coupled to the processor and configured to receive a request to copy a file or data object from a computer system at a first location to a second location, wherein the first location and the second location are geographically remote from each other;

a file cache component at the first location configured to;

receive the file or data object to be copied from the computer system; and

store the file or data object before it is copied to the second location; and

a single instance database component at the first location configured to;

extract metadata associated with the file or data object;

query the second location to determine whether the file or data object is already stored at the second location,

wherein the query includes determining whether the extracted metadata matches metadata from any files or data objects stored at the second location;

receive a response from the second location that indicates whether the file or data object is already stored at the second location;

when the file or data object is not already stored at the second location, copy the file or data object from the file cache component to the second location; and

when the extracted metadata does not match metadata from any files or data objects stored at the second location, copy the extracted metadata to the second location.

8. The system of claim 7 wherein the storage operation manager component is further configured to receive a second request to copy a second file or data object from a second computer system at a third location to the second location, wherein the second location and the third location are geographically remote, and further comprising:

a second file cache component at the third location configured to;

receive the second file or data object to be copied from the second computer system; and

store the second file or data object before it is copied to the second location; and

a second single instance database component at the third location configured to;

query the second location to determine whether the second file or data object is already stored at the second location;

receive a response from the second location that indicates whether the second file or data object is already stored at the second location; and

when the second file or data object is not already stored at the second location, copy the second file or data object from the second file cache component to the second location.

9. The system of claim 7 wherein the single instance database component is further configured to:

ascertain incremental changes in files or data objects stored at the file cache component; and

copy any incremental changes in files or data objects from the file cache component to the second location.

10. The system of claim 7 wherein the first location is associated with a satellite office and the second location is associated with a main office, wherein the file cache component is further configured to only store one copy of each file or data object, and wherein the single instance database component is further configured to copy the files or data objects that are unique to the first location to the second location.

11. The system of claim 7 wherein the single instance database component is further configured to:

create a substantially unique identifier that represents the file or data object; and

send the substantially unique identifier to the second location as part of the query to determine whether the file or data object is already stored at the second location.

12. The system of claim 7 wherein the single instance database component is further configured to copy the extracted metadata to the second location in order to store two instances of metadata at the second location for a single stored instance of an associated file or data object.

13. The system of claim 7, wherein the extracted metadata is associated with permissions for the file or data object, a property of the file or data object, an access control list for the file or data object, an identifier for the file or data object, a size of the file or data object, a creation date of the file or data object, a creation date for the file or data object, or an access date for the file or data object.

14. A computer-readable storage medium encoded with instructions for controlling a computer system to transfer files from a computer system at a source location to a target location, by a method comprising:

receiving a request to transfer a file from a computer system at a source location to a target location, wherein the target location includes a single instance database, and wherein the source location and the target location are geographically remote from each other;

sending a request to the single instance database to determine whether the file matches any file already stored by the single instance database and wherein any metadata extracted from the file matches any metadata associated with any file already stored by the single instance database;

receiving a determination from the single instance database as to whether the file matches any file already stored by the single instance database;

when the file does not match any file already stored at the target location, storing the file from the computer system to the target location;

receiving a determination from the single instance database as to whether the extracted metadata matches any metadata associated with any file stored by the single instance database; and

when the extracted metadata does not match the metadata associated with any file stored by the single instance database, storing the extracted metadata from the computer system to the target location.

15. The computer-readable storage medium of claim 14 wherein the single instance database stores files received from computer systems at locations other than the source location and tracks such locations, and wherein the single instance database contains a reference count for each file indicating how many different tracked locations refer to the file.

16. The computer-readable storage medium of claim 14 wherein the request to store the file is received as part of a continuous data replication operation.

17. The computer-readable storage medium of claim 14 wherein sending a request to the single instance database includes:

generating a substantially unique identifier that represents the file; and

sending the substantially unique identifier to the single instance database.

18. The computer-readable storage medium of claim 14 wherein sending a request to the single instance database includes:

identifying data objects within the file;

generating substantially unique identifiers that represent each identified data object; and

sending the substantially unique identifiers to the single instance database.

19. The computer-readable storage medium of claim 14, wherein the method further comprises:

when the file matches a file already stored at the target location;

incrementing a reference count in an index that refers to the file; and

indicating to the requestor that the request to copy the file to the target location succeeded, without actually copying the file to the target location.

20. The computer-readable storage medium of claim 14 wherein sending a request to a single instance database at the target location includes:

determining the size of the file; and

sending the file size to the single instance database.