Systems and methods for selective data replication

• US 8,868,494 B2
• Filed: 08/02/2013
• Issued: 10/21/2014
• Est. Priority Date: 03/29/2010
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Agrawal, Vijay H.
• Primary Examiner(s): Bullock, Joshua
• Application Number:
  US13/957,824
• Publication Number:
  US 20140074777A1
• Time in Patent Office: 445 Days
• Field of Search: 707/610, 707/691
• US Class Current: 707/610
• CPC Class Codes: G06F 16/178 Techniques for file synchronisation in file systems
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation

Abstract

Systems and methods for performing data replication are disclosed. Determining whether to update replicated data typically involves comparison of readily obtainable attributes of a given source file and its corresponding replicated file. Such attributes can be obtained from, for example, metadata. In certain situations, an additional assessment of the source and replicated files can be beneficial. For example, if integrity of an existing replicated file'"'"'s content is maintained, one may not want to re-replicate the corresponding source file. For large source files, such a decision can provide substantial reductions in expenditures of available computing and network resources. In certain embodiments, a threshold for identifying such large files can be based on one or more operating parameters such as network type and available bandwidth. In certain embodiments, replication file'"'"'s integrity can be checked by calculating and comparing checksums for the replication file and its corresponding source file.

Claims

1. A computer-implemented method for performing data replication, the computer-implemented method comprising:

performing a first assessment with computer hardware on a first set of files stored on a first storage device that is associated with a source system and on a second set of files stored on a second storage device that is associated with a destination system in networked communication with the source system, the second set of files corresponding to the first set of files and having been previously replicated from the first set of files, the first assessment comprising,

comparing one or more attributes of the first set of files with one or more corresponding attributes of the second set of files, and

determining whether a first attribute from the one or more attributes of a file from the first set of files differs from a corresponding attribute of the one or more corresponding attributes of a corresponding file from the second set of files;

in response to determining that the first attribute differs from the corresponding attribute, determining with the computer hardware whether a size of the file satisfies a size threshold value;

in response to determining that the size of the file satisfies the size threshold value, performing with the computer hardware a second assessment on the file based at least in part on a checksum for the file; and

based on the results of the second assessment, determining whether to replicate the file from the first storage device to the second storage device.

2. The computer-implemented method of claim 1, wherein performing the second assessment comprises:

accessing the checksum for the file;

accessing a checksum for the corresponding file; and

comparing the checksum for the file with the checksum for the corresponding file to determine whether the checksum for the file differs from the checksum for the corresponding file.

3. The computer-implemented method of claim 2, wherein, in response to determining that the checksum for the file differs from the checksum for the corresponding file, the method further comprises replicating the file from the first storage device to the second storage device.

4. The computer-implemented method of claim 3, wherein replicating the file comprises replicating data blocks of the file with different checksum values than checksum values of corresponding data blocks of the corresponding file.

5. The computer-implemented method of claim 2, wherein, in response to determining that the checksum for the file matches the checksum for the corresponding file, the method further comprises synchronizing the attribute of the file with the corresponding attribute of the corresponding file without replicating the file from the first storage device to the second storage device.

6. The computer-implemented method of claim 2, wherein accessing the checksum for the file comprises accessing a checksum for each data block of the file and wherein accessing the checksum for the corresponding file comprises accessing a checksum for each data block of the corresponding file.

7. The computer-implemented method of claim 1, wherein the size threshold value is determined based at least in part on balancing processing resource expenditures for performing the second assessment versus bandwidth resources for replicating the file.

8. The computer-implemented method of claim 1, wherein if the size of the file does not satisfy the size threshold value, the file is replicated from the first storage device to the second storage device without performing the second assessment.

9. A data replication system comprising:

a replication agent comprising computer hardware, the replication agent in communication with a source storage system and a destination storage system, the replication agent configured to;

perform a first assessment of a file stored on the source storage system to determine whether the file has a metadata attribute that differs from a corresponding metadata attribute of a corresponding file on the destination storage system;

in response to a determination that the metadata attribute differs from the corresponding metadata attribute, the replication agent is further configured to;

obtain a size of the file; and

compare the size of the file to a size threshold value;

in response to a determination that the size of the file does not satisfy the size threshold value, the replication agent is further configured to replicate the file from the source storage device to the destination storage device regardless of whether a checksum for the file matches a checksum for the corresponding file; and

in response to a determination that the size of the file satisfies the size threshold value, the replication agent is further configured perform a second assessment of the file based at least in part on the checksum for the file.

10. The data replication system of claim 9, wherein the replication agent is further configured to perform the second assessment by:

accessing the checksum for the file;

accessing the checksum for the corresponding file; and

comparing the checksum for the file with the checksum for the corresponding file to determine whether the checksum for the file differs from the checksum for the corresponding file.

11. The data replication system of claim 10, wherein, in response to determining that the checksum for the file differs from the checksum for the corresponding file, the replication agent is further configured to replicate the file from the source storage device to the destination storage device.

12. The data replication system of claim 11, wherein the replication agent is further configured to replicate the file by replicating data blocks of the file with different checksum values than data blocks of the corresponding file.

13. The data replication system of claim 10, wherein, in response to determining that the checksum for the file matches the checksum for the corresponding file, the replication agent is further configured to synchronize the metadata attribute of the file with the corresponding metadata attribute of the corresponding file without replicating the file from the source storage device to the destination storage device.

14. The data replication system of claim 10, wherein accessing the checksum for the file comprises accessing a checksum for each data block of the file and wherein accessing the checksum for the corresponding file comprises accessing a checksum for each data block of the corresponding file.

15. The data replication system of claim 9, wherein the replication agent is further configured to determine the size threshold value based at least in part on balancing processing resource expenditures for performing the second assessment versus bandwidth resources for replicating the file.

16. The data replication system of claim 9, wherein the replication agent is further configured to:

access one or more storage policies; and

determine the size threshold value based at least in part on the one or more storage policies.

17. The data replication system of claim 9, further comprising a data storage system, the data storage system comprising the source storage system and the destination storage system.

18. The data replication system of claim 9, further comprising a user interface configured to receive user input, and wherein the replication agent is further configured to determine the size threshold value based at least in part on the user input.

19. Non-transitory computer storage comprising instructions which, when executed, cause the computing system to perform steps comprising:

performing a first assessment on a first file stored on a first storage device that is associated with a source system and on a second file stored on a second storage device that is associated with a destination system, the second file corresponding to the first file and having been previously replicated from the first file, the first assessment comprising,

comparing an attribute of the first file with a corresponding attribute of the second file, and

determining whether the attribute from the first file differs from the corresponding attribute of the second file;

in response to determining that the attribute differs from the corresponding attribute, determining whether a size of the first file satisfies a threshold value;

in response to determining that the size of the first file satisfies the threshold value, performing a second assessment on the first file based at least in part on the checksum for the first file; and

based on the results of the second assessment, determining whether to replicate the file from the first storage device to the second storage device.

20. The non-transitory computer storage of claim 19, wherein performing the second assessment comprises:

accessing the checksum for the first file;

accessing a checksum for the second file;

comparing the checksum for the first file with the checksum for the second file to determine whether the checksum for the first file differs from the checksum for the second file;

in response to determining that the checksum for the first file differs from the checksum for the second file, replicating the first file from the first storage device to the second storage device; and

in response to determining that the checksum for the first file matches the checksum for the second file, synchronizing the attribute of the first file with the corresponding attribute of the second file without replicating the first file.