Systems and methods for performing data replication

• US 8,572,038 B2
• Filed: 05/27/2011
• Issued: 10/29/2013
• Est. Priority Date: 05/28/2010
• Status: Expired due to Fees

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Erofeev, Andrei
• Primary Examiner(s): Lee, Wilson
• Assistant Examiner(s): Hershley, Mark E
• Application Number:
  US13/118,182
• Publication Number:
  US 20110295804A1
• Time in Patent Office: 886 Days
• Field of Search: 707/634
• US Class Current: 707/634
• CPC Class Codes: G06F 11/1435 using file system or storage system metadata
• CPC Class Codes: G06F 11/1458 Management of the backup or restore process
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 11/1662 the resynchronized component or unit being a persistent storage device re-synchronization of failed mirror storage G06F11/2082; rebuild or reconstruction of parity RAID storage G06F11/1008
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
• CPC Class Codes: G06F 16/1844 Management specifically adapted to replicated file systems
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes

Abstract

Preparing source data to be replicated in a continuous data replication environment. Certain systems and methods populate a file name database with entries having a unique file identifier descriptor (FID), short name and a FID of the parent directory of each directory or file on a source storage device. Such information is advantageously gathered during scanning of a live file system without requiring a snapshot of the source storage device. The database can be further used to generate absolute file names associated with data operations to be replayed on a destination storage device. Based on the obtained FIDs, certain embodiments can further combine write operations to be replayed on the destination storage device and/or avoid replicating temporary files to the destination system.

Claims

1. A method for performing data replication, the method comprising:

monitoring with one or more processors of a source storage system a plurality of journal entries associated with writing data to a source storage device associated with the source storage system;

identifying a first journal entry of the plurality of journal entries, the first journal entry comprising a first data write operation, a first file identifier descriptor (FID) of a file to be modified by the first data write operation on the source storage device, and a first location of a first portion of the file to be modified,

wherein a FID identifies a file or directory of a file system on the source storage system and is usable to construct an absolute file name for transmitting data to a replication system;

identifying a second journal entry of the plurality of journal entries, the second journal entry comprising a second data write operation, a second FID of a file to be modified by the second data write operation on the source storage device, and a second location of a second portion of the file to be modified,

wherein the first journal entry and the second journal entry are from the same journal;

determining with the one or more processors that the first and second data write operations can be combined into a single write operation based on a determination that the first and second FIDs both correspond to a first value;

combining the first and second data write operations based on said determination;

constructing with the one or more processors an absolute file name by associating the first value with a short name and at least one directory name, wherein neither the first nor second journal entries comprises the absolute file name; and

transmitting the single write operation and the absolute file name to a destination storage device to replay on the destination storage device the data modifications associated with the first and second write operations,

wherein the destination storage device stores a replicated version of data written to the source storage device.

2. The method of claim 1, wherein the first location comprises a first byte range and the second location comprises a second byte range.

3. The method of claim 2, wherein said determining that the first and second data write operations can be combined further comprises analyzing a distance between the first and second byte ranges.

4. The method of claim 3, wherein determining that the first and second data write operations can be combined further comprises determining that the distance is less than a threshold amount.

5. The method of claim 4, wherein the threshold amount is less than 1 KB.

6. The method of claim 5, wherein the threshold amount is approximately 200 bytes.

7. The method of claim 1, wherein said monitoring operations comprises processing the plurality of journal entries with a filter driver.

8. The method of claim 1, wherein said monitoring comprises analyzing a portion of the plurality of journal entries as received over a predetermined period of time.

9. The method of claim 8, additionally comprising adjusting the predetermined period of time based on a known good state of the source storage device data.

10. The method of claim 8, wherein said predetermined period of time is approximately four seconds.

11. The method of claim 1, additionally comprising:

identifying multiple ones of the plurality of journal entries having other write operations that cannot be combined;

reading byte ranges associated with the other write operations in order of increasing file offsets; and

transmitting the other write operations to the destination storage device for replay on the destination storage device in the order of the increasing file offsets.

12. A system for performing data replication, the system comprising:

at least one computer application executing on a computing device and configured to generate operations associated with data on a source storage device associated with a source storage system;

a filter module disposed between the at least one computer application and the source storage device, the filter module configured to;

monitor a plurality of journal entries associated with writing data to the source storage device;

identify a first journal entry of the plurality of journal entries, the first journal entry comprising a first data modification operation, a first file identifier descriptor (FID) of a file to be modified by the first data modification operation, and a first location of a first portion of the file to be modified,

wherein a FID identifies a file or directory of a file system on the source storage system and is usable to construct an absolute file name for transmitting data to a replication system;

identify a second journal entry of the plurality of journal entries, the second journal entry comprising a second data modification operation, a second FID of a file to be modified by the second data modification operation, and a second location of a second portion of the file to be modified,

wherein the first journal entry and the second journal entry are from the same journal;

a processing module configured to;

determine that the first and second data modification operations can be combined into a single modification operation based on a determination that the first and second FIDs both correspond to a first value; and

combine the first and second data modification operations based on said determination; and

at least one database thread configured to construct an absolute file name for replaying the single modification operation on replication data of a destination storage device by associating the first value with a short name and at least one directory name, wherein neither the first nor second journal entries comprises the absolute file name,

wherein the destination storage device stores a replicated version of data written to the source storage device.

13. The system of claim 12, wherein the filter module comprises a file system filter driver.

14. The system of claim 13, wherein the first data modification operation comprises a write operation, and wherein the file system filter driver is configured to obtain the first data modification operation and the first FID without receiving a copy of data to be written to the source storage device.

15. The system of claim 12, further comprising a database comprising file name data that associates each of the first and second FIDs with a short name and a parent FID.

16. The system of claim 12, wherein the first file location comprises an offset and a length of a byte range of the file to be modified.

17. The system of claim 12, further comprising a memory configured to store a portion of the operations generated over a predetermined period of time.

18. The system of claim 17, wherein the predetermined period of time is at least three seconds.

19. A system for performing data replication, the system comprising:

means for monitoring a plurality of journal entries associated with writing data to a source storage device associated with a source storage system;

means for identifying a first journal entry of the plurality of journal entries, the first journal entry comprising a first data write operation, a first file identifier descriptor (FID) of a file to be modified on the source storage device, and a first location of a first portion of the file to be modified, and for identifying a second journal entry of the plurality of journal entries, the second journal entry comprising a second data write operation, a second FID of a file to be modified on the source storage device, and a second location of a second portion of the file to be modified,

wherein a FID identifies a file or directory of a file system on the source storage system and is usable to construct an absolute file name for transmitting data to a replication system, and

wherein the first journal entry and the second journal entry are from the same journal;

means for determining that the first and second data write operations can be combined into a single write operation based on a determination that the first and second FIDs both correspond to a first value, and for combining the first and second data write operations based on said determination;

means for constructing an absolute file name by associating the first value with a short name and at least one directory name, wherein neither the first nor second journal entries comprises the absolute file name; and

means for transmitting the single write operation and the absolute file name to a destination storage device to replay on the destination storage device the data modifications associated with the first and second write operations,

wherein the destination storage device stores a replicated version of data written to the source storage device.

20. The system of claim 19, further comprising means for storing file name data that associates each of the first and second FIDs with a short name and a parent FID usable to construct the absolute file name.