Log based data replication system with disk swapping below a predetermined rate

• US 7,870,355 B2
• Filed: 12/18/2006
• Issued: 01/11/2011
• Est. Priority Date: 12/19/2005
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Erofeev, Andrei
• Primary Examiner(s): Elmore; Stephen C
• Assistant Examiner(s): Ayash; Marwan
• Application Number:
  US11/640,825
• Publication Number:
  US 20070183224A1
• Time in Patent Office: 1,485 Days
• Field of Search: None
• US Class Current: 711/162
• CPC Class Codes: G06F 11/1456 Hardware arrangements for backup
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 11/1662 the resynchronized component or unit being a persistent storage device re-synchronization of failed mirror storage G06F11/2082; rebuild or reconstruction of parity RAID storage G06F11/1008
• CPC Class Codes: G06F 11/2064 while ensuring consistency
• CPC Class Codes: G06F 11/2074 Asynchronous techniques
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/184 implemented as replicated file system
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 2201/82 Solving problems relating to consistency ensuring consistency in mirrored systems G06F11/2064
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data

Abstract

Performing data management operations on replicated data in a computer network. Log entries are generated for data management operations of an application executing on a source system. Consistency point entries are used to indicate a time of a known good, or recoverable, state of the application. A destination system is configured to process a copy of the log and consistency point entries to replicate data in a replication volume, the replicated data being a copy of the application data on the source system. When the replicated data represents a known good state of the application, as determined by the consistency point entries, the destination system(s) may perform a storage operation (e.g., snapshot, backup) to copy the replicated data and to logically associate the copied data with a time information (e.g., time stamp) indicative of the source system time when the application was in the known good state.

Claims

1. A method for transmitting data from a source computer to a destination computer in a data replication system, the method comprising:

monitoring file system requests of an application program on a source computer;

identifying, from the file system requests, application-specific data management operations directed to data on a first storage device, wherein the application-specific data management operations identify at least one recoverable state of the application program;

storing in a buffer a plurality of log entries representing the application-specific data management operations, wherein at least a plurality of the log entries in the buffer do not have an absolute file pathname;

pathname processing each of the plurality of the log entries to determine an absolute file pathname associated with the plurality of log entries;

transmitting each processed log entry and the associated absolute file pathname to a destination computer;

replaying on the destination computer each processed log entry with the associated absolute file pathname to perform the application-specific data management operations on the destination computer to generate a replication of the data existing on the first storage device on a second storage device;

when the pathname processing drops below a predetermined rate, transferring recently received log entries that do not have an absolute file pathname to a memory;

modifying the execution of the application program on the source computer when the log entries that do not have an absolute file pathname in the memory exceed a threshold; and

recovering the application program on the source computer to the recoverable state with the data replicated on the destination computer in the event the application program on the source computer needs to be recovered.

2. The method of claim 1, wherein said pathname processing comprises translating an inode representing a data location on the first storage device to a pathname representing a data location on the second storage device.

3. The method of claim 2, wherein said translating comprises accessing pathname information in a database.

4. The method of claim 1, wherein the memory is on the first storage device.

5. The method of claim 1, wherein the predetermined rate is associated with a threshold storage amount of the buffer.

6. The method of claim 1, additionally comprising:

storing the plurality of log entries in a queue; and

transferring the plurality of log entries from the queue to the buffer.

7. The method of claim 6, additionally comprising throttling the application when the number of log entries in the queue exceeds a threshold storage amount of the queue.

8. The method of claim 1, wherein the buffer comprises a first-in first-out buffer.

9. A system for transmitting data to be replicated in a computer network, the system comprising:

a monitor module located between an application program and a file system, the monitor module executing in one or more computer processors, wherein the monitor module is configured to monitor file system requests generated by the application program executing on a first computer, the monitor module being further configured to generate log entries indicative of file system requests comprising data management operations to be executed on the first storage device, wherein the data management operations identify at least one recoverable state of the application program;

a buffer, wherein the buffer stores the log entries representing the data management operations, and wherein at least a plurality of the log entries in the buffer do not have an absolute file pathname;

a pathname processor excuting in one or more computer processors, the pathname processor configured to retrieve the log entries from the buffer, process the log entries to determine an absolute file pathname associated with each log entry, and forward the processed log entries with the absolute file pathname to a second computer for replaying the log entries with the absolute file pathname on the second computer to generate replicate data on a second storage device that is a replication of the data existing on the first storage device,

wherein the monitor module is further configured to modify the processing of the data management operations by the application program on the first computer when a rate at which the pathname processor processes the log entries that do not have an absolute file name is below a predetermined rate; and

wherein the system is configured to recover the application program to the recoverable state with the data replicated on the second computer in the event the application program needs to be recovered.

10. The system of claim 9, wherein the predetermined rate is associated with a storage threshold of the queue.

11. The system of claim 10, wherein the monitor module is configured to track the number of log entries stored in a queue.

12. The system of claim 11, further comprising a counter usable by the monitor module to track the number of log entries stored in the queue.

13. The system of claim 9, wherein the predetermined rate is associated with a storage threshold of the buffer.

14. The system of claim 9, wherein the buffer comprises an application-level buffer.

15. The system of claim 14, wherein the application-level buffer comprises a first-in first-out buffer.

16. A system for transmitting data to be replicated in a network environment, the system comprising:

means for monitoring file system requests of an application program executing on a source computer and for identifying, from the file system requests, data management operations directed to data on a first storage device, wherein the data management operations identify at least one recoverable state of the application program;

means for storing a plurality of log entries representing the data management operations, wherein at least a plurality of the log entries buffer do not have an absolute file pathname;

means for pathname processing each of the plurality of log entries to determine an absolute file pathname associated with the plurality of log entries;

means for transmitting each processed log entry and the associated absolute file pathname to a destination computer;

means for replaying on the destination computer each processed log entry with the associated absolute file pathname to perform the data management operations on the destination computer to generate a replication of the data existing on the first storage device on a second storage device;

means for disk swapping most recently received log entries from said storing means to a memory when said pathname processing falls below a predetermined rate;

means for modifying the execution of the application program on the source computer when the log entries that do not have an absolute file pathname in the memory exceed a threshold; and

means for recovering the application program on the source computer to the recoverable state with the data replicated on the destination computer in the event the application program on the source computer needs to be recovered.

17. The system of claim 16, wherein said processing means is further configured to translate an inode representing a data location on the first storage device to a pathname representing a data location on the second storage device.

18. The system of claim 16, further comprising means for tracking the number of log entries stored in said storing means.

19. The system of claim 18, further comprising means for throttling the application when the number of log entries in said storing means exceeds a threshold storage amount.

20. The system of claim 16, wherein said data management operations are associated with a single computer application.