Buffer configuration for a data replication system

• US 20070183224A1
• Filed: 12/18/2006
• Published: 08/09/2007
• Est. Priority Date: 12/19/2005
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Erofeev, Andrei
• Granted Patent: US 7,870,355 B2
• Time in Patent Office: Days
• US Class Current: 365/189.50
• CPC Class Codes: G06F 11/1456 Hardware arrangements for backup
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 11/1662 the resynchronized component or unit being a persistent storage device re-synchronization of failed mirror storage G06F11/2082; rebuild or reconstruction of parity RAID storage G06F11/1008
• CPC Class Codes: G06F 11/2064 while ensuring consistency
• CPC Class Codes: G06F 11/2074 Asynchronous techniques
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/184 implemented as replicated file system
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 2201/82 Solving problems relating to consistency
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data

Abstract

Performing data management operations on replicated data in a computer network. Log entries are generated for data management operations of an application executing on a source system. Consistency point entries are used to indicate a time of a known good, or recoverable, state of the application. A destination system is configured to process a copy of the log and consistency point entries to replicate data in a replication volume, the replicated data being a copy of the application data on the source system. When the replicated data represents a known good state of the application, as determined by the consistency point entries, the destination system(s) may perform a storage operation (e.g., snapshot, backup) to copy the replicated data and to logically associate the copied data with a time information (e.g., time stamp) indicative of the source system time when the application was in the known good state.

Claims

1. A method for transmitting data from a source computer to a destination computer in a data replication system, the method comprising:

monitoring file system requests of an application on a source computer;

identifying, from the file system requests, data management operations directed to data on a first storage device;

storing in a buffer a plurality of log entries representing the data management operations;

sequentially processing each of the plurality of log entries;

transmitting each processed log entry to a destination computer;

replaying each processed log entry to replicate the data on the first storage device to a second storage device; and

when said sequentially processing drops below a predetermined rate, disk swapping most recently received log entries from the buffer to a memory.

2. The method of claim 1, wherein said sequentially processing comprises translating an inode representing a data location on the first storage device to a pathname representing a data location on the second storage device.

3. The method of claim 2, wherein said translating comprises accessing pathname information in a database.

4. The method of claim 1, wherein the memory is on the first storage device.

5. The method of claim 1, wherein the predetermined rate is associated with a threshold storage amount of the buffer.

6. The method of claim 1, additionally comprising:

storing the plurality of log entries in a queue; and

transferring the plurality of log entries from the queue to the buffer.

7. The method of claim 6, additionally comprising throttling the application when the number of log entries in the queue exceeds a threshold storage amount of the queue.

8. The method of claim 1, wherein the buffer comprises a first-in first-out buffer.

9. A system for transmitting data to be replicated in a computer network, the system comprising:

a queue;

a monitor module configured to monitor file system requests generated by a computer application, the monitor module being further configured to populate the queue with log entries indicative of file system requests comprising data management operations to be executed on a first storage device;

a buffer;

a first thread configured to transfer the log entries from the queue to the buffer; and

a second thread configured to retrieve the log entries from the buffer, process the log entries, and forward the processed log entries to a destination system for replaying the log entries to replicate data on a second storage device,

wherein the monitor module is further configured to throttle the computer application when a rate at which the second thread processes the log entries is below a predetermined rate.

10. The system of claim 9, wherein the predetermined rate is associated with a storage threshold of the queue.

11. The system of claim 10, wherein the monitor module is configured to track the number of log entries stored in the queue.

12. The system of claim 11, further comprising a counter usable by the monitor module to track the number of log entries stored in the queue.

13. The system of claim 9, wherein the predetermined rate is associated with a storage threshold of the buffer.

14. The system of claim 9, wherein the buffer comprises an application-level buffer.

15. The system of claim 14, wherein the application-level buffer comprises a first-in first-out buffer.

16. A system for transmitting data to be replicated in a network environment, the system comprising:

means for monitoring file system requests of an application on a source computer and for identifying, from the file system requests, data management operations directed to data on a first storage device;

means for storing a plurality of log entries representing the data management operations;

means for sequentially processing each of the plurality of log entries;

means for transmitting each processed log entry to a destination computer;

means for replaying each processed log entry to replicate the data on the first storage device to a second storage device; and

means for disk swapping most recently received log entries from said storing means to a memory when said sequentially processing falls below a predetermined rate.

17. The system of claim 16, wherein said processing means is further configured to translate an inode representing a data location on the first storage device to a pathname representing a data location on the second storage device.

18. The system of claim 16, further comprising means for tracking the number of log entries stored in said storing means.

19. The system of claim 18, further comprising means for throttling the application when the number of log entries in said storing means exceeds a threshold storage amount.

20. The system of claim 16, wherein said data management operations are associated with a single computer application.