Hybrid real-time data replication

• US 20050149578A1
• Filed: 11/03/2004
• Published: 07/07/2005
• Est. Priority Date: 11/04/2003
• Status: Active Grant

• Current Assignee: Quest Software, Inc.
• Original Assignee: Quest Software, Inc.
• Inventors: Haddi, Abderrahman Aba A. El, Banks, Justin, Sustman, Paul A., Lee, Victor T. Jr.
• Granted Patent: US 7,836,014 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/2071 using a plurality of controllers
• CPC Class Codes: G06F 11/2079 Bidirectional techniques

Abstract

Techniques are described for replicating data from one to one or more heterogeneous data processing or communication devices for the purpose of remote backup, redundancy, content distribution, communications, observations or measurements. In a first phase, the attributes of the data that are modified or created on a device or that are passing through the device are tracked and journaled in volatile or non-volatile storage in real-time. In a second phase, the attributes that match patterns pre-specified in a configuration are used to determine which data to replicate and which modifications to make the devices. In a third phase, the data is replicated. In a preferred embodiment, the described techniques comprise an application that runs on a host device or is embedded in a logic or memory device. The described invention is designed to be transparent for system redundancy and error recovery processes such as error correction, re-transmission on links, and raid configuration.

Claims

1. A data replication method comprising:

accepting a request from a client device to modify data;

adding data attributes of the modified data to a message queue;

saving the data attributes of modifications on a storage device;

performing modifications and saving a status of the data modification operation; and

communicating the status of the operation to the client device if the client device requests that the status be communicated.

2. The method of claim 1, wherein accepting the request from a client device comprises accepting a request to forward data from the client device to another client device.

3. The method of claim 1, further comprising:

fetching at least one of modified and new data attributes from a device that performed the data alteration;

adding the fetched data attributes to a journal message queue;

removing each message from the journal message queue; and

saving data attributes of the message if the attributes match specified patterns for a destination device, wherein the data attributes are saved on storage of the device that performed the data alteration.

4. The method of claim 3, further comprising:

retrieving one or more data attributes from the storage device for a destination device if replication to the destination device has not been suspended; and

sending content described by the data attribute to the destination device if replication to the destination device has not been suspended.

5. The method of claim 4, further comprising initializing replication to perform synchronization of the content of source and destination devices by reading a list of all data attributes on the source device and adding the list of the data attributes to the journal message queue for additional processing if the data attributes match the patterns of data attributes to be replicated.

6. A machine-readable medium comprising instructions for causing a programmable processor to:

accept a request from a client device to modify data;

add data attributes of the modified data to a message queue;

save the data attributes of modifications on a storage device;

perform modifications and saving a status of the data modification operation; and

communicate the status of the operation to the client device if the client device requests that the status be communicated.

7. The machine-readable medium of claim 6, wherein the instructions cause the processor to accept the request from a client device comprises accepting a request to forward data from the client device to another client device.

8. The machine-readable medium of claim 6, further comprising instructions that cause the processor to:

fetch at least one of modified and new data attributes from a device that performed the data alteration;

add the fetched data attributes to a journal message queue;

remove each message from the journal message queue; and

save data attributes of the message if the attributes match specified patterns for a destination device, wherein the data attributes are saved on storage of the device that performed the data alteration.

9. The machine-readable medium of claim 8, further comprising instructions that cause the processor to:

retrieve one or more data attributes from the storage for a destination device if replication to the destination device is not suspended; and

send the content described by the data attribute to the destination device if replication to the destination device is not suspended.

10. The machine-readable medium of claim 9, further comprising instructions that cause the processor to initialize replication to perform synchronization of the content of source and destination devices by reading a list of all data attributes on the source device and adding the list of the data attributes to the journal message queue for additional processing if the data attributes match the patterns of data attributes to be replicated.

11. A device for replication of data across a distributed computing system, the device comprising:

a pass-through component for intercepting data modification requests from a client device; and

a data replication engine for receiving the data modification requests from the pass-through component and replicating the modification on one or more remote storage devices, wherein the data replication engine;

accepts the request to modify data from the pass-through component;

adds data attributes of modified data to a message queue;

saves the data attributes on one or more of the storage devices; and

performs modifications and saves a status of the data alteration operation.

12. The device of claim 11, wherein the data replication engine communicates the status of the operation to the client device that requested the change when the client device requests that the status be communicated.

13. The device of claim 11, wherein the data replication engine captures the data changes synchronously by saving the data attributes, and performs data replication asynchronously.

14. The device of claim 11, wherein the data replication engine includes a transport thread to replicate the data to a destination device.

15. The device of claim 12, wherein the data replication engine comprises:

an input thread that retrieves one or more modification attributes from the pass-through component;

a journal queue that stores the retrieved attributes; and

a journal thread that retrieves the attributes from the journal queue and inserts each attribute into each of one or more pathways, wherein each of the pathways comprises;

a pathway journal that stores the inserted attributes;

a remote thread to retrieve the attributes stored in the pathway journal and pass each of the attributes to a transport thread that replicates data to a represented by the attribute to a destination device;

a work journal that stores the attributes retrieved by the remote thread if the attributes are not presently stored in the work journal; and

a complete thread that removes each attributes stored in the work journal and the pathway journal if a reference count for the attribute is zero.

16. A device for replication of data across a distributed computing system, the device comprising:

a pass-through component for intercepting data modification requests from a client device to a set of intermediate client devices; and

a data replication engine for receiving the data modification requests from the pass-through component and replicating the modification on one or more remote storage devices, wherein the data replication engine;

accepts the request to modify data from the pass-through component;

adds data attributes of modified data to a message queue;

saves the data attributes on one or more of the storage devices;

replicates the saved data attributes on a set of intermediate storage devices; and

performs modifications and saves a status of the data alteration operation.

17. A data replication method comprising:

accepting a request from a client device to modify data from the client device to a set of intermediate client devices;

adding data attributes of the modified data to a message queue;

saving the data attributes of modifications on a storage device;

replicating the saved data attributes on a set of intermediate storage devices performing modifications and saving a status of the data modification operation; and

communicating the status of the operation to the client device if the client device requests that the status be communicated.

18. The method of claim 17, further comprising replicating the saved data attributes on each of the intermediate storage devices to another set of storage devices for each of the intermediate storage devices.