System and method for database replication by interception of in memory transactional change records

• US 20060047713A1
• Filed: 07/25/2005
• Published: 03/02/2006
• Est. Priority Date: 08/03/2004
• Status: Abandoned Application

• Current Assignee: WisdomForce Technologies, Inc. (Informatica, Inc. (California))
• Original Assignee: WisdomForce Technologies, Inc. (Informatica, Inc. (California))
• Inventors: Gornshtein, David, Tamarkin, Boris
• Application Number:
  US11/189,220
• Publication Number:
  US 20060047713A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Abstract

A system and method are directed towards providing a database replication technique using interception in memory of the transaction change data records. The invention employs Input/Output instrumentation to capture and split out the in memory transaction change journal records. Captured memory blocks are sent to a parser, which concatenates the records into a single record, and creates a redo/undo vector that can be converted to original DML/DDL statements. Source level transformations can be applied to the vectors, which are then sent to a post agent on the same or a different computing device. The post agents may perform destination level transformations, and generate DML/DDL statements to be executed by the corresponding destination RDBMS instance. Post agents may also perform conflict detection and resolution during DML/DDL statement executions. Transaction consistency is supported by performing commits/rollback on the destination after receiving the redo/undo vector representing a commit/rollback on the source.

Claims

1. A method for database replication, comprising:

intercepting a write operation before a log buffer flush to a transactional change log by employing an I/O instrumentation component;

selecting a portion of information from the intercepted write operation; and

forwarding the selected portion of information to a destination database system for use in replicating the source database system.

2. The method of claim 1, wherein the I/O instrumentation component includes at least one of an instance level storage manager, an operating system function, or a kernel level device driver.

3. The method of claim 1, where further comprising:

updating the destination database system with the selected portion of information after receiving at least one of a commit or rollback statement from the source database system.

4. The method of claim 1, wherein forwarding the selected portion of information further comprises sending the portion of information synchronously for use with real-time replication or asynchronously for other than real-time replication.

5. The method of claim 1, further comprising:

receiving the portion of information at the destination database system;

determining at least one log record within the portion of information;

performing a loopback filtering of at least one log record to determine, at least in part, a redo vector; and

posting the redo vector to the destination database system for use in replicating the source database.

6. The method of claim 5, wherein performing loopback filtering further comprises:

if the source database system supports XA style transactions;

generating a transaction identifier (id) before execution of a first Data Manipulation Language (DML) operation associated with at least one log record, and

employing the transaction identifier to extract selected statements from the redo vector.

7. The method of claim 5, wherein performing loopback filtering further comprises:

if the source database system not does support XA style transactions;

opening a transaction control statement to generate a transaction identifier, and

employing the transaction identifier to parse the redo vector to remove operations that are not to be performed.

8. The method of claim 5, wherein the redo vector is posted to a destination database system within the destination database system as at least one of a Data Manipulation Language (DML) operation or a Data Definition Language (DDL) operation.

9. The method of claim 1, wherein selecting a portion of information further comprises filtering of the records based on a loopback avoidance state hash.

10. A server for database replication, comprising:

a transceiver to send and receive information; and

a processor programmed to perform actions including;

performing a transaction on a source database, wherein the source database is to be replicated;

sending to an in memory transactional change log an instance associated with the performed transaction;

intercepting the instance using an Input/Output (I/O) interceptor that includes at least one of an instance level storage manager, an operating system function, or a kernel level device driver;

generating a vector from the instance; and

sending the vector to an agent, wherein the agent employs the vector to modify a destination database.

11. The server of claim 10, wherein employing the vector further comprises:

transforming the vector to an operation; and

posting the operation to the destination database, wherein the operation includes at least one of a Data Manipulation Language (DML) operation or a Data Definition Language (DDL) operation.

12. The server of claim 10, wherein sending the vector to the agent further comprises:

sending the vector over a channel that comprises at least one of a TCP/IP channel, a named pipe, shared memory, or through a persistent queue.

13. The server of claim 10, wherein generating the vector from the instance, further comprises duplicating a memory block by mapping information associated with the instance to a state hash table.

14. The server of claim 10, wherein employing the vector to modify the destination database further comprises employing a collision avoidance and resolution policy that includes at least one of discarding a conflict update, performing an update based on an earliest timestamp, performing the update based on a latest timestamp, or performing an update based on a priority associated with the update of the destination database.

15. The server of claim 10, further comprising:

if the source database and the destination database are master databases, implementing a master-to-master replication mechanism using real-time ping pong avoidance.

16. A system for database replication, comprising:

(a) a source database system that comprises;

(i) a transaction change log that is in communication with a source database and is configured to receive changes to the source database;

(ii) an Input/Output (I/O) interceptor that is configured to perform actions, including;

intercepting a write operation to the transaction change log;

splitting the write operation to generate a copy of the write operation; and

sending the copy of the write operation within a log buffer to a parsing engine; and

(iii) the parsing engine configured to communicate with the I/O interceptor and to perform actions, including;

parsing the log buffer into at least one log record;

performing loopback post filtering on the at least one log record;

generating a redo vector from at least one log record; and

sending the redo vector to a destination database system; and

(b) the destination database system that is in communication with the source database system and comprises;

(i) a replication post agent that is configured to perform actions, including;

receiving the redo vector;

generating a record based on the redo vector; and

posting the record to a destination database; and

(ii) the destination database that is configured to perform actions, including;

receiving the record; and

employing the record to replicate the source database.

17. The system of claim 16, wherein generating a record based on the redo vector further comprises generating the record to include at least one of a Data Manipulation Language (DML) operation or a Data Definition Language (DDL) operation.

18. The system of claim 16, wherein the I/O interceptor is configured to perform actions further comprising:

if a no-logging transaction to the database is detected;

intercepting the no-logging transaction,

duplicating a data block associated with the no-logging transaction for use in replicating the no-logging transaction on the destination database, and

providing the duplicated data block within the log buffer to the parsing engine.

19. The system of claim 16, wherein performing loopback post filtering further comprises:

if the source database system supports XA style transactions;

receiving a transaction identifier before execution of a first Data Manipulation Language (DML) operation associated with the at least one log record, and

employing the transaction identifier to filter the redo vector prior to sending the redo vector.

20. The system of claim 16, wherein performing loopback post filtering further comprises:

if the source database system does not support XA style transactions;

receiving a transaction identifier based on a “

begin transaction”

statement, and

employing the transaction identifier to filter the redo vector prior to sending the redo vector.

21. An apparatus for replicating a database, comprising:

a transaction change log for receiving and storing changes to a source database;

an Input/Output (I/O) interceptor that is configured to intercept a write operation at the transaction change log, wherein the I/O interceptor comprises at least one of an instance level storage manager, an operating system function, or a kernel level device driver;

means for generating a copy of the intercepted write operation;

means for generating a redo vector based on the intercepted write operation; and

means for posting a record to a destination database based on the redo vector, wherein the record is in a form of at least one of a Data Manipulation Language (DML) operation or a Data Definition Language (DDL) operation.