Combining capture and apply in a distributed information sharing system

• US 8,799,213 B2
• Filed: 07/31/2007
• Issued: 08/05/2014
• Est. Priority Date: 07/31/2007
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Wong, Lik, Arora, Nimar S., Lakshminath, Anand, Wu, Jingwei, Gao, Lei, Hoang, Thuvan
• Primary Examiner(s): Perveen, Rehana
• Assistant Examiner(s): Wong, Huen
• Application Number:
  US11/831,478
• Publication Number:
  US 20090037422A1
• Time in Patent Office: 2,562 Days
• Field of Search: 707/610
• US Class Current: 707/610
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Abstract

In accordance with an embodiment of the present invention, an information sharing mechanism comprising a memory structure may be provided in a database system. In an embodiment, mined information transferred by a capture process to an associated apply process can be written into the memory structure without taking any latch. Similarly, the mined information can be read by the apply process from the memory structure without taking any latch. The capture and apply processes may work cooperatively to establish a safe point in log mining under various circumstances such as in an initial startup state, in a steady state, in a process restart scenario in the middle of information sharing. In some embodiments, the information sharing mechanism supports both checkpoint-free and checkpoint modes of log mining by the capture process. In addition, both the capture process and the apply process may employ an eager apply approach to increase processing capacity.

Claims

1. A method, comprising:

a capture process on a source node creating a stream of messages including a plurality of logical change records, of a source database, mined from transaction logs at said source node;

a destination node setting up a memory structure on said destination node, wherein said capture process on said source node adds all of said plurality of logical change records into said memory structure by way of a single capture proxy process on said destination node, and wherein a single apply process on said destination node receives said all of said plurality of logical change records by way of said memory structure;

said capture process sending said plurality of logical change records in said stream of messages to said single capture proxy process on said destination node;

said single capture proxy process receiving said plurality of logical change records in said stream of messages from said capture process;

upon receiving said plurality of logical change records, said single capture proxy process, without taking a latch for adding said plurality of logical change records to said memory structure, adding said plurality of logical change records in said stream of messages from said capture process to said memory structure on said destination node;

while said single capture proxy process is concurrently adding more logical change records to said memory structure, said single apply process, without taking a latch for changing said plurality of logical change records from said memory structure, retrieving said plurality of logical change records from said memory structure and marking said plurality of logical change records;

wherein the method is performed by one or more computing devices.

2. The method of claim 1, wherein the source node is a local database system in a distributed database system and wherein the destination node is another local database system in the distributed database system.

3. The method of claim 1, wherein the memory structure is a circular buffer that comprises multiple entries.

4. The method of claim 3, wherein the capture process is associated with the single apply process through the single capture proxy process located at the destination node.

5. The method of claim 4, wherein the single capture proxy process maintains a tail pointer to the circular buffer.

6. The method of claim 4, wherein the single apply process maintains a head pointer to the circular buffer.

7. The method of claim 1, wherein a message in said stream of messages contains a redo record that has a system change number.

8. The method of claim 1, further comprising:

identifying a transaction of which a message in said stream of messages is a part;

determining an apply processor to process the transaction; and

forwarding the message to the apply processor.

9. The method of claim 1, further comprising:

determining whether the memory structure has memory space to store a message; and

in response to determining that the memory structure does not have memory space to store the message, writing the message in a secondary memory structure.

10. The method of claim 1, further comprising:

the single apply process sending a request to the capture process to start mining logs at a particular logical time, wherein the particular logical time corresponds to a specific system change number that was assigned to a particular transaction record of a particular transaction, wherein the particular transaction record is the earliest record received by the single apply process of the particular transaction, and wherein the specific system change number is indicated in the request; and

the single apply process receiving one or more redo records that begins at the start transaction record of the particular transaction from the capture process.

11. The method of claim 1, wherein the capture process avoids taking checkpoints when capturing the interesting logical change records from one or more logs at the source node.

12. The method of claim 1, further comprising:

reestablishing an association between the single apply process and the capture process after the single apply process restarts at the destination node;

sending, by the single apply process, a particular system change number to the capture process, wherein the single apply process retrieves the particular system change number from a persistent data store and wherein the particular system change number indicates the lowest value for all subsequent system change numbers; and

receiving, from the capture process, one or more messages that contain one or more interesting logical change records captured at the source node, wherein each of the one or more interesting logical change records carries a system change number that is no less than the particular system change number.

13. The method of claim 1, further comprising:

reestablishing an association between the single apply process and the capture process after the capture process restarts at the source node;

sending a particular system change number to the capture process, wherein the single apply process stores the particular system change number as the lowest value for all subsequent system change numbers; and

receiving, from the capture process, one or more messages that contain one or more interesting logical change records captured at the source node, wherein each of the one or more interesting logical change records carries a system change number that is no less than the particular system change number.

14. The method of claim 1, wherein the step of marking said plurality of logical change records is performed before a transaction that is associated with a message in said stream of messages is consumed at the destination node.

15. A volatile or non-volatile computer-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform:

a capture process on a source node creating a stream of messages including a plurality of change records, of a source database, mined from transaction logs at said source node;

a destination node setting up a memory structure on said destination node, wherein said capture process on said source node adds all of said plurality of logical change records into said memory structure by way of a single capture proxy process on said destination node, and wherein a single apply process on said destination node receives said all of said plurality of logical change records by way of said memory structure;

said capture process sending said plurality of logical change records in said stream of messages to said single capture proxy process on said destination node;

said single capture proxy process receiving said plurality of logical change records in said stream of messages from said capture process;

upon receiving said plurality of logical change records, said single capture proxy process, without taking a latch for adding said plurality of logical change records to said memory structure, adding said plurality of logical change records in said stream of messages from said capture process to said memory structure on said destination node;

while said single capture proxy process is concurrently adding more logical change records to said memory structure, said single apply process, without taking a latch for changing said plurality of logical change records from said memory structure, retrieving said plurality of logical change records from said memory structure and marking said plurality of logical change records.

16. The medium of claim 15, wherein the source node is a local database system in a distributed database system and wherein the destination node is another local database system in the distributed database system.

17. The medium of claim 15, wherein the memory structure is a circular buffer that comprises multiple entries.

18. The medium of claim 17, wherein the capture process is associated with the single apply process through the single capture proxy process located at the destination node.

19. The medium of claim 18, wherein the single capture proxy process maintains a tail pointer to the circular buffer.

20. The medium of claim 18, wherein the single apply process maintains a head pointer to the circular buffer.

21. The medium of claim 15, wherein a message in said stream of messages contains a redo record that has a system change number.

22. The medium of claim 15, wherein the one or more sequences of instructions further comprise instructions which, when executed by one or more processors, causes the one or more processors to perform:

identifying a transaction of which a message in said stream of messages is a part;

determining an apply processor to process the transaction; and

forwarding the message to the apply processor.

23. The medium of claim 15, wherein the one or more sequences of instructions further comprise instructions which, when executed by one or more processors, causes the one or more processors to perform:

determining whether the memory structure has memory space to store a message; and

in response to determining that the memory structure does not have memory space to store the message, writing the message in a secondary memory structure.

24. The medium of claim 15, wherein the one or more sequences of instructions further comprise instructions which, when executed by one or more processors, causes the one or more processors to perform:

the single apply process sending a request to the capture process to start mining logs at a particular logical time, wherein the particular logical time corresponds to a specific system change number that was assigned to a particular transaction record of a particular transaction, wherein the particular transaction record is the earliest record received by the single apply process of the particular transaction, and wherein the specific system change number is indicated in the request; and

the single apply process receiving one or more redo records that begins at the start transaction record of the particular transaction from the capture process.

25. The medium of claim 15, wherein the capture process avoids taking checkpoints when capturing the interesting logical change records from one or more logs at the source node.

26. The medium of claim 15, wherein the one or more sequences of instructions further comprise instructions which, when executed by one or more processors, causes the one or more processors to perform:

reestablishing an association between the single apply process and the capture process after the single apply process restarts at the destination node;

sending, by the single apply process, a particular system change number to the capture process, wherein the single apply process retrieves the particular system change number from a persistent data store and wherein the particular system change number indicates the lowest value for all subsequent system change numbers; and

receiving, from the capture process, one or more messages that contain one or more interesting logical change records captured at the source node, wherein each of the one or more interesting logical change records carries a system change number that is no less than the particular system change number.

27. The medium of claim 15, wherein the one or more sequences of instructions further comprise instructions which, when executed by one or more processors, causes the one or more processors to perform:

reestablishing an association between the single apply process and the capture process after the capture process restarts at the source node;

sending a particular system change number to the capture process, wherein the single apply process stores the particular system change number as the lowest value for all subsequent system change numbers; and

receiving, from the capture process, one or more messages that contain one or more interesting logical change records captured at the source node, wherein each of the one or more interesting logical change records carries a system change number that is no less than the particular system change number.

28. The medium of claim 15, wherein the step of marking said plurality of logical change records is performed before a transaction that is associated with a message in said stream of messages is consumed at the destination node.