Database Log Capture that Publishes Transactions to Multiple Targets to Handle Unavailable Targets by Separating the Publishing of Subscriptions and Subsequently Recombining the Publishing

• US 20090112947A1
• Filed: 12/24/2008
• Published: 04/30/2009
• Est. Priority Date: 08/27/2003
• Status: Active Grant

• Current Assignee: Domo, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Bourbonnais, Serge, Lindsay, Bruce Gilbert, Li, Siqun
• Granted Patent: US 7,991,745 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/2056 by mirroring
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 2201/80 Database-specific techniques
• CPC Class Codes: G06Q 40/12 Accounting
• CPC Class Codes: Y10S 707/99938 Concurrency, e.g. lock management in shared database
• CPC Class Codes: Y10S 707/99953 Recoverability

Abstract

A first log reader publishes first messages to a plurality of queues. The messages comprise changes for transactions extracted from a log by the first log reader. In response to one of the queues becoming unavailable, a second log reader is launched to read and extract from the log, and to publish second messages comprising changes for transactions extracted from the log by the second log reader to the unavailable queue as a catch-up queue. In response to the catch-up queue becoming available and the second log reader reaching the end of the log, the publishing of the second messages for the catch-up queue is transferred from the second log reader to the first log reader.

Claims

1. A computer program product for log capture comprising:

a computer readable medium;

first program instructions to publish, by a first log reader, first messages to a plurality of queues, the first messages comprising changes for transactions extracted from a log by the first log reader;

second program instructions to, in response to one of the queues becoming unavailable, launch a second log reader to read and extract from the log, and to publish second messages comprising changes for transactions extracted from the log by the second log reader to the unavailable queue as a catch-up queue; and

third program instructions to, in response to the catch-up queue becoming available and the second log reader reaches the end of the log, transfer the publishing of the second messages for the catch-up queue from the second log reader to the first log reader,

wherein the third program instructions re-synchronize the second log reader and the first log reader,

the third program instructions to;

send synchronization messages between the first log reader and the second log reader;

a first synchronization message of the synchronization messages being sent from the second log reader to the first log reader with second log reader restart information; and

in response to the second log reader being behind the first log reader in reading the log based on the second log reader restart information, a second synchronization message of the synchronization messages being sent from the first log reader to the second log reader, the second synchronization message comprising first log reader restart information having a first-last-queue-commit point timestamp, and suspend the publishing by the first log reader until the second log reader reaches the first-last-queue-commit point timestamp or the end of the log;

wherein the first, second and third program instructions are stored on the computer readable medium.

2. The computer program product of claim 1 wherein the first log reader launches the second log reader.

3. The computer program product of claim 1 further comprising:

fourth program instructions to terminate the second log reader;

wherein the fourth program instructions are stored on the computer readable medium.

4. The computer program product of claim 1 wherein the first log reader and the second log reader execute separately.

5. The computer program product of claim 1 further comprising:

fourth program instructions to store the first log reader restart information, associated with the first log reader, in persistent memory;

fifth program instructions to store the second log reader restart information, associated with the second log reader, in the persistent memory;

sixth program instructions to stop the first log reader and the second log reader;

seventh program instructions to first launch the first log reader based on the first log reader restart information; and

eighth program instructions to second launch the second log reader based on the second log reader restart information;

wherein the fourth, fifth, sixth, seventh and eighth program instructions are stored on the computer readable medium.

6. A computer program product for log capture comprising:

a computer readable medium;

first program instructions to publish, by a first log reader, first messages to a plurality of queues, the first messages comprising changes for transactions extracted from a log by the first log reader;

second program instructions to, in response to one of the queues becoming unavailable, launch a second log reader to read and extract from the log, and to publish second messages comprising changes for transactions extracted from the log by the second log reader to the unavailable queue as a catch-up queue; and

third program instructions to, in response to the catch-up queue becoming available and the second log reader reaches the end of the log, transfer the publishing of the second messages for the catch-up queue from the second log reader to the first log reader,

wherein the third program instructions to transfer re-synchronizes the second log reader and the first log reader, the third program instructions to;

send synchronization messages between the first log reader and the second log reader;

a first synchronization message of the synchronization messages being sent from the second log reader to the first log reader with second log reader restart information;

in response to the second log reader being ahead of the first log reader in reading the log based on the second log reader restart information, a stop message of the synchronization messages being sent to the second log reader; and

in response to the second log reader being behind the first log reader in reading the log based on the second log reader restart information, a second synchronization message of the synchronization messages being sent from the first log reader to the second log reader, the second synchronization message comprising first log reader restart information having a first-last-queue-commit point timestamp, and suspend the publishing by the first log reader until the second log reader reaches the first-last-queue-commit point timestamp or the end of the log;

wherein the first, second and third program instructions are stored on the computer readable medium.

7. The computer program product of claim 6 wherein the first log reader launches the second log reader.

8. The computer program product of claim 6 further comprising:

fourth program instructions to terminate the second log reader;

wherein the fourth program instructions are stored on the computer readable medium.

9. The computer program product of claim 6 wherein the first log reader and the second log reader execute separately.

10. The computer program product of claim 6 further comprising:

fourth program instructions to store the first log reader restart information, associated with the first log reader, in persistent memory;

fifth program instructions to store the second log reader restart information, associated with the second log reader, in the persistent memory;

sixth program instructions to stop the first log reader and the second log reader, seventh program instructions to first launch the first log reader based on the first log reader restart information; and

eighth program instructions to second launch the second log reader based on the second log reader restart information;

wherein the fourth, fifth, sixth, seventh and eighth program instructions are stored on the computer readable medium.

11. A computer system comprising:

a processor; and

a memory comprising instructions, executable by the processor, for;

publishing, by a first log reader, first messages to a plurality of queues, the first messages comprising changes for transactions extracted from a log by the first log reader;

in response to one of the queues becoming unavailable, launching a second log reader to read and extract from the log, and to publish second messages comprising changes for transactions extracted from the log by the second log reader to the unavailable queue as a catch-up queue; and

in response to the catch-up queue becoming available and the second log reader reaches the end of the log, transferring the publishing of the second messages for the catch-up queue from the second log reader to the first log reader,

wherein said transferring comprises re-synchronizing the second log reader and the first log reader,

wherein said re-synchronizing comprises;

sending synchronization messages between the first log reader and the second log reader;

a first synchronization message of the synchronization messages being sent from the second log reader to the first log reader with second log reader restart information; and

in response to the second log reader being behind the first log reader in reading the log based on the second log reader restart information, a second synchronization message of the synchronization messages being sent from the first log reader to the second log reader, the second synchronization message comprising first log reader restart information having a first-last-queue-commit point timestamp, and suspending the publishing by the first log reader until the second log reader reaches the first-last-queue-commit point timestamp or the end of the log.

12. The computer system of claim 11 wherein the first log reader launches the second log reader.

13. The computer system of claim 11 further comprising instructions for:

terminating the second log reader.

14. The computer system of claim 11 wherein the first log reader and the second log reader execute separately.

15. The computer system of claim 11 further comprising instructions for:

storing the first log reader restart information, associated with the first log reader, in persistent memory;

storing the second log reader restart information, associated with the second log reader, in the persistent memory;

stopping the first log reader and the second log reader,

first launching the first log reader based on the first log reader restart information; and

second launching the second log reader based on the second log reader restart information.

16. A computer system comprising:

a processor; and

a memory storing instructions, executable by the processor, for;

publishing, by a first log reader, first messages to a plurality of queues, the first messages comprising changes for transactions extracted from a log by the first log reader;

in response to one of the queues becoming unavailable, launching a second log reader to read and extract from the log, and to publish second messages comprising changes for transactions extracted from the log by the second log reader to the unavailable queue as a catch-up queue; and

in response to the catch-up queue becoming available and the second log reader reaches the end of the log, transferring the publishing of the second messages for the catch-up queue from the second log reader to the first log reader,

wherein said transferring comprises re-synchronizing the second log reader and the first log reader,

wherein said re-synchronizing comprises;

sending synchronization messages between the first log reader and the second log reader;

a first synchronization message of the synchronization messages being sent from the second log reader to the first log reader with second log reader restart information; and

in response to the second log reader being ahead of the first log reader in reading the log based on the second log reader restart information, a stop message of the synchronization messages being sent to the second log reader;

in response to the second log reader being behind the first log reader in reading the log based on the second log reader restart information, a second synchronization message of the synchronization messages being sent from the first log reader to the second log reader, the second synchronization message comprising first log reader restart information having a first-last-queue-commit point timestamp, and suspending the publishing by the first log reader until the second log reader reaches the first-last-queue-commit point timestamp or the end of the log.

17. The computer system of claim 16 wherein the first log reader launches the second log reader.

18. The computer system of claim 16 further comprising instructions for:

terminating the second log reader.

19. The computer system of claim 16 wherein the first log reader and the second log reader execute separately.

20. The computer system of claim 16 further comprising instructions for:

storing the first log reader restart information, associated with the first log reader, in persistent memory;

storing the second log reader restart information, associated with the second log reader, in the persistent memory;

stopping the first log reader and the second log reader,

first launching the first log reader based on the first log reader restart information; and

second launching the second log reader based on the second log reader restart information.