Method for streaming transactions in database cluster

• US 10,078,680 B2
• Filed: 12/17/2014
• Issued: 09/18/2018
• Est. Priority Date: 12/17/2014
• Status: Active Grant

• Current Assignee: Codership Oy
• Original Assignee: Codership Oy
• Inventors: Yurchenko, Alexey, Jaakola, Seppo, Ollakka, Teemu
• Primary Examiner(s): Truong, Dennis
• Application Number:
  US14/574,256
• Publication Number:
  US 20160179917A1
• Time in Patent Office: 1,371 Days
• Field of Search: 707713, 707770, 707613, 707674, 707718, 707769, 707623, 707966
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 9/466 Transaction processing

Abstract

A method for use in a database cluster is provided. A transaction is received at a first node of the database cluster. The transaction is executed at the first node, wherein the transaction produces a plurality of replication events. The plurality of replication events of the transaction are divided into a plurality of fragments, each of which comprises one or more replication events. Subsequently, a set of fragment-specific parameters is assigned to each of the plurality of fragments. The plurality of fragments are then communicated to a second node of the database cluster, one by one, as soon as they are prepared in the first node. A transaction is established in the second node to apply the arriving fragments. With this streaming replication method, the replication of the transaction in the first node, can begin early, before the transaction commits in the first node. This will even out the delay of large transaction replication.

Claims

1. A method for use in a database cluster, wherein the database cluster comprises a plurality of nodes, the method comprising:

receiving a transaction at a first node to be committed at the first node of the database cluster;

executing the transaction at the first node, wherein the transaction produces a plurality of replication events;

beginning a replication of the transaction at the first node before committing the transaction in the first node by;

dividing the plurality of replication events of the transaction into a plurality of fragments, wherein each of the plurality of fragments comprises one or more replication events;

assigning a set of fragment-specific parameters to each of the plurality of fragments;

communicating the plurality of fragments to a second node of the database cluster one by one as they are prepared in the first node, wherein at least one of the plurality of fragments is communicated before the transaction is committed at the first node;

the method further comprising;

certifying each fragment of the plurality of fragments separately and individually; and

committing the transaction at both first and second nodes when each fragment of the plurality of fragments completes certification successfully.

2. The method of claim 1, wherein the plurality of replication events of the transaction are divided into the plurality of fragments based on at least one of:

a size of data that is replicated,

a number of replication events,

a number of database rows that are modified,

a number of query statements that are processed.

3. The method of claim 1, wherein the dividing the plurality of replication events of the transaction is triggered by an application.

4. The method of claim 1, further comprising rolling back the transaction, when one or more of the plurality of fragments fail certification.

5. The method of claim 1, wherein the set of fragment-specific parameters comprises a transaction identifier.

6. The method of claim 1, wherein the communicating the plurality of fragments comprises streaming a first fragment before a second fragment, the first fragment being created before the second fragment at the first node.

7. A method for use in a database cluster, wherein the database cluster comprises a plurality of nodes, the method comprising:

beginning reception of a plurality of fragments at a second node of the database cluster, one by one as they are replicated from a transaction to be committed at another node of the database cluster, wherein at least one of the plurality of fragments is received before the transaction is committed at the other node, each of the fragments comprising one or more replication events;

the method further comprising, upon receipt of each received fragment;

certifying each received fragment separately and individually;

identifying a set of fragment-specific parameters associated with the received fragment;

determining, from the set of fragment-specific parameters, whether or not a streaming transaction to which the received fragment belongs exists in a transaction pool; and

applying the one or more replication events of the received fragment at the second node whilst addressing the streaming transaction to which the fragment belongs;

the method still further comprising committing the transaction at the first and second node when each fragment of the plurality of received fragments completes certification successfully, and a last one of the plurality of received fragments includes a fragment-specific parameter comprising a commit flag indicating that all of the plurality of received fragments should be committed.

8. The method of claim 7 further comprising starting a new streaming transaction in the transaction pool, when a streaming transaction to which the received fragment belongs does not exist in the transaction pool.

9. The method of claim 8 further comprising associating the new streaming transaction with a transaction identifier included in the set of fragment-specific parameters, wherein the transaction identifier uniquely identifies the transaction with which the received fragment is associated.

10. The method of claim 7, wherein the set of fragment-specific parameters comprises a transaction identifier, and wherein the method further comprises determining from the status of the commit flag whether or not the received fragment is a last fragment of the transaction.

11. The method of claim 7, wherein the set of fragment-specific parameters comprises a begin flag, and wherein the method further comprises determining from a status of the begin flag whether or not the received fragment is a first fragment of the transaction.

12. A computing apparatus comprising:

a processor;

a memory coupled to the processor; and

a network interface coupled to the processor,

wherein the processor is configured to;

receive a transaction to be committed at a first node of a database cluster;

execute the transaction at the first node, wherein the transaction produces a plurality of replication events; and

begin a replication of the transaction at the first node before committing the transaction in the first node by;

dividing the plurality of replication events of the transaction into a plurality of fragments, wherein each of the plurality of fragments comprises one or more replication events;

assigning a set of fragment-specific parameters to each of the plurality of fragments; and

communicating the plurality of fragments to one or more second nodes of a database cluster one by one as they are prepared when dividing the plurality of replication events of the transaction, wherein at least one of the plurality of fragments is communicated before the transaction is committed at the first node;

wherein the processor is further configured to;

certify each fragment of the plurality of fragments separately and individually; and

commit the transaction at both the first and second nodes when each fragment of the plurality of fragments completes certification successfully.

13. The apparatus of claim 12, wherein the processor is configured to divide the plurality of replication events of the transaction based on at least one of:

a size of data that is replicated,

a number of replication events,

a number of database rows that are modified,

a number of query statements that are processed.

14. The apparatus of claim 12, wherein the set of fragment-specific parameters comprises a transaction identifier.

15. The apparatus of claim 12, wherein the processor is configured to stream a first fragment before a second fragment when communicating the plurality of fragments, the first fragment being created before the second fragment.

16. A computing apparatus comprising:

a processor;

a memory coupled to the processor; and

a network interface coupled to the processor,

wherein the processor is configured to;

begin reception of a plurality of fragments one by one as they are replicated from a transaction to be committed at a node of a database cluster, wherein at least one of the plurality of fragments is received before the transaction is committed at the node, each of the fragments comprising one or more replication events;

wherein upon receipt of each fragment, the processor is configured to;

certify each received fragment separately and individually;

identify a set of fragment-specific parameters associated with the received fragment;

determine, from the set of fragment-specific parameters, whether or not a streaming transaction to which the received fragment belongs exists in a transaction pool; and

apply the one or more replication events of the received fragment whilst addressing the streaming transaction to which the fragment belongs;

wherein the processor is further configured to commit the transaction at the apparatus and cause the transaction to be committed at the node when each fragment of the plurality of received fragments completes certification successfully.

17. The apparatus of claim 16, wherein the processor is configured to start a new streaming transaction in the transaction pool, when a streaming transaction to which the received fragment belongs does not exist in the transaction pool.

18. The apparatus of claim 17, wherein the processor is configured to associate the new streaming transaction with a transaction identifier included in the set of fragment-specific parameters, wherein the transaction identifier uniquely identifies the transaction with which the received fragment is associated.

19. The apparatus of claim 16, wherein the set of fragment-specific parameters comprises a transaction identifier, and wherein the processor is configured to determine from the status of the commit flag whether or not the received fragment is a last fragment of the transaction.

20. The apparatus of claim 16, wherein the set of fragment-specific parameters comprises a begin flag, and wherein the processor is configured to determine from a status of the begin flag whether or not the received fragment is a first fragment of the transaction.

21. A computer program product comprising a non-transitory machine-readable data storage medium having stored thereon program instructions that, when accessed by a processing device, cause the processing device to:

receive a transaction to be committed at a first node of a database cluster;

execute the transaction at the first node, wherein the transaction produces a plurality of replication events;

begin a replication of the transaction at the first node before committing the transaction in the first node by;

dividing the plurality of replication events of the transaction into a plurality of fragments, wherein each of the plurality of fragments comprises one or more replication events;

assigning a set of fragment-specific parameters to each of the plurality of fragments; and

communicating the plurality of fragments to each of one or more second nodes of a database cluster one by one as they are prepared in the first node, wherein at least one of the plurality of fragments is communicated before the transaction is committed at the first node;

wherein the computer program product further causes the processing device to;

certify each fragment of the plurality of fragments separately and individually; and

commit the transaction at the first node and at each of the one or more second nodes of the database cluster when each fragment of the plurality of fragments completes certification successfully and a last one of the plurality of fragments has been assigned a fragment-specific parameter comprising a commit flag indicating that all of the plurality of fragments should be committed.

22. The computer program product of claim 21, wherein the plurality of replication events of the transaction are divided into the plurality of fragments based on at least one of:

a size of data that is replicated,

a number of replication events,

a number of database rows that are modified,

a number of query statements that are processed.