Approaches for the replication of write sets

• US 8,666,939 B2
• Filed: 06/28/2011
• Issued: 03/04/2014
• Est. Priority Date: 06/28/2010
• Status: Expired due to Fees

• Current Assignee: SanDisk Technologies LLC (Western Digital Corporation)
• Original Assignee: SanDisk Enterprise IP LLC (Western Digital Corporation)
• Inventors: O'Krafka, Brian W., Dinker, Darpan, Krishnan, Manavalan, George, Johann
• Primary Examiner(s): Ehichioya, Fred I
• Assistant Examiner(s): BOWEN, RICHARD L
• Application Number:
  US13/170,968
• Publication Number:
  US 20110320403A1
• Time in Patent Office: 980 Days
• Field of Search: None
• US Class Current: 707/615
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 16/275 Synchronous replication

Abstract

Approaches for replicating data in a distributed transactional system. At a first node of a cluster, a per-transaction write set that comprises a plurality of write operations that are performed against a first data store maintained by the first node is committed. The per-transaction write set is replicated from the first node to a second node of the cluster. At the second node, the plurality of write operations, specified by the per-transaction write set, may be performed in parallel against a second data store maintained by the second node. At the second node, two or more threads may perform a portion of the plurality of write operations against data blocks stored within an in-memory buffer.

Claims

1. A method for replicating data in a distributed transactional system, comprising:

at a first node of a cluster,

committing a plurality of transactions, including grouping all write operations performed by committed transactions into distinct per-transaction write sets, and committing a per-transaction write set for each committed transaction, each per-transaction write set comprising all write operations that are performed in an individual transaction against a first data store maintained by the first node,

wherein at least some of the committed per-transaction write sets each comprise a plurality of write operations;

replicating the plurality of per-transaction write sets from the first node to a second node of the cluster, the replicating including;

at the second node of the cluster,

replicating, in parallel threads, distinct sets of the plurality of per-transaction write sets against a second data store maintained by the second node, wherein the replicating in each parallel thread is on a per-transaction write set by per-transaction write set basis.

2. The method of claim 1, wherein replicating, in parallel threads, the distinct per-transaction write sets comprises:

two or more threads, executing at the second node, each performing write operations on said per-transaction write set by per-transaction write set basis, to accomplish replication of respective per-transaction write sets of the plurality of per-transaction write sets against data blocks stored within an in-memory buffer in the second node.

3. The method of claim 1, including associating a global transaction commit number with each per-transaction write set;

wherein replicating, in parallel threads, the distinct per-transaction write sets comprises ensuring that any write operations, while replicating a respective per-transaction write set, that reference the same object are performed in order of the global transaction commit numbers associated with the per-transaction write sets that contain said write operations.

4. The method of claim 1, wherein replicating, in parallel threads, the distinct per-transaction write sets comprises ensuring, while replicating a respective per-transaction write set, that any write operations that reference a column of a parent table which is used as a foreign key in another child table, are performed at the second node in an order determined by write operations to the child table.

5. The method of claim 4, wherein foreign key constraints are enforced at the second node by serializing the processing of any per-transaction write sets that operate on tables having foreign key dependencies.

6. The method of claim 1, further comprising:

at the second node, pre-fetching one or more data blocks referenced by the distinct per-transaction write sets prior to replicating, in parallel threads, the distinct per-transaction write sets against the second data store.

7. The method of claim 6, further comprising:

at the second node, identifying the one or more data blocks to pre-fetch by scanning a set of unprocessed per-transaction write sets received by the second node to determine what data blocks are referenced by the set of unprocessed per-transaction write sets.

8. The method of claim 1, wherein replicating the distinct per-transaction write sets is performed asynchronously.

9. The method of claim 1, wherein replicating the distinct per-transaction write sets is performed semi-synchronously.

10. The method of claim 1, wherein the cluster is a first cluster, wherein replicating the plurality of per-transaction write sets from the first node to a second node is performed synchronously, and wherein the second node replicates changes to the second data store asynchronously to a third node in a different logical cluster than the first cluster.

11. The method of claim 1, wherein the first node and the second node are in different logical clusters.

12. The method of claim 1, wherein the first data store is persistently maintained on a first set of solid state devices, wherein the second data store is persistently maintained on a second set of solid state devices, and wherein the method further comprises:

at the second node, pre-fetching one or more data blocks referenced by the distinct per-transaction write sets prior to replicating, in parallel threads, the distinct per-transaction write sets of the plurality of per-transaction write sets, against the second data store.

13. The method of claim 1, wherein the first node waits a configurable period of time before replicating the plurality of per-transaction write sets to the second node.

14. The method of claim 1, further comprising:

at the second node, after replicating, in parallel threads, the distinct per-transaction write sets, against the second data store, committing the distinct per-transaction write sets using a global commit number received from a global-commit-number daemon, wherein the global commit number is one of a block of global commit numbers that the global-commit-number daemon assigns to the second node at once.

15. The method of claim 1, wherein replicating the plurality of per-transaction write sets from the first node to a second node is performed using a network packet based stream.

16. The method of claim 1, wherein replicating the plurality of per-transaction write sets from the first node to a second node comprises the first node sending one or more files to the second node.

17. The method of claim 1, wherein replicating the plurality of per-transaction write sets from the first node to a second node is performed in response to the first node determining, without human input, that the first data store is running out of space at the first node and a portion of the first data store should be migrated to the second node.

18. The method of claim 17, wherein the first node established the second data store at the second node in response to the first node determining that the first data store is running out of space at the first node and a portion of the first data store should be migrated to the second node.

19. A non-transitory machine readable storage medium storing one or more sequences of instructions for replicating data in a distributed transactional system comprising machines at a first node and a second node of a cluster, wherein execution of the one or more sequences of instructions causes the machines of the distributed transaction system to perform operations including:

at the first node of the cluster,

committing a plurality of transactions, including grouping all write operations performed by committed transactions into distinct per-transaction write sets, and committing a per-transaction write set for each committed transaction, each per-transaction write set comprising all write operations that are performed in an individual transaction against a first data store maintained by the first node,

wherein at least some of the committed per-transaction write sets each comprise a plurality of write operations;

replicating the plurality of per-transaction write sets from the first node to the second node of the cluster, the replicating including;

at the second node of the cluster,

replicating, in parallel threads, distinct sets of the plurality of per-transaction write sets against a second data store maintained by the second node, wherein the replicating in each parallel thread is on a per-transaction write set by per-transaction write set basis.

20. The non-transitory machine readable storage medium of claim 19, wherein replicating, in parallel threads, the distinct per-transaction write sets comprises:

two or more threads, executing at the second node, each performing write operations on said per-transaction write set by per-transaction write set basis, to accomplish, replication of respective per-transaction write sets of the plurality of per-transaction write sets against data blocks stored within an in-memory buffer.

21. A distributed transactional system, comprising:

machines at a first node and a second node of a cluster, each said machine comprising;

one or more processors; and

a machine readable storage medium storing one or more sequences of instructions;

wherein the machines of the distributed transaction system are configured to perform operations including;

at the first node of the cluster,

committing a plurality of transactions, including grouping all write operations performed by committed transactions into distinct per-transaction write sets, and committing a per-transaction write set for each committed transaction, each per-transaction write set comprising all write operations that are performed in an individual transaction against a first data store maintained by the first node,

wherein at least some of the committed per-transaction write sets each comprise a plurality of write operations;

replicating the plurality of per-transaction write sets from the first node to the second node of the cluster, the replicating including;

at the second node of the cluster,

replicating, in parallel threads, distinct sets of the plurality of per-transaction write sets against a second data store maintained by the second node, wherein the replicating in each parallel thread is on a per-transaction write set by per-transaction write set basis.

22. The distributed transactional system of claim 21, wherein replicating, in parallel threads, the distinct per-transaction write sets comprises:

two or more threads, executing at the second node, each performing write operations on said per-transaction write set by per-transaction write set basis, to accomplish, replication of respective per-transaction write sets of the plurality of per-transaction write sets against data blocks stored within an in-memory buffer in the second node.