Data replication protocol with efficient update of replica machines

• US 9,418,130 B2
• Filed: 03/15/2013
• Issued: 08/16/2016
• Est. Priority Date: 11/06/2012
• Status: Active Grant

• Current Assignee: Cloud Software Group
• Original Assignee: TIBCO Software Incorporated (Cloud Software Group)
• Inventors: Leshchiner, Dan, Hanel, Matthias M.
• Primary Examiner(s): PENG, HUAWEN A
• Application Number:
  US13/842,389
• Publication Number:
  US 20140143205A1
• Time in Patent Office: 1,250 Days
• Field of Search: 707/626
• US Class Current: 1/1
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Abstract

Steady state data distribution is provided between a client application, a leader machine, and a plurality of replica machines. The distribution comprises the leader machine receiving an operation request from the client application, the leader machine sending the prepare message to each of the plurality of replica machines, the replica machines recording in their logs information on the operation, the replica machines sending acknowledgement messages to the leader machine, and the leader machine sending commit command messages to the replica machines. A new quorum of the replica machines is created by using log information. Replica machines that become part of the new quorum are updated in an efficient manner.

Claims

1. A method of updating replica machines that are connected to a leader machine, the method comprising:

maintaining, by each of the replica machines, an operation log, each log entry in each operation log including an operation log number, a key, and an operation performed on the key, the operation log sorted by operation log number;

assigning, by the leader machine, a monotonically increasing operation log number to an operation command;

receiving by the replica machines the operation command and operation log number assigned to the operation command from the leader machine to operate on the operation logs of the replica machines;

operating by the replica machines on the operation command;

inserting, into the operation log of each of the replica machines, a new log entry, the new log entry including the operation log number assigned to the operation command, a key specified in the operation command, and an operation to perform on the key specified in the operation command;

updating a secondary index of each of the replica machines, the secondary index sorted by keys; and

updating an operation log of a first replica machine based on an operation log of the leader machine by;

iteratively inserting, into the operation log of the first replica machine, those log entries in the tail end of the operation log of the leader machine that are missing in the operation log of the first replica machine; and

for each log entry being iteratively inserted into the operation log of the first replica machine, performing by the first replica machine an operation included in the said log entry being iteratively inserted into the operation log of the first replica machine.

2. The method of claim 1, wherein each of the operation commands comprises an operation type, key, and a value, further wherein the operation type is an insert, delete, or update.

3. The method of claim 2, wherein a positive number is assigned as the operation log number of an initial operation command.

4. The method of claim 3, wherein the operation log number is monotonically increased for each subsequent operation command.

5. The method of claim 4, wherein the operation commands with the operation type insert are recorded at an end of the operation log, with the operation type update are recorded at the end of the operation log, and with the operation type delete are recorded at the end of the log.

6. The method of claim 5, wherein the operation command with the operation type update can only replace the value of the record with the same key.

7. The method of claim 6, wherein the operation commands with the operation type update or delete can only be recorded in the operation log after the operation command with the same key with the operation type insert has been recorded in and subsequently removed from the log.

8. The method of claim 7, wherein the records of the operation commands with the operation type delete get merged to a single record at the end of the operation log if the records are sequentially recorded.

9. The method of claim 8, wherein the records of the operation commands with the operation type delete get merged by referring to the secondary index, wherein the secondary index is updated by merging delete records.

10. The method of claim 1, wherein the operation log allows multiple secondary indexes sorted by key.

11. The method of claim 10, wherein secondary indexes may themselves be assigned ids and would be managed by another secondary index that tracks those ids.

12. The method of claim 11, wherein each record in the secondary index sorted by the keys does not have a one-to-one connection with records in the operation logs.

13. The method of claim 12, wherein each key inserted may not be consecutive if unused, lower keys are deleted as well.

14. The method of claim 13, wherein delete operations for a given key may precede the insert operation for that same key.

15. The method of claim 1, wherein, keys may be inserted in an arbitrary order when distinguishing between a key that had never been inserted and a key that had been inserted and deleted is not required.

16. The method of claim 15, wherein, keys may be of any sortable type.

17. The method of claim 1, further comprising simultaneously receiving by the leader machine new operation commands from a client application.

18. A method of updating a plurality of replica machines connected to a leader machine, wherein each replica machine comprises its own tree structure having four levels, further wherein a first level comprises a root node, a second level comprises interior nodes stemming from the root node, wherein the interior nodes are themselves organized into a tree, a third level comprises leaf nodes stemming from the interior nodes, and a fourth level comprises data stores stemming from the leaf nodes, the method comprising:

sending by the leader machine operation commands to the replica machines, wherein each operation command comprises an operation, a key, a monotonically increasing operation log number assigned to the operation command by the leader machine, and data;

receiving, by each of the replica machines, the operation command; and

updating, by each of the replica machines, its own tree structure according to the received operation command;

updating a tree structure of a first replica machine based on a tree structure of the leader machine by;

decomposing, into sub-ranges, a range of key values stored in the tree structure of the first replica machine, each sub-range having its own maximum operation log number; and

for each sub-range, recursively comparing and updating a maximum operation log number in the sub-range with a maximum operation log number in a corresponding sub-range of key values stored in the tree structure of the leader machine;

wherein each replica machine constructs and maintains its own tree structure.

19. The method of claim 18, wherein the root node has at most a positive set number of nodes stemming from the root node, further wherein each of the interior nodes has a number of nodes stemming from the interior node between the positive number and half of the positive number, and further wherein each of the leaf nodes comprises a number of key values between the set positive number and half of the positive number.

20. The method of claim 19, wherein the set positive number may be an operating parameter that is configured by an administrator.

21. The method of claim 20, wherein the set positive number may be different for trees for each of the replica machines.

22. The method of claim 19, wherein each leaf node comprises pairs, wherein each node pair comprises a key and an operation version number, and further wherein each non-leaf node comprises pairs, wherein each non-leaf node pair comprises a largest key and a largest operation version number from nodes that stem from the non-leaf node.

23. The method of claim 22, wherein each leaf node pair represents a sequence of consecutive keys that ranges from a key that is immediately left of the node pair that the leaf node pair stems from through to the key.

24. The method of claim 18, wherein the operation is a delete, insert, or update.

25. The method of claim 24, wherein, if the operation specified in the operation command is the delete operation, the operation log number for the key specified in the operation command is negated.

26. The method of claim 25, wherein consecutive records of the operation commands with the operation type delete are merged.

27. The method of claim 18, wherein updating of the tree structure further comprises updating by the replica machines a list of unsynchronized key ranges and a list of synchronized key ranges.

28. The method of claim 27, wherein each key of the list of unsynchronized key ranges and each key of the list of synchronized key ranges form an ordered pair of keys.

29. The method of claim 28 further comprising decomposing the unsynchronized key ranges into sub-ranges, wherein each of the sub-ranges comprises a sub-key with its own maximum operation log number.

30. The method of claim 29 further comprising recursively updating the sub-ranges of a tree structure to provide updating with each of a same sub-range in the tree structure to be updated, wherein,

if the largest operation log number is the same in a tree structure to provide the updating and the tree structure to be updated, then the range is moved from the list of unsynchronized key ranges to the list of synchronized key ranges, further wherein,

if the largest operation log number is not the same in the tree structure to provide the updating and the tree structure to be updated, and the sub-ranges are from a leaf node of the tree structure of the leader machine, then the operation log number and the data for the key is copied from the tree structure to provide the updating to the tree to be updated, and the sub-range is merged into the list of synchronized key ranges, and further wherein,

if the largest operation log number is not the same in the tree structure to provide the updating and the tree structure to be updated, and the sub-ranges are not from a leaf node of the tree structure to provide the updating, then the sub-range is added to a temporary list, which is then substituted into the list of unsynchronized key ranges for the sub-range that was decomposed to produce this temporary list.

31. The method of claim 18, further comprising simultaneously receiving by the leader machine new operation commands from a client application.