Data replication protocol with steady state data distribution and quorum formation

• US 9,081,843 B2
• Filed: 03/15/2013
• Issued: 07/14/2015
• Est. Priority Date: 11/06/2012
• Status: Active Grant

• Current Assignee: Cloud Software Group
• Original Assignee: TIBCO Software Incorporated (Cloud Software Group)
• Inventors: Leshchiner, Dan
• Primary Examiner(s): PENG, HUAWEN A
• Application Number:
  US13/842,443
• Publication Number:
  US 20140129519A1
• Time in Patent Office: 851 Days
• Field of Search: 707/613
• US Class Current: 1/1
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation

Abstract

Steady state data distribution is provided between a client application, a leader machine, and a plurality of replica machines. The distribution comprises the leader machine receiving an operation request from the client application, the leader machine sending the prepare message to each of the plurality of replica machines, the replica machines recording in their logs information on the operation, the replica machines sending acknowledgement messages to the leader machine, and the leader machine sending commit command messages to the replica machines. A new quorum of the replica machines is created by using log information. Replica machines that become part of the new quorum are updated in an efficient manner.

Claims

1. A method of steady state data distribution between a client application, a leader machine, and a plurality of replica machines, the method comprising:

receiving by the leader machine an operation request message specifying an operation and associated data from the client application;

distributing by the leader machine a prepare message to each of the plurality of replica machines collectively called a quorum, wherein the prepare message comprises the operation, the associated data, a quorum version number identifying the quorum, and an operation version number identifying the operation;

recording in a log by each of the replica machines the operation, the associated data, the quorum version number and the operation version number from the prepare message;

sending by each of the replica machines a first acknowledgement message to the leader machine upon receiving the prepare message, wherein the first acknowledgement message comprises the quorum version number and the operation version number;

sending by the leader machine a commit command message to the replica machines upon receiving the first acknowledgement messages from a majority of the replica machines in the quorum;

executing by the replica machines the operation specified in the prepare message upon receiving the commit command message from the leader machine; and

removing by each of the replica machines from its log the operation and the associated data provided in the prepare message upon executing the operation.

2. The method of claim 1 further comprising:

sending by the leader machine a second acknowledgement message to the client application upon receiving the first acknowledgement messages from the majority of the replica machines in the quorum.

3. The method of claim 1, wherein the leader machine is initially selected from the quorum with an election protocol.

4. The method of claim 1, wherein a positive number is assigned to an initial operation version number in an initial operation request message sent to a quorum.

5. The method of claim 4, wherein operation version numbers of subsequent operation request messages sent to the quorum are monotonically increased.

6. The method of claim 5, wherein a positive number is assigned to an initial quorum version number for the quorum.

7. The method of claim 6, wherein the quorum version number is monotonically increased when a new quorum is formed.

8. The method of claim 5, wherein the leader machine and the replica machines execute operation request messages with lower operation version numbers prior to executing operation request messages with higher operation version numbers.

9. A method of forming a quorum of replica machines connected to a leader machine that is in communication with a client application, the method comprising:

sending by the leader machine a solicit message to each of the replica machines;

sending by each of the replica machines a reply message comprising a log to the leader machine upon receiving the solicit message, the log comprising a quorum commit version number identifying the quorum each of the replica machines has committed to;

identifying by the leader machine a most recent log from the received logs, the most recent log identified by identifying a log with the highest quorum commit version number from the received logs;

forming a new quorum with the replica machines based on the most recent log; and

updating the replica machines in the quorum to have the most recent log.

10. The method of claim 9, wherein the log comprises an operation version number identifying an operation each of the replica machines has committed to.

11. The method of claim 10, wherein the log further comprises a quorum asked version number identifying the quorum each of the replica machines has been asked to prepare for or commit to.

12. The method of claim 10, wherein a positive number is assigned to an initial operation version number in an initial operation request message sent to a quorum.

13. The method of claim 12, wherein the operation version number is monotonically increased for each new operation, the quorum commit version number is monotonically increased for each new quorum the replica machine has committed to, and the quorum version number is monotonically increased for each new quorum the replica machine has been asked to commit to.

14. The method of claim 10, wherein, if there is only one replica machine with a log that has the highest quorum commit version number, then the log with the highest quorum commit version number is identified as the most recent log, and further wherein,

if there are two or more replica machines with logs that have the highest quorum commit version number, then the log with the highest operation version number is identified as the most recent log.

15. The method of claim 14, wherein, if the replica machines that sent the reply messages and have the most recent log constitute a majority of all the replica machines, then the new quorum is formed with the replica machines that have the most recent log, further wherein,

if the replica machines that sent the reply messages and have the most recent log do not constitute the majority of all the replica machines, but when combined with the replica messages that sent the reply messages and have logs that are less recent than the most recent log, constitute the majority of all the replica machines, then the new quorum is formed with the replica machines that have the most recent log and with the replica machines that have the logs that are less recent than the most recent log, and further wherein,

if the replica machines that sent the reply messages and have the most recent log combined with the replica machines that sent the reply message and have logs that are less recent than the most recent log do not constitute the majority of all the replica machines, but when combined with the replica machines that sent the reply messages and have empty logs, constitute all the replica machines, then the new quorum is formed with the replica machines that have the most recent log, the replica machines that have the logs that are less recent than the most recent logs, and the replica machines that have the empty logs.

16. The method of claim 15, wherein the empty logs are logs with a highest quorum commit version number that is zero or logs with a highest operation version number that is zero.

17. The method of claim 16, wherein the new quorum is formed at least with a minimum number of the replica machines to constitute a majority of all the replica machines.

18. The method of claim 17, wherein the updating of the replica machines in the new quorum to have the most recent log comprises synchronizing uncommitted operations and committed data values of the most recent log.

19. The method of claim 18, wherein, if the most recent log and the logs that are less recent than the most recent log comprise uncommitted operations from different quorums identified with different quorum commit version numbers, then the uncommitted operations are removed from the logs that are less recent than the most recent log, and further wherein,

if the most recent log and the logs that are less recent than the most recent log comprise uncommitted operations from a same quorum identified with the same quorum commit version number, then the uncommitted operations are removed from the logs that are less recent than the most recent log and uncommitted operations that are present in the most recent log but absent in the logs that are less recent than the most recent log are copied into the logs that are less recent than the most recent log.

20. The method of claim 9 further comprising sending by the leader machine a start message to the new quorum comprising replica machines, wherein the start message instructs the quorum'"'"'s replica machines to return to a steady state data replication protocol by processing remaining uncommitted operations in their logs.

21. The method of claim 9, wherein the identifying by the leader machine the most recent log is performed upon receiving reply messages from a majority of the replica machines.

22. The method of claim 9, wherein the identifying by the leader machine the most recent log is performed when a timer instructs it to.

23. The method of claim 9, wherein forming the new quorum further comprises sending by the leader machine prepare messages to the replica machines to be included in the new quorum, and sending by the replica machines acknowledge messages to the leader machine to acknowledge receipt of the prepare messages.

24. The method of claim 23 further comprising sending by the leader machine commit messages to the replica machines to be included in the new quorum, and sending by the replica machines the acknowledge messages to the leader machine to acknowledge receipt of the commit messages.