Synchronizing cluster time to a master node with a faster clock

• US 7,814,360 B2
• Filed: 01/25/2007
• Issued: 10/12/2010
• Est. Priority Date: 01/25/2007
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Srivastava, Alok, Rai, Vikram
• Primary Examiner(s): Du; Thuan N
• Application Number:
  US11/698,605
• Publication Number:
  US 20080183894A1
• Time in Patent Office: 1,356 Days
• Field of Search: 713/400, 370/503
• US Class Current: 713/400
• CPC Class Codes: H04J 3/0641 Change of the master or reference, e.g. take-over or failure of the master
• CPC Class Codes: H04L 69/28 Timers or timing mechanisms used in protocols

Abstract

Synchronizing cluster time is described. Times of one or more nodes in the cluster to a master node of the cluster are synchronized. It is discovered that a node of a cluster has a clock time that is ahead of a master clock time of the master node. In response to discovering the node with the clock time ahead of the master clock time, the master node is replaced with the node ahead thereof in time, which thus becomes a new master node. The clock times of the cluster nodes are synchronized with the clock time of the new master node.

Claims

1. A method for synchronizing time in a computer cluster of nodes, comprising the steps of:

synchronizing times of one or more nodes in the cluster to a master node of the cluster;

discovering a node of the cluster has a clock time that is ahead of a master clock time of the master node;

in response to discovering the node with the clock time ahead of the master clock time, replacing the master node with the node ahead thereof in time as a new master node; and

synchronizing the clock times of the nodes in the cluster with the clock time of the new master node;

wherein the steps are performed by one or more nodes in the cluster.

2. The method as recited in claim 1 wherein the replacing step is performed atomically between the original master node and the node ahead thereof in time.

3. The method as recited in claim 1 wherein the replacing step is performed when one or more criteria are satisfied.

4. The method as recited in claim 3 wherein one of the criteria is based on how advanced the clock of the node ahead of the original master node is, in relation to the other nodes in said cluster.

5. The method as recited in claim 4 wherein one of the criteria is based on the mean time of the clocks of the nodes in the cluster.

6. The method as recited in claim 4 further comprising, upon the master node leaving the cluster:

informing the nodes of the master leaving the cluster;

determining which of the remaining nodes has a clock furthest ahead wherein the clock conforms to the criteria; and

transferring master status to the node with the conforming clock furthest ahead.

7. The method as recited in claim 4 further comprising, upon determining that a node does not conform to the criteria,

evicting the non-conforming node from the cluster;

or

performing a graceful shutdown on the non-conforming node.

8. The method as recited in claim 4 further comprising, upon discovering that more than one node is ahead of the master:

determining which node of the more than one nodes is furthest ahead and in conformance with the criteria; and

replacing the master node;

wherein the node that is furthest ahead and in conformance with the criteria becomes the new master.

9. The method as recited in claim 4 further comprising, upon detecting that a new node is attempting to join the cluster:

determining whether a clock of the new node conforms to the criteria; and

upon determining that the clock of the new node conforms to the criteria, allowing the new node to join the cluster; and

upon determining that the clock of the new node fails to conform to the criteria, deterring the new node from joining the cluster.

10. The method as recited in claim 9, wherein discovering a node of a cluster has a clock time that is ahead of the master clock time comprises:

detecting that the new node has the clock ahead of the master clock in time upon that node joining the cluster;

wherein replacing the master node comprises replacing the master node with the new node.

11. The method as recited in claim 10, wherein replacing the master node further comprises:

notifying all other nodes in the cluster that the joining node has achieved the master status;

wherein the other nodes function as slave nodes.

12. The method as recited in claim 1, wherein synchronizing times comprises:

exchanging synchronization messages between the master node and the slave nodes;

based on data in the synchronization messages, calculating a time offset between the clock of a particular slave node and the master clock; and

upon the time offset indicating that the clock of the particular slave node is behind the master clock in time, adjusting the clock of the particular slave node to agree with the master clock.

13. The method as recited in claim 1 wherein, upon initial formation of the cluster, a first node to join the cluster becomes the master and, upon a subsequent node joining the cluster, the subsequent node becomes a slave.

14. The method as recited in claim 1 wherein, upon a cluster reboot, the master node before the reboot remains the master node upon performing the reboot.

15. A method for synchronizing time in a computer cluster of nodes, comprising the steps of:

monitoring the clock times of the nodes in the cluster in relation to that of a first master node;

discovering that a node of a cluster has a clock time that is ahead of a master clock time of the first master node;

ascertaining that the node that has a clock time that is ahead of the first master clock in time is ahead in time to a degree that satisfies one or more criteria;

in response to ascertaining that the node with a clock time that is ahead, atomically transferring a master status from the first master node to the node ahead thereof in time wherein the node ahead in time becomes a new master node of the cluster and the clock time of the new master node becomes a new master clock time of the cluster;

synchronizing the clock times of other nodes in the cluster with the new master clock time; and

iteratively repeating the transferring step based on the discovering step;

wherein the steps are performed by one or more nodes in the cluster.

16. The method as recited in claim 15 wherein, the ascertaining comprises:

logging the time kept with each clock of nodes in the cluster;

computing a mean cluster time based on the time logged for said each clock of nodes in the cluster; and

determining that the clock time of the node ahead of the pre-existing master clock is within a range of the mean cluster time.

17. A computer readable volatile or non-volatile storage medium storing instructions encoded therewith which, when executed with one or more processors of a computer system, cause the processors to execute:

synchronizing times of one or more nodes in the cluster to a master node of the cluster;

discovering a node of a cluster has a clock time that is ahead of a master clock time of the master node;

in response to discovering the node with the clock time ahead of the master clock time,

replacing the master node with the node ahead thereof in time as a new master node; and

synchronizing the clock times of the nodes in the cluster with the clock time of the new master node.

18. The computer readable volatile or non-volatile storage medium of claim 17,

wherein replacing the master node is performed atomically between the original master node and the node ahead thereof in time.

19. The computer readable volatile or non-volatile storage medium of claim 17,

wherein replacing is performed when one or more criteria are satisfied.

20. The computer readable volatile or non-volatile storage medium of claim 19,

wherein one of the criteria is based on how advanced the clock of the node ahead of the original master node is, in relation to the other nodes in said cluster.

21. The computer readable volatile or non-volatile storage medium of claim 20,

wherein one of the criteria is based on the mean time of the clocks of the nodes in the cluster.

22. The computer readable volatile or non-volatile storage medium of claim 20,

the instructions further comprising instructions, which when executed by the one or more processors, cause:

upon the master node leaving the cluster;

informing the nodes of the master leaving the cluster;

determining which of the remaining nodes has a clock furthest ahead wherein the clock conforms to the criteria; and

transferring master status to the node with the conforming clock furthest ahead.

23. The computer readable volatile or non-volatile storage medium of claim 20,

the instructions further comprising instructions, which when executed by the one or more processors, cause:

upon determining that a node does not conform to the criteria,

evicting the non-conforming node from the cluster;

or

performing a graceful shutdown on the non-conforming node.

24. The computer readable volatile or non-volatile storage medium of claim 20,

the instructions further comprising instructions, which when executed by the one or more processors, cause:

upon discovering that more than one node is ahead of the master;

determining which node of the more than one nodes is furthest ahead and in conformance with the criteria; and

replacing the master node;

wherein the node that is furthest ahead and in conformance with the criteria becomes the new master.

25. The computer readable volatile or non-volatile storage medium of claim 20,

the instructions further comprising instructions, when executed by the one or more processors, cause:

upon detecting that a new node

is attempting to join the cluster;

determining whether a clock of the new node conforms to the criteria; and

upon determining that the clock of the new node conforms to the criteria, allowing the new node to join the cluster; and

upon determining that the clock of the new node fails to conform to the criteria, deterring the new node from joining the cluster.

26. The computer readable volatile or non-volatile storage medium of claim 25,

wherein the instructions that cause discovering further comprise instructions that cause:

detecting that the new node has the clock ahead of the master clock in time upon that node joining the cluster;

wherein replacing the master node comprises replacing the master node with the new node.

27. The computer readable volatile or non-volatile storage medium of claim 26,

wherein the instructions that cause replacing the master node, further comprise instructions that cause:

notifying all other nodes in the cluster that the joining node has achieved the master status;

wherein the other nodes function as slave nodes.

28. The computer readable volatile or non-volatile storage medium of claim 17,

wherein the instructions that cause synchronizing times further comprise instructions that cause:

exchanging synchronization messages between the master node and the slave nodes;

based on data in the synchronization messages, calculating a time offset between the clock of a particular slave node and the master clock; and

upon the time offset indicating that the clock of the particular slave node is behind the master clock in time, adjusting the clock of the particular slave node to agree with the master clock.

29. The computer readable volatile or non-volatile storage medium of claim 17,

wherein, upon initial formation of the cluster, a first node to join the cluster becomes the master;

and

upon a subsequent node joining the cluster, the subsequent node becomes a slave.

30. The computer readable volatile or non-volatile storage medium of claim 17,

wherein, upon a cluster reboot, the master node before the reboot remains the master node upon performing the reboot.

31. A computer readable volatile or non-volatile storage medium storing instructions encoded therewith which, when executed with one or more processors of a computer system, cause the processors to perform:

monitoring the clock times of the nodes in the cluster in relation to that of a first master node;

discovering that a node of a cluster has a clock time that is ahead of a master clock time of the first master node;

ascertaining that the node that is ahead of the first master clock in time is ahead in time to a degree that satisfies one or more criteria;

in response to ascertaining that the node is ahead, atomically transferring a master status from the first master node to the node ahead thereof in time wherein the node ahead in time becomes a new master node of the cluster and the clock time of the new master node becomes a new master clock time of the cluster;

synchronizing the clock times of other nodes in the cluster with the new master clock time; and

iteratively repeating the transferring step based on the discovering step.

32. The computer readable volatile or non-volatile storage medium of claim 31, wherein ascertaining comprises:

logging the time kept with each clock of nodes in the cluster;

computing a mean cluster time based on the time logged for said each clock of nodes in the cluster; and

determining that the clock time of the node ahead of the pre-existing master clock is within a range of the mean cluster time.