Time synchronization in cluster systems

• US 8,169,856 B2
• Filed: 10/24/2008
• Issued: 05/01/2012
• Est. Priority Date: 10/24/2008
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Rai, Vikram, Srivastava, Alok, Pruscino, Angelo, Joshi, Sameer
• Primary Examiner(s): Miska, Vit
• Application Number:
  US12/258,344
• Publication Number:
  US 20100103781A1
• Time in Patent Office: 1,285 Days
• Field of Search: 368/46, 368/47, 370/350, 370/503, 370/509, 375/354, 375/356, 709/248
• US Class Current: 368/46
• CPC Class Codes: G06F 1/12 Synchronisation of different clock signals provided by a plurality of clock generators
• CPC Class Codes: H04J 3/0667 Bidirectional timestamps, e.g. NTP or PTP for compensation of clock drift and for compensation of propagation delays arrangements for monitoring round trip delays in packet switching networks H04L43/0864

Abstract

Techniques are described herein for synchronizing cluster time. According to one technique, a master node is appointed in a cluster. Other “slave” nodes periodically synchronize their clocks with the master node. To synchronize its clock with the master node, a slave node sends a timestamped message to the master node, which also timestamps the message and sends the message back to the slave node, which then timestamps the message again. Based on the timestamps, the slave node is able to determine the difference between the master node'"'"'s clock'"'"'s time and slave node'"'"'s clock'"'"'s time, compensating for the message travel time between master node and slave node. Depending on various circumstances, and based on the determined difference, the slave node adjusts its clock so that the time indicated by the slave node'"'"'s clock at least begins to approach more closely the time indicated by the master node'"'"'s clock.

Claims

1. A computer-implemented method comprising:

sending, toward a master node, a first message that indicates a first time at which the first message is sent toward the master node, as determined based on a local clock of a particular node that sends the first message;

receiving, from the master node, a second message that indicates a second time at which the master node received the first message, as determined based on a master clock of the master node;

based on at least the first time, the second time, and a third time at which the second message was received from the master node, as determined based on the local clock, adjusting a setting of the local clock to follow more closely a setting of the master clock.

2. The method as recited in claim 1, wherein the step of adjusting the setting of the local clock comprises:

determining a time delay value by subtracting the first time from the third time;

determining a delta value by subtracting the first time from the second time to produce a difference, and then subtracting half of the time delay value from the difference; and

adjusting the setting of the local clock based on the delta value.

3. The method as recited in claim 2, wherein the step of adjusting the setting of the local clock comprises:

determining whether the delta value is greater than a specified threshold; and

in response to determining that the delta value is greater than the specified threshold, refraining from adjusting the setting of the local clock until further messages are exchanged with the master node.

4. The method as recited in claim 2, wherein the step of adjusting the setting of the local clock comprises:

determining whether the delta value is greater than a specified threshold; and

in response to determining that the delta value is not greater than the specified threshold, determining whether the delta value is positive; and

in response to determining that the delta value is positive, adjusting the setting of the local clock by increasing the setting of the local clock by the delta value.

5. The method as recited in claim 2, wherein the step of adjusting the setting of the local clock comprises:

determining whether the delta value is greater than a specified threshold; and

in response to determining that the delta value is not greater than the specified threshold, determining whether the delta value is negative; and

in response to determining that the delta value is negative, adjusting the setting of the local clock by slowing down a rate at which the local clock advances.

6. The method as recited in claim 2, wherein the step of adjusting the setting of the local clock comprises:

determining whether the delta value is greater than a specified threshold; and

in response to determining that the delta value is not greater than the specified threshold, determining whether the delta value is negative; and

in response to determining that the delta value is negative, increasing a frequency with which time synchronization messages are exchanged with the master node.

7. The method as recited in claim 1, wherein the step of adjusting the setting of the local clock comprises:

adjusting the setting of the local clock by slowing down a rate at which the local clock advances by a maximum amount allowed by an operating system of the local node; and

increasing a frequency with which the local node slows down the rate at which the local clock advances;

wherein the frequency is determined based on a number of times that the local clock will need to be slowed down by the maximum rate in order to cause the local clock to become synchronized with the master node within a specified amount of time.

8. The method as recited in claim 1, wherein the step of adjusting the setting of the local clock comprises:

adjusting the setting of the local clock without reference to any time source outside of a node cluster to which the local node and the master node belong.

9. The method as recited in claim 1, further comprising:

determining that the master node cannot continue to function in a master node role; and

in response to determining that the master node cannot continue to function in the master node role, initiating a process by which another node in a node cluster to which the master node and the local node belong is selected to become a new master node for the cluster, with which all other nodes in the cluster will synchronize clocks of those other nodes.

10. The method as recited in claim 1, further comprising:

determining that the master clock is unsynchronized with a majority of clocks of other nodes in a node cluster to which the master node belongs; and

in response to determining that the master clock is unsynchronized with a majority of clocks of other nodes in the cluster, initiating a process by which another node in the node cluster is selected to become a new master node for the cluster, with which all other nodes in the cluster will synchronize clocks of those other nodes.

11. The method as recited in claim 1, further comprising:

the master node voluntarily relinquishing the master node'"'"'s status as master node; and

in response to the master node voluntarily relinquishing the master node'"'"'s status as master node, appointing another node in the cluster to become a new master node for a cluster to which the previous master node belongs.

12. The method as recited in claim 1, further comprising:

in response to an attempt, by a node that is not a member of a cluster to which the master node belongs, to become a member of the cluster, determining an extent to which a clock of the node that is not a member of the cluster is synchronized with the master clock; and

in response to determining that the clock of the node that is not a member of the cluster is out of synchrony with the master clock by at least a specified amount, preventing the node that is not a member of the client from becoming a member of the cluster.

13. The method as recited in claim 1, further comprising:

determining whether an additional time synchronization facility is operational within a cluster to which the master node belongs;

in response to determining that said additional time synchronization facility is operational within said cluster, placing said master node into an observer status in which said master node does not attempt to synchronize times kept by any node in said cluster;

periodically determining whether said additional time synchronization facility is configured improperly;

in response to determining that said additional time synchronization facility is configured improperly, alerting a user that said additional time synchronization facility is configured improperly; and

in response to a determination that said additional time synchronization facility has been removed, transitioning said master node out of said observer status into an active status in which said master node actively synchronizes times kept by other nodes in said cluster.

14. A volatile or non-volatile computer-readable storage medium on which are encoded instructions which, when executed by one or more processors, cause the one or more processors to perform steps comprising:

sending, toward a master node, a first message that indicates a first time at which the first message is sent toward the master node, as determined based on a local clock of a particular node that sends the first message;

receiving, from the master node, a second message that indicates a second time at which the master node received the first message, as determined based on a master clock of the master node;

based on at least the first time, the second time, and a third time at which the second message was received from the master node, as determined based on the local clock, adjusting a setting of the local clock to follow more closely a setting of the master clock.

15. The computer-readable medium as recited in claim 14, wherein the step of adjusting the setting of the local clock comprises:

determining a time delay value by subtracting the first time from the third time;

determining a delta value by subtracting the first time from the second time to produce a difference, and then subtracting half of the time delay value from the difference; and

adjusting the setting of the local clock based on the delta value.

16. The computer-readable medium as recited in claim 15, wherein the step of adjusting the setting of the local clock comprises:

determining whether the delta value is greater than a specified threshold; and

in response to determining that the delta value is greater than the specified threshold, refraining from adjusting the setting of the local clock until further messages are exchanged with the master node.

17. The computer-readable medium as recited in claim 15, wherein the step of adjusting the setting of the local clock comprises:

determining whether the delta value is greater than a specified threshold; and

in response to determining that the delta value is not greater than the specified threshold, determining whether the delta value is positive; and

in response to determining that the delta value is positive, adjusting the setting of the local clock by increasing the setting of the local clock by the delta value.

18. The computer-readable medium as recited in claim 15, wherein the step of adjusting the setting of the local clock comprises:

determining whether the delta value is greater than a specified threshold; and

in response to determining that the delta value is not greater than the specified threshold, determining whether the delta value is negative; and

in response to determining that the delta value is negative, adjusting the setting of the local clock by slowing down a rate at which the local clock advances.

19. The computer-readable medium as recited in claim 15, wherein the step of adjusting the setting of the local clock comprises:

determining whether the delta value is greater than a specified threshold; and

in response to determining that the delta value is not greater than the specified threshold, determining whether the delta value is negative; and

in response to determining that the delta value is negative, increasing a frequency with which time synchronization messages are exchanged with the master node.

20. The computer-readable medium as recited in claim 15, wherein the step of adjusting the setting of the local clock comprises:

adjusting the setting of the local clock by slowing down a rate at which the local clock advances by a maximum amount allowed by an operating system of the local node; and

increasing a frequency with which the local node slows down the rate at which the local clock advances;

wherein the frequency is determined based on a number of times that the local clock will need to be slowed down by the maximum rate in order to cause the local clock to become synchronized with the master node within a specified amount of time.

21. The computer-readable medium as recited in claim 14, wherein the step of adjusting the setting of the local clock comprises:

adjusting the setting of the local clock without reference to any time source outside of a node cluster to which the local node and the master node belong.

22. The computer-readable medium as recited in claim 14, wherein the steps further comprise:

determining that the master node cannot continue to function in a master node role; and

in response to determining that the master node cannot continue to function in the master node role, initiating a process by which another node in a node cluster to which the master node and the local node belong is selected to become a new master node for the cluster, with which all other nodes in the cluster will synchronize clocks of those other nodes.

23. The computer-readable medium as recited in claim 14, wherein the steps further comprise:

determining that the master clock is unsynchronized with a majority of clocks of other nodes in a node cluster to which the master node belongs; and

in response to determining that the master clock is unsynchronized with a majority of clocks of other nodes in the cluster, initiating a process by which another node in the node cluster is selected to become a new master node for the cluster, with which all other nodes in the cluster will synchronize clocks of those other nodes.

24. The computer-readable medium as recited in claim 14, wherein the steps further comprise:

the master node voluntarily relinquishing the master node'"'"'s status as master node; and

in response to the master node voluntarily relinquishing the master node'"'"'s status as master node, appointing another node in the cluster to become a new master node for a cluster to which the previous master node belongs.

25. The computer-readable medium as recited in claim 14, wherein the steps further comprise:

in response to an attempt, by a node that is not a member of a cluster to which the master node belongs, to become a member of the cluster, determining an extent to which a clock of the node that is not a member of the cluster is synchronized with the master clock; and

in response to determining that the clock of the node that is not a member of the cluster is out of synchrony with the master clock by at least a specified amount, preventing the node that is not a member of the client from becoming a member of the cluster.

26. The computer-readable medium as recited in claim 14, wherein the steps further comprise:

determining whether an additional time synchronization facility is operational within a cluster to which the master node belongs;

in response to determining that said additional time synchronization facility is operational within said cluster, placing said master node into an observer status in which said master node does not attempt to synchronize times kept by any node in said cluster;

periodically determining whether said additional time synchronization facility is configured improperly;

in response to determining that said additional time synchronization facility is configured improperly, alerting a user that said additional time synchronization facility is configured improperly; and

in response to a determination that said additional time synchronization facility has been removed, transitioning said master node out of said observer status into an active status in which said master node actively synchronizes times kept by other nodes in said cluster.