ADAPTIVE CLUSTER TIMER MANAGER

• US 20100211959A1
• Filed: 02/13/2009
• Published: 08/19/2010
• Est. Priority Date: 02/13/2009
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Ho, Yu Kin, Yurek, Tolga, Srivastava, Deepti, Chan, Wilson, Pruscino, Angelo, Hsu, John, Wang, Tak Fung
• Granted Patent: US 8,220,001 B2
• Time in Patent Office: Days
• US Class Current: 718/107
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0757 by exceeding a time limit, i.e. time-out, e.g. watchdogs

Abstract

Described herein are techniques for adaptively managing timers that are used in various layers of a node. In many cases, the number of timers that occur in the system is reduced by proactively and reactively adjusting values of the timers based on conditions affecting the system, thereby making such a system to perform significantly better and more resiliently than otherwise.

Claims

1. A computer-implemented method comprising:

determining a set of current timeout period values for a set of timers, wherein the set of timers forms a hierarchy based on a set of inter-dependent relationships among a plurality of components on a node; and

based on the hierarchy, adjusting a first timer in the set of timers to a first new timeout period value, wherein the first timer is used by a first task performed by a first component in the plurality of components;

wherein the method is performed by a computer programmed to be a special purpose machine pursuant to instructions from program software.

2. The method of claim 1, further comprising:

monitoring one or more conditions that affect tasks performed by one or more components in the plurality of components on the node; and

in response to the one or more conditions, preventing one or more timeout errors from occurring by adjusting a second timer in the set of timers to a second new timeout period value, wherein the second timer is used by a second task performed by a second component in the plurality of components.

3. The method of claim 2, wherein the node is one of multiple nodes in a multi-node system and wherein the one or more conditions include a condition that affects all of the multiple nodes.

4. The method of claim 2, wherein said monitoring one or more conditions that affect one or more components in the plurality of components on the node include determining one or more average times of completion for the tasks performed by the one or more components in the plurality of components.

5. The method of claim 2, wherein said monitoring one or more conditions that affect tasks performed by one or more components in the plurality of components on the node include:

detecting that one or more timers have expired; and

lengthening at least one of the one or more timers.

6. The method of claim 2, further comprising:

based on the hierarchy, adjusting a third timer to a third new timeout period value, wherein the hierarchy indicates that the third timer depends on the second timer and wherein the third new timeout period value is greater than the second new timeout period value.

7. The method of claim 2, wherein the node is one of multiple nodes in a multi-node system and wherein the one or more conditions include a condition that only affects the node.

8. The method of claim 1, further comprising based on the hierarchy, identifying a proper subset of timers in the set of timers, wherein other timers in the set of timers depend on the proper subset of timers.

9. The method of claim 1, wherein:

the first timer sets a time boundary for completion of a second task performed by a second component in the set of components;

the first task depends on the second task; and

a second timeout period value of a second timer is used by the second task performed by the second component; and

wherein said adjusting a first timer in the set of timers to a first new timeout period value comprises;

determining whether the first timeout period value is not greater than the second timeout period value plus a safety margin; and

in response to determining that the first timeout period value is not greater than the second timeout period value plus the safety margin, adjusting the first timer to the new first timeout period value, wherein the new first timeout period value is greater than the second timeout period value plus the safety margin.

10. The method of claim 1, wherein the set of components forms two or more layers.

11. The method of claim 1, further comprising:

monitoring one or more conditions that affect tasks performed by one or more components in the plurality of components on the node; and

suppressing expiration of a timer that is associated with the first timer during runtime of the first task.

12. The method of claim 1, further comprising:

monitoring one or more conditions that affect tasks performed by one or more components in the plurality of components on the node; and

delaying expiration of a timer that is associated with the first timer during runtime of the first task.

13. The method of claim 1, further comprising:

based on the hierarchy, creating a partially ordered set of timers from the set of timers; and

ensuring that a timeout period value of a descendent timer in the partially ordered set is greater than a timeout period value of a precedent timer in the partially ordered set.

14. The method of claim 1, wherein said adjusting a first timer in the set of timers to a first new timeout period value comprises:

based on the hierarchy, determining a range of acceptable timeout period values for the first timer; and

based on one or more considerations, selecting the first new timeout period value from the range of acceptable timeout period values.

15. A computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform:

determining a set of current timeout period values for a set of timers, wherein the set of timers forms a hierarchy based on a set of inter-dependent relationships among a plurality of components on a node; and

based on the hierarchy, adjusting a first timer in the set of timers to a first new timeout period value, wherein the first timer is used by a first task performed by a first component in the plurality of components.

16. The medium of claim 15, wherein the one or more sequences of instructions further comprise instructions which, when executed by one or more processors, causes the one or more processors to perform:

monitoring one or more conditions that affect tasks performed by one or more components in the plurality of components on the node; and

in response to the one or more conditions, preventing one or more timeout errors from occurring by adjusting a second timer in the set of timers to a second new timeout period value, wherein the second timer is used by a second task performed by a second component in the plurality of components.

17. The medium of claim 16, wherein the one or more sequences of instructions further comprise instructions which, when executed by one or more processors, causes the one or more processors to perform:

detecting that one or more timers have expired; and

lengthening at least one of the one or more timers.

18. The medium of claim 16, wherein the one or more sequences of instructions further comprise instructions which, when executed by one or more processors, causes the one or more processors to perform:

based on the hierarchy, adjusting a third timer to a third new timeout period value, wherein the hierarchy indicates that the third timer depends on the second timer and wherein the third new timeout period value is greater than the second new timeout period value.

19. The medium of claim 15, wherein the one or more sequences of instructions further comprise instructions which, when executed by one or more processors, causes the one or more processors to perform:

based on the hierarchy, identifying a proper subset of timers in the set of timers, wherein other timers in the set of timers depend on the proper subset of timers.

20. The medium of claim 15, wherein:

the first timer sets a time boundary for completion of a second task performed by a second component in the set of components;

the first task depends on the second task; and

a second timeout period value of a second timer is used by the second task in the second component; and

wherein the one or more sequences of instructions further comprise instructions which, when executed by one or more processors, causes the one or more processors to perform;

determining whether the first timeout period value is not greater than the second timeout period value plus a safety margin; and

in response to determining that the first timeout period value is not greater than the second timeout period value plus the safety margin, adjusting the first timer to the new first timeout period value, wherein the new first timeout period value is greater than the second timeout period value plus the safety margin.

21. The medium of claim 15, wherein the one or more sequences of instructions further comprise instructions which, when executed by one or more processors, causes the one or more processors to perform:

monitoring one or more conditions that affect tasks performed by one or more components in the plurality of components on the node; and

suppressing expiration of a timer that is associated with the first timer during runtime of the first task.

22. The medium of claim 15, wherein the one or more sequences of instructions further comprise instructions which, when executed by one or more processors, causes the one or more processors to perform:

monitoring one or more conditions that affect tasks performed by one or more components in the plurality of components on the node; and

delaying expiration of a timer that is associated with the first timer during runtime of the first task.

23. An apparatus comprising:

one or more processors; and

one or more sequences of instructions which, when executed by the one or more processors, causes the one or more processors to perform;

determining a set of current timeout period values for a set of timers, wherein the set of timers forms a hierarchy based on a set of inter-dependent relationships among a plurality of components on a node; and

based on the hierarchy, adjusting a first timer in the set of timers to a first new timeout period value, wherein the first timer is used by a first task performed by a first component in the plurality of components.

24. The apparatus of claim 23, wherein the one or more sequences of instructions further comprise instructions which, when executed by one or more processors, causes the one or more processors to perform:

monitoring one or more conditions that affect tasks performed by one or more components in the plurality of components on the node; and

in response to the one or more conditions, preventing one or more timeout errors from occurring by adjusting a second timer in the set of timers to a second new timeout period value, wherein the second timer is used by a second task performed by a second component in the plurality of components.

25. The apparatus of claim 23, wherein the one or more sequences of instructions further comprise instructions which, when executed by one or more processors, causes the one or more processors to perform:

monitoring one or more conditions that affect tasks performed by one or more components in the plurality of components on the node; and

suppressing expiration of a timer that is associated with the first timer during runtime of the first task.

26. The apparatus of claim 23, wherein the one or more sequences of instructions further comprise instructions which, when executed by one or more processors, causes the one or more processors to perform:

monitoring one or more conditions that affect tasks performed by one or more components in the plurality of components on the node; and

delaying expiration of a timer that is associated with the first timer during runtime of the first task.