CONTROLLED TAKE OVER OF SERVICES BY REMAINING NODES OF CLUSTERED COMPUTING SYSTEM

• US 20030159084A1
• Filed: 01/10/2000
• Published: 08/21/2003
• Est. Priority Date: 01/10/2000
• Status: Active Grant

• Current Assignee: Oracle America, Inc. (Oracle Corporation)
• Original Assignee: Sun Microsystems Incorporated (Oracle Corporation)
• Inventors: Kumar, Krishna, Murphy, Declan J., Hisgen, Andrew L.
• Granted Patent: US 6,789,213 B2
• Time in Patent Office: Days
• US Class Current: 714/13
• CPC Class Codes: G06F 11/203 using migration
• CPC Class Codes: G06F 11/2035 without idle spare hardware
• CPC Class Codes: G06F 11/2046 where the redundant components share persistent storage G06F11/2043 takes precedence

Abstract

Improved techniques for controlled take over of services for clustered computing systems are disclosed. The improved techniques can be implemented to allow one sub-cluster of the clustered computing system to safely take over services of one or more other sub-clusters in the clustered computing system. Accordingly, if the clustered computing system is fragmented into two or more disjointed sub-clusters, one sub-cluster can safely take over services of the one or more other sub-clusters after the one or more other sub-clusters have been shutdown. As a result, the clustered computing system can continue to safely provide services even when the clustered computing system has been fragmented into two or more disjointed sub-clusters due to an operational failure.

Claims

1. A method for taking over services by a sub-cluster of a clustered computing system from one or more other sub-clusters of the clustered computing system after the one or more other sub-clusters have been shutdown, said method comprising:

(a) attempting to determine whether a sub-cluster of the clustered computing system is to remain active;

(b) initiating shutdown of the sub-cluster when said attempting (a) does not determine within a first predetermined amount of time that the sub-cluster is to remain active;

(c) delaying for a second predetermined amount of time after the first predetermined amount of time expires when said attempting (a) determines within the first predetermined amount of time that the sub-cluster is to remain active; and

(d) taking over services of one or more other sub-clusters of the clustered computing system after said delaying (c) for the second predetermined amount of time.

2. A method as recited in claim 1, wherein said method is performed by each of the sub-clusters of the clustered computing system.

3. A method as recited in claim 1, said method further comprising:

initiating shutdown of the sub-cluster when said attempting (a) determines within the first predetermined amount of time that the sub-cluster is not to remain active.

4. A method as recited in claim 1, wherein the first predetermined amount of time represents an upper estimate of time required to determine whether a sub-cluster is to remain active.

5. A method as recited in claim 1, wherein the second predetermined amount of time represents an upper estimate for a delay typically encountered in initiating said attempting (a) after an error condition has actually occurred.

6. A method for taking over services by a sub-cluster of a clustered computing system from one or more other sub-clusters of the clustered computing system after the one or more other sub-clusters have been shutdown, said method comprising:

(a) determining whether one or more computing nodes in a cluster have become one or more non-responsive nodes;

(b) starting a first timer when said determining (a) determines that one or more of the computing nodes in the cluster have become one or more non-responsive nodes, the first timer having a first duration;

(c) attempting to determine whether a sub-cluster vote is at least a majority of a total votes available, the sub-cluster vote representing votes for a sub-cluster of one or more computing nodes, the sub-cluster representing a portion of the cluster that remains responsive;

(d) initiating shutdown of the one or more computing nodes of the sub-cluster when said attempting (c) does not determine within the first duration of the first timer that the sub-cluster vote is at least a majority of the total votes available;

(e) starting a second timer after the first timer expires when the said attempting (c) has determined within the first duration of the first timer that the sub-cluster vote is at least a majority of the total votes available, the second timer having a second duration; and

(f) taking over services from the one or more non-responsive nodes by at least one of the computing nodes of the sub-cluster after the second timer expires.

7. A method as recited in claim 6, wherein said method is performed by each of the sub-clusters of the clustered computing system.

8. A method as recited in claim 6, wherein the (a) determining further comprises:

(a1) detecting that a computing node in the cluster that has become non-responsive with respect to a first node;

(a2) determining whether all responsive nodes to the first node agree that the computing node detected in detecting (a1) has become non-responsive.

9. A method as recited in claim 6, wherein the method further comprises:

determining whether there is at least one service of the one or more non-responsive nodes that needs to be taken over.

10. A clustered computing system, comprising:

a cluster of computing nodes having at least two computing nodes; and

an integrity protector provided with each one of the computing nodes, said integrity protector operating to determine whether a set of computing nodes in the cluster are to remain active, the set of computing nodes representing at least a portion of the cluster, and said integrity protector operating to allow one or more computing nodes in the set of computing nodes to take over services of one or more other computing nodes of the clustered computing system only after the one or more other computing nodes have shutdown.

11. A clustered computing system as recited in claim 10,

wherein each node being assigned a node vote, and

wherein the integrity protector operates to determine whether the votes possessed by the set of computing nodes are a majority of total votes available in the clustered computing system.

12. A clustered computing system as recited in claim 11, wherein the clustered computing system further comprises:

a proxy device, the proxy device being assigned a proxy vote which is capable of being acquired by the integrity protector on behalf of at least one of the computing nodes in the set of computing nodes.

13. A clustered computing system as recited in claim 10, wherein the integrity protector comprises:

a cluster error detector suitable for detecting formation of two or more sub-clusters within the clustered computing system;

a cluster shutdown controller suitable to determining whether the set of computing nodes is to be shutdown; and

a takeover controller suitable for allowing at least one computing node of the set of computing nodes to take over services of one or more other computing nodes after the one or more other computing nodes have shutdown.

14. A clustered computing system as recited in claim 13, wherein the cluster shutdown controller comprises:

a first timer suitable for measuring an amount time that the cluster shutdown controller has to determine whether the set of computing nodes is to be shutdown; and

a second timer suitable for measuring an amount of time after the first timer has expired that the cluster shutdown controller delays before activating the takeover controller.

15. A computer readable medium including computer program code for taking over services by a sub-cluster of a clustered computing system from one or more other sub-clusters of the clustered computing system after the one or more other sub-clusters have been shutdown, said computer readable medium comprising:

computer program code for attempting to determine whether a sub-cluster of the clustered computing system is to remain active;

computer program code for initiating shutdown of the sub-cluster when said computer program code for attempting does not determine within a first predetermined amount of time that the sub-cluster is to remain active;

computer program code for delaying for a second predetermined amount of time after the first predetermined amount of time expires when said computer program code for attempting determines within the first predetermined amount of time that the sub-cluster is to remain active; and

computer program code for taking over services of one or more other sub-clusters of the clustered computing system after said computer program code for delaying has delayed for the second predetermined amount of time.

16. A computer readable medium as recited in claim 15, wherein the computer readable medium is provided for each of the sub-clusters of the clustered computing system.

17. A method as recited in claim 15, said method further comprising:

computer program code for initiating shutdown of the sub-cluster when said computer program code for attempting determines within the first predetermined amount of time that the sub-cluster is not to remain active.

18. A computer readable medium for taking over services by a sub-cluster of a clustered computing system from one or more other sub-clusters of the clustered computing system after the one or more other sub-clusters have been shutdown, said computer readable medium comprising:

computer program code for determining whether one or more computing nodes in a cluster have become one or more non-responsive nodes;

computer program code for starting a first timer when said computer program code for determining determines that one or more of the computing nodes in the cluster have become one or more non-responsive nodes, the first timer having a first duration;

computer program code for attempting to determine whether a sub-cluster vote is at least a majority of a total votes available, the sub-cluster vote representing votes for a sub-cluster of one or more computing nodes, the sub-cluster representing a portion of the cluster that remains responsive;

computer program code for initiating shutdown of the one or more computing nodes of the sub-cluster when said computer program code for attempting does not determine within the first duration of the first timer that the sub-cluster vote is at least a majority of the total votes available;

computer program code for starting a second timer after the first timer expires when the said computer program code for attempting has determined within the first duration of the first timer that the sub-cluster vote is at least a majority of the total votes available, the second timer having a second duration; and

computer program code for taking over services from the one or more non-responsive nodes by at least one of computing nodes of the sub-cluster after the second timer expires.

19. A computer readable medium as recited in claim 18, wherein the computer readable medium is provided for each of the sub-clusters of the clustered computing system.