Method and apparatus for managing operations of clustered computer systems

• US 6,862,613 B1
• Filed: 01/10/2000
• Issued: 03/01/2005
• Est. Priority Date: 01/10/2000
• Status: Expired due to Term

• Current Assignee: Oracle America, Inc. (Oracle Corporation)
• Original Assignee: Sun Microsystems Incorporated (Oracle Corporation)
• Inventors: Kumar, Krishna, Hisgen, Andrew L., Block, Robert, Murphy, Declan J.
• Primary Examiner(s): Alam, Hosain
• Assistant Examiner(s): Qureshi, Shabana
• Application Number:
  US09/480,785
• Time in Patent Office: 1,877 Days
• Field of Search: 709/221, 709/202, 709/224, 709/226, 709/220, 714/4, 714/11, 714/22, 714/13
• US Class Current: 709/220
• CPC Class Codes: G06F 11/181 Eliminating the failing redundant component

Abstract

Improved techniques for managing operations of clustered computing system are disclosed. The improved techniques provide protection against potential problems encountered in operation of clustered computing. More particularly, the improved techniques can be implemented as an integral solution that provide protection against undesired partitions in space and partitions in time. The improved techniques do not require any human intervention.

Claims

1. A method for managing operation of a clustered computing system, the clustered computing system including at least a cluster of computing nodes and at least one peripheral device, wherein said clustered computing system is configured to interact with a user as a single entity, said method comprising:

(a) determining whether one or more of the computing nodes in the cluster have become one or more non-responsive nodes;

(b) determining a sub-cluster vote for a sub-cluster of one or more responsive computing nodes, wherein the sub-cluster represents a portion of the cluster that remains responsive;

(c) obtaining a total votes for the clustered computing system;

(d) determining whether the sub-cluster vote is at least a majority of the total votes;

(e) initiating shut down of the one or more computing nodes within the sub-cluster when said determining (d) determines that the sub-cluster vote is not at least a majority of the total votes; and

wherein said determining of sub-cluster vote includes soliciting a proxy vote from the at least one device using a reservation key.

2. A method as recited in claim 1, wherein the (a) determining comprises:

(a1) detecting that a computing node in the cluster that has become non-responsive with respect to a first computing node; and

(a2) determining whether all responsive nodes to the first computing node agree that the computing node detected in detecting (a1) has become non-responsive.

3. A method as recited in claim 1, wherein the method further comprises:

(f) taking over services from at least one of the non-responsive nodes by at least one of computing nodes in the sub-cluster when said determining (d) determines that the sub-cluster vote is at least a majority of the total votes.

4. A method as recited in claim 1, wherein said determining (d) of the sub-cluster vote comprises:

(d1) identifying the computing nodes within the sub-cluster; and

(d2) summing node votes for each of the computing nodes within the sub-cluster to obtain the sub-cluster vote.

5. A method as recited in claim 1,

wherein the computing nodes of the sub-cluster are assigned a node vote, and

wherein said determining (d) of the sub-cluster vote comprises summing the node votes for the computing nodes of the sub-cluster.

6. A method as recited in claim 5, wherein the node vote is an integer equal to or greater than 1.

7. A method as recited in claim 1,

wherein the computing nodes in the sub-cluster are assigned a node vote,

wherein the clustered computing system includes at least one peripheral device that is assigned a proxy vote such that at least one of the computing nodes in the sub-cluster is capable of soliciting the proxy vote, and

wherein said determining (d) of the sub-cluster vote comprises summing the node votes for the computing nodes of the sub-cluster and the proxy vote if acquired by one of the computing nodes of the sub-cluster.

8. A method as recited in claim 7, wherein the proxy vote assigned to the at least one device is a number (N−

1), where N is the number of nodes that can solicit the proxy vote.

9. A method as recited in claim 1, wherein said determining (d) of the sub-cluster vote comprises:

(d1) determining a first number of votes for a first computing node in the sub-cluster;

(d2) distributing the first number of votes to each of the other computing nodes in the sub-cluster;

(d3) receiving a second number of votes from a second computing node in the sub-cluster; and

(d4) adding the first number of votes to the second number of votes.

10. A method as recited in claim 9, wherein the clustered computing system includes at least one device that is assigned a proxy vote, and

wherein the computing nodes in the sub-cluster are assigned a node vote, and

wherein said determining (d1) the first number of votes comprises:

determining whether the proxy vote of the device can be solicited by the first computing node;

soliciting the proxy vote when said determining whether a proxy vote can be solicited by the first computing node determines that at least one proxy vote can be solicited by the first computing node;

acquiring the proxy vote by the first computing node when said soliciting has been successful; and

adding the proxy vote to the node vote assigned to the first computing node when the proxy vote has been acquired by the first computing node.

11. A method as recited in claim 10, wherein said determining whether a proxy vote can be solicited by the first computing node comprises reading configuration information to determine whether the device associated with the proxy vote is configured for access by the first computing node.

12. A method as recited in claim 11, wherein said soliciting of the proxy vote comprises:

reading a reservation key associated with the device associated with the proxy vote; and

determining whether the first computing node has permission to solicit the proxy vote based on the reservation key.

13. A method as recited in claim 12, wherein said acquiring of the proxy vote comprises modifying the proxy information to disallow other computing nodes from acquiring the proxy vote when the proxy vote has been successfully acquired by the first computing node.

14. A method as recited in claim 10, wherein said determining (d1) the first number of votes further comprises:

preventing other ones of the computing nodes of the sub-cluster from acquiring of the proxy vote when the proxy vote has been successfully acquired by the first computing node.

15. A method for managing startup operations of a cluster of computing nodes in a clustered computing system including at least one peripheral device, said method comprising:

(a) determining a cluster vote for the cluster with each node being assigned a node vote and each at least one peripheral device being assigned a proxy vote and wherein the cluster vote includes the node votes and proxy votes associated with the cluster,

(b) obtaining a total votes for the clustered computing system wherein the total votes include each node vote and each proxy vote in the clustered computing system;

(c) determining whether the cluster vote is at least a majority of the total votes; and

(d) initiating shut down of the computing nodes within the cluster when said (c) determining determines that the cluster vote is not at least a majority of the total votes.

16. A method as recited in claim 15, wherein the method further comprises:

(e) determining whether configuration information needs to be provided to at least one computing node; and

(f) providing the configuration information to the at least one node when the determining (e) determines that configuration information needs to be provided.

17. A clustered computing system, comprising:

a cluster of computing nodes having at least two computing nodes and at least one peripheral device, with each node being assigned a node vote and said at least one peripheral device being assigned a proxy vote; and

an integrity protector provided on each one of the computing nodes, the integrity protector determining a vote count for a set of computing nodes in the cluster, the set of nodes representing at least a portion of the cluster, and the integrity protector determining whether the set of computing nodes should be shut down based on the vote count.

18. A clustered computing system as recited in claim 17, wherein the integrity protector initiates shut down of the set of computing nodes when the vote count is not at least a majority of a total vote, the total vote representing the total votes available in the clustered computing system.

19. A clustered computing system as recited in claim 18, wherein the services of the set of computing nodes are taken over by one or more other computing nodes when the vote count is not at least a majority of the total vote.

20. A clustered computing system as recited in claim 17, wherein the clustered computing system further comprises:

a proxy device, the proxy device being assigned a proxy vote which is capable of being acquired by the integrity protector.

21. A clustered computing system as recited in claim 20, wherein the integrity protector solicits the proxy vote.

22. A clustered computing system as recited in claim 21,

wherein the integrity protector acquires the proxy vote, and

wherein the proxy vote acquired is used to determine the vote count.

23. A clustered computing system as recited in claim 22, wherein integrity protector prevents the proxy vote from being acquired when the proxy vote is acquired by the integrity protector.

24. A clustered computing system as recited in claim 20, wherein the integrity protector utilizes a SCSI-3 protocol.

25. A clustered computing system as recited in claim 17, wherein the integrity protector prevents partitions in space and partitions in time.

26. A computer readable media including computer program code for managing operation of a clustered computing system, the clustered computing system including at least one cluster of computing nodes and a peripheral device, said computer readable media comprising:

computer program code for determining whether one of the computing nodes in the cluster has become a non-responsive node in a non-responsive sub-cluster;

computer program code for determining a sub-cluster vote for a sub-cluster wherein the sub-cluster votes include votes for said computing nodes and said peripheral device, wherein the sub-cluster representing a portion of the cluster that remains responsive;

computer program code for obtaining a total votes for said clustered computing system, wherein the total votes include votes for the computing nodes and said peripheral device;

computer program code for determining whether the sub-cluster vote is at least a majority of the total votes; and

computer program code for initiating shut down of the computing nodes within the sub-cluster when said computer program code for determining whether the sub-cluster vote is at least a majority of the total votes determines that the sub-cluster vote is not at least a majority of total votes.

27. A computer readable media as recited in claim 26, wherein said computer readable media further comprises:

computer program code for taking over services from the non-responsive node by at least one of computing nodes in the sub-cluster when said computer program code for determining whether the sub-cluster vote is at least a majority of the total votes determines that the sub-cluster vote is at least a majority of the total votes.

28. A computer readable media as recited in claim 27, wherein said computer program code for determining of the sub-cluster vote comprises:

computer program code for determining the computing nodes within the sub-cluster; and

computer program code for summing node votes for each of the computing devices within the sub-cluster to obtain the sub-cluster vote.