Using a cluster-wide shared repository to provide the latest consistent definition of the cluster (avoiding the partition-in time problem)

• US 7,020,695 B1
• Filed: 05/28/1999
• Issued: 03/28/2006
• Est. Priority Date: 05/28/1999
• Status: Expired due to Term

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Kundu, Joydip, O'Shaughnessy, Daniel E., Cusson, Michael J., Frank, Richard L.
• Primary Examiner(s): Vaughn, Jr., William C.
• Application Number:
  US09/322,472
• Time in Patent Office: 2,496 Days
• Field of Search: 709/103, 709/209, 709/226, 709/220, 709/208, 709/213, 709/242, 709/223, 709/224, 709/221, 709/204, 707/203
• US Class Current: 709/223
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A quorumless network cluster provides a highly available system by addressing the partition-in-space and partition-in-time problems in network clusters.

In a particular solution, a cluster manager (CM) can use disk based messaging to manage the operation of the cluster. Each node within the cluster must have access to a shared disk to operate within the cluster.

A particular methodology can operate the cluster in a closed loop between nodes 1 to N. If a node fails to receive a heartbeat message from its predecessor in the loop, it initiates a cluster reconfiguration by sending a reconfiguration message to each other node in the cluster.

The quorumless cluster can also include a common storage for a cluster definition. Each node may provide a proposed change to the cluster definition, however only a single coordinator node may update the cluster definition and apply the suggested changes.

Claims

1. A method for maintaining a cluster definition for a network cluster of computing nodes having a plurality of member nodes and a network structure for facilitating network connectivity between nodes, the method comprising:

coupling at least two member nodes to a shared repository over a storage connection separate from the network structure;

storing a cluster definition for the network cluster in the shared repository;

selecting a coordinator node from the at least two member nodes of the network cluster;

accessing, by a member node, the cluster definition on the shared repository using the storage connection, the member node having membership in the network cluster regardless of network connectivity with another node;

at a member node, requesting a change to the cluster definition by sending a proposed change to the shared repository over the storage connection; and

in response to the proposed change request, updating, from the coordinator node, the cluster definition stored in the shared repository to reflect the requested change.

2. The method of claim 1 wherein requesting a change to the cluster definition includes:

sending the proposed change to a scratch area of the shared repository; and

setting a valid bit associated with the scratch area of the shared repository.

3. The method of claim 2 wherein updating the cluster definition includes:

verifying the valid bit;

setting an update flag;

modifying the cluster definition to reflect the requested change;

logging a progress of modifying the cluster definition in a log file in parallel with modifying the cluster definition;

incrementing a version number associated with the shared repository; and

clearing the valid bit and the update flag.

4. The method of claim 3 wherein modifying the cluster definition includes:

copying the proposed change from the scratch area to the cluster definition.

5. The method of claim 2 further including the step of:

re-requesting, by the member node, the change to the cluster definition if after a period of time, the change is not made to the cluster definition.

6. The method of claim 1 further comprising:

requesting, by a potential member node, membership in the network cluster; and

accessing, by the potential member node, the cluster definition stored in the shared repository.

7. The method of claim 6 wherein accessing the cluster definition includes:

determining a version number of the shared repository to yield a first version number;

reading the cluster definition;

re-determining a version number of the shared repository to yield a second version number;

comparing the first version number with the second version number; and

repeating the step of accessing the cluster definition until the first version number equals the second version number.

8. The method of claim 1 further comprising:

recovering from a failure of the coordinating node.

9. The method of claim 8 wherein recovering includes:

selecting a new coordinator from the member nodes of the network cluster; and

completing, by the new coordinator node, an update of the cluster definition to reflect the proposed change if there is a set valid bit and an incomplete log file in the shared repository.

10. The method of claim 9 wherein completing an update includes:

reading the incomplete log file; and

continuing the update of the cluster definition from a point, as indicated by the incomplete log file, where the coordinating node ceased updating the cluster definition due to the failure of the coordinating node.

11. An apparatus for updating a cluster definition for a network cluster of computing nodes having a plurality of member nodes and a network structure for facilitating network connectivity between nodes, comprising:

a shared repository coupled to at least two member nodes of the cluster with a storage connection separate from the network structure, the shared repository including the cluster definition;

a member node to access the cluster definition on the shared repository over the storage connection separate from the network structure, the member node having membership in the network cluster regardless of network connectivity with another node;

a member node to request a change to the cluster definition by sending a proposed change to the shared repository over the network connection; and

a coordinator node, selected from the at least two member nodes of the network cluster, to update the cluster definition with the proposed change.

12. The apparatus of claim 11 further including:

a log file, indicating a progress of updating the cluster definition.

13. The apparatus of claim 11 wherein the proposed change is stored in a scratch area of the shared repository.

14. The apparatus of claim 13 further comprises:

a set valid bit associated with a scratch area;

an update flag indicating the valid bit is verified by the coordinator node; and

a version number of the shared repository, incremented by the coordinator node, to indicate an update to the cluster definition.

15. The apparatus of claim 13 wherein the coordinator node updates the cluster definition by copying the proposed change from the scratch area to the cluster definition and clearing the valid bit and the update flag.

16. The apparatus of claim 11 wherein a member node requests the proposed change to the cluster definition, and if after a period of time the proposed change is not made to the cluster definition, the member node re-requests the proposed change to the cluster definition.

17. The apparatus of claim 11 further comprises:

a potential member node to request membership in the network cluster by accessing the cluster definition.

18. The apparatus of claim 17 wherein the potential member node further includes logic for:

determining a version number of the shared repository to yield a first version number;

reading the cluster definition;

re-determining a version number of the shared repository to yield a second version number;

comparing the first version number with the second version number; and

repeating the step of accessing the cluster definition until the first version number equals the second version number.

19. The apparatus of claim 11 further comprises:

a new coordinator node, selected from the at least two member nodes, to update the cluster definition if the coordinator node fails to operate.

20. The apparatus of claim 19 wherein the new coordinator node completes the update to cluster definition to reflect the requested change if there is a set valid bit and an incomplete log file in the shared repository.

21. The apparatus of claim 20 wherein the new coordinator node completes the update by reading the incomplete log file, and continuing the update from a point, as indicated by the incomplete log file, where the coordinating node ceased updating the cluster definition due to the failure of the coordinator node.

22. A computer program product for maintaining a cluster definition for a network cluster of computing nodes having a plurality of member nodes and a network structure for facilitating network connectivity between nodes, the computer program product comprising:

a computer usable medium having computer readable program code thereon, including program code for;

coupling at least two member nodes to a shared repository over a storage connection separate from the network structure;

storing a cluster definition for the network cluster in the shared repository;

selecting a coordinator node from the at least two member nodes of the network cluster;

directing a member node to access the cluster definition on the shared repository over the storage connection, the member node being a member in the network cluster regardless of network connectivity with another node;

requesting a change to the cluster definition by sending a proposed change to the shared repository over the storage connection; and

directing the coordinator node to update the cluster definition in response to the requested change to the cluster definition.

23. The computer program product of claim 22 wherein the request to change the cluster definition further includes program code for:

sending a proposed change to a scratch area of the shared repository; and

setting a valid bit associated with the scratch area of the shared repository.

24. The computer program product of claim 23 wherein the program code which directs the coordinator node to update the cluster definition further comprises program code for:

verifying the valid bit;

setting an update flag;

modifying the cluster definition to reflect the requested change;

logging a progress of modifying the cluster definition in a log file in parallel with modifying the cluster definition;

incrementing a version number associated with the shared repository; and

clearing the valid bit and the update flag.

25. The computer program product of claim 24 wherein the program code for modifying the cluster definition further includes program code for:

copying the proposed change from the scratch area to the cluster definition.

26. The computer program product of claim 22 further comprises program code for:

directing a potential member node to request membership in the network cluster; and

directing the potential member node to access the cluster definition.

27. The computer program product of claim 26 wherein the program code for directing the potential member node to access the cluster definition further includes program code for:

determining a version number of the shared repository to yield a first version number;

reading the cluster definition;

re-determining a version number of the shared repository to yield a second version number;

comparing the first version number with the second version number; and

accessing the cluster definition until the first version number equals the second version number.

28. The computer program product of claim 22 further includes program code for:

recovering from a failure of the coordinating node.

29. The computer program product of claim 28 wherein the program code for recovering further includes program code for:

selecting a new coordinator node from the member nodes of the network cluster; and

directing the new coordinator node to complete an update of the cluster definition to reflect the requested change if there is a set valid bit and an incomplete log file in the shared repository.

30. The computer program product of claim 29 wherein the program code for directing the new coordinator node to complete an update further includes program code for directing the new coordinator node to:

read the incomplete log file; and

continue the update of the cluster definition from a point, as indicated by the incomplete log file, where the coordinating node ceased updating the cluster definition due to the failure of the coordinating node.

31. The computer program product of claim 22 further comprises program code for:

directing the member node to re-request the change to the cluster definition if after a period of time, the change is not made to the cluster definition.

32. A system for maintaining a cluster definition for a network cluster of computing nodes having a plurality of member nodes and a network structure for facilitating network connectivity between nodes, the system comprising:

a means for coupling at least two member nodes to a shared repository over a storage connection;

a means for storing a cluster definition for the network cluster in the shared repository;

a means for selecting a coordinator node from the at least two member nodes of the network cluster;

a means for accessing, by a member node, the cluster definition on the shared repository over the storage connection, the member node having membership in the cluster regardless of network connectivity with another node;

a means for requesting a change to the cluster definition by sending a proposed change the shared repository over the storage connection; and

a means for the coordinator node to update the cluster definition to reflect the requested change.

33. A method for updating a cluster definition for a network cluster of computing nodes having a plurality of member nodes and a network structure for facilitating network connectivity between nodes, the method comprising:

coupling at least two member nodes to a shared repository over a storage connection separate from the network structure;

storing a cluster definition for the network cluster in the shared repository;

selecting a coordinator node from the at least two member nodes of the network cluster;

at a member node, requesting a change to the cluster definition;

from the coordinator node, updating the cluster definition over the storage connection to reflect the requested change; and

from a potential member node, accessing the cluster definition stored in the shared repository over the storage connection, the potential member node being granted membership in the network cluster regardless of network connectivity with another node.

34. The method of claim 33 wherein requesting a change to the cluster definition further includes:

sending a proposed change to a scratch area; and

setting a valid bit associated with the scratch area.

35. The method of claim 34 wherein updating the cluster definition includes:

verifying the valid bit;

setting an update flag;

modifying the cluster definition to reflect the requested change by copying the proposed change from the scratch area to the cluster definition;

logging a progress of modifying the cluster definition in a log file in parallel with modifying the cluster definition;

incrementing a version number associated with the shared repository; and

clearing the valid bit and the update flag.

36. The method of claim 33 further including the step of:

re-requesting, by the member node, the change to the cluster definition if after a period of time, the change is not made to the cluster definition.

37. The method of claim 33 wherein the potential member node accessing the cluster definition is requesting member in the network cluster.

38. The method of claim 37 wherein accessing the cluster definition includes:

determining a version number of the shared repository to yield a first version number;

reading the cluster definition;

re-determining a version number of the shared repository to yield a second version number;

comparing the first version number with the second version number; and

repeating the step of accessing the cluster definition until the first version number equals the second version number.

39. The method of claim 33 further comprises:

recovering from a failure of the coordinating node including selecting a new coordinator node from the member nodes of the network cluster, and completing, by the new coordinator node, an update of the cluster definition to reflect the requested change if there is a set valid bit and an incomplete log file in the shared repository.

40. The method of claim 39 wherein completing an update includes:

reading the incomplete log file; and

continuing the update of the cluster definition from a point, as indicated by the incomplete log file, where the coordinating node ceased updating the cluster definition due to the failure of the coordinating node.

41. An apparatus for maintaining a cluster definition for a network cluster of computing nodes having a plurality of member nodes and a network structure for facilitating network connectivity between nodes, comprising:

a shared repository coupled to at least two member nodes of the cluster over a storage connection separate from the network structure, the shared repository including the cluster definition and a proposed change to the cluster definition;

a coordinator node, selected from the at least two member nodes of the network cluster, to update the cluster definition with the proposed change; and

a potential member node to accessing the cluster definition on the shared repository over the storage connection, the potential member node being granted membership in the cluster regardless of network connectivity with another node.

42. The apparatus of claim 41 wherein a member node requests the proposed change to the cluster definition, and if after a period of time the proposed change is not made to the cluster definition, the member node re-requests the proposed change to the cluster definition.

43. The apparatus of claim 41 wherein the potential member node requests membership in the network cluster.

44. The apparatus of claim 43 wherein the potential member node further includes logic for:

determining a version number of the shared repository to yield a first version number;

reading the cluster definition;

re-determining a version number of the shared repository to yield a second version number;

comparing the first version number with the second version number; and

repeating the step of accessing the cluster definition until the first version number equals the second version number.

45. The apparatus of claim 41 further comprises:

a new coordinator node, selected from the at least two member nodes, to update the cluster definition if the coordinator node fails to operate.

46. A computer program product for maintaining a cluster definition for a network cluster of computing nodes having a plurality of member nodes and a network structure for facilitating network connectivity between nodes, the computer program product comprising:

a computer usable medium having computer readable program instructions thereon, including instructions for;

coupling at least two member nodes to a shared repository over a storage connection separate from the network structure;

storing a cluster definition for the network cluster in the shared repository;

selecting a coordinator node from the at least two member nodes of the network cluster;

requesting a change to the cluster definition by sending a proposed change to the shared repository over the storage connection;

directing the coordinator node to update the cluster definition to reflect the requested change; and

directing a potential member node to access the cluster definition on the shared repository over the storage connection, the potential member node being granted membership in the cluster regardless of network connectivity with another node.

47. The computer program product of claim 46 further comprising program instructions for:

directing the potential member node to request membership in the network cluster.

48. The computer program product of claim 47 further comprises program instructions for directing the potential member node to:

determine a version number of the shared repository to yield a first version number;

read the cluster definition;

re-determine a version number of the shared repository to yield a second version number;

compare the first version number with the second version number; and

access the cluster definition until the first version number equals the second version number.

49. The computer program product of claim 46 further comprising program instructions for:

recovering from a failure of the coordinating node.

50. A system for maintaining a cluster definition for a network cluster of computing nodes having a plurality of member nodes and a network structure for facilitating network connectivity between nodes, the system comprising:

a means for coupling at least two member nodes to a shared repository over a storage connection;

a means for storing a cluster definition for the network cluster in the shared repository;

a means for selecting a coordinator node from the at least two member nodes of the network cluster;

a means for requesting a change to the cluster definition by sending a proposed change to the shared repository over the storage connection;

a means for the coordinator node to update the cluster definition to reflect the requested change; and

a means for a potential member node to access the cluster definition on the shared repository over the storage connection, the potential member node being granted membership in the cluster regardless of network connectivity with another node.

51. A method for maintaining a cluster definition for a network cluster having at least one member node, the method comprising:

coupling the at least one member node to a shared repository over a storage connection separate from the network cluster;

storing a cluster definition for the network cluster in the shared repository;

selecting a coordinator node from the at least one member node of the network cluster;

accessing, by a member node, the cluster definition on the shared repository over the storage connection;

at a member node, requesting a change to the cluster definition by;

sending a proposed change to a scratch area of the shared repository over the storage connection;

setting a valid bit associated with the scratch area;

verifying the valid bit;

setting an update flag;

modifying the cluster definition to reflect the requested change; and

logging a progress of modifying the cluster definition in a log file in parallel with modifying the cluster definition;

incrementing a version number associated with the shared repository; and

clearing the valid bit and the update flag; and

from the coordinator node, updating the cluster definition to reflect the requested change.

52. A method for maintaining a cluster definition for a network cluster of computing nodes having at least one member node, the method comprising:

coupling the at least one member node to a shared repository over a storage connection separate from the network cluster;

storing a cluster definition for the network cluster in the shared repository;

selecting a coordinator node from the at least one member node of the network cluster;

accessing, by a member node, the cluster definition on the shared repository over the storage connection;

at a member node, requesting a change to the cluster definition by sending a proposed change to the shared repository over the storage connection;

from the coordinator node, updating the cluster definition to reflect the requested change;

requesting, by a potential member node, membership in the network cluster; and

accessing, by the potential member node, the cluster definition on the shared repository over the storage connection, for each potential member node accessing the cluster definition;

determining a version number of the shared repository to yield a first version number;

reading the cluster definition;

re-determining a version number of the shared repository to yield a second version number;

comparing the first version number with the second version number; and

repeating the step of accessing the cluster definition until the first version number equals the second version number.

53. A method of maintaining a cluster definition for a network cluster of computing nodes, the method comprising:

coupling each member node to a shared repository over a storage connection separate from the network cluster;

storing a current cluster definition for the network cluster at a single location in the shared repository;

directing a non-member node to access the cluster definition on the shared repository over the storage connection, the non-member node being granted membership in the network cluster regardless of network connectivity with another node;

selecting a coordinator node from one of the member nodes to update the cluster definition at the single location in the shared repository over the storage connection; and

determining, by each member node, the current cluster definition by accessing the updated, current cluster definition at the single location in the shared repository over the storage connection.

54. A method for updating a cluster definition for a network cluster of computing nodes having a plurality of member nodes, the method comprising:

coupling a shared repository to the at least two member nodes of the cluster over a storage connection separate from the network cluster, the shared repository including the cluster definition;

accessing, by a member node, the cluster definition on the shared repository over the storage connection; and

directing a non-member node to access the cluster definition on the shared repository over the storage connection before establishing connectivity with the network cluster, the non-member node being granted membership in the network cluster regardless of network connectivity with another node.

55. The method of claim 54 further including a member node requesting a change to the cluster definition by:

sending a proposed change to a scratch area; and

setting a valid bit associated with the scratch area.

56. The method of claim 55 further including updating, from a coordinator node, the cluster definition includes:

verifying the valid bit;

setting an update flag;

modifying the cluster definition to reflect the requested change by copying the proposed change from the scratch area to the cluster definition;

logging a progress of modifying the cluster definition in a log file in parallel with modifying the cluster definition;

incrementing a version number associated with the shared repository; and

clearing the valid bit and the update flag.

57. The method of claim 55 further including the step of:

re-requesting, by the member node, the change to the cluster definition if after a period of time, the change is not made to the cluster definition.

58. The method of claim 54 wherein the non-member node accessing the cluster definition is requesting member in the network cluster.

59. The method of claim 54 wherein accessing the cluster definition includes:

determining a version number of the shared repository to yield a first version number;

reading the cluster definition;

re-determining a version number of the shared repository to yield a second version number;

comparing the first version number with the second version number; and

repeating the step of accessing the cluster definition until the first version number equals the second version number.

60. The method of claim 54 further comprises recovering from a failure of a coordinating node by:

selecting a new coordinator node from the member nodes of the network cluster, and

completing, by the new coordinator node, an update of the cluster definition to reflect the requested change if there is a set valid bit and an incomplete log file in the shared repository.

61. The method of claim 60 wherein completing an update includes:

reading the incomplete log file; and

continuing the update of the cluster definition from a point, as indicated by the incomplete log file, where the coordinating node ceased updating the cluster definition due to the failure of the coordinating node.

62. An apparatus for maintaining a cluster definition for a network cluster of computing nodes having a plurality of member nodes, comprising:

a shared repository coupled to the at least two member nodes of the cluster over a storage connection separate from the network cluster, the shared repository including the cluster definition;

a member node accessing the cluster definition on the shared repository over the storage connection, the member node having membership in the cluster regardless of network connectivity with another node; and

a non-member node accessing the cluster definition on the shared repository over the storage connection before establishing connectivity with the network cluster.

63. The apparatus of claim 62 further includes a member node requesting a proposed change to the cluster definition, and if after a period of time the proposed change is not made to the cluster definition, the member node re-requests the proposed change to the cluster definition.

64. The apparatus of claim 62 wherein the non-member node requests membership in the network cluster.

65. The apparatus of claim 64 wherein the non-member node further includes logic for:

determining a version number of the shared repository to yield a first version number;

reading the cluster definition;

re-determining a version number of the shared repository to yield a second version number;

comparing the first version number with the second version number; and

repeating the step of accessing the cluster definition until the first version number equals the second version number.

66. The apparatus of claim 62 further comprises:

a new coordinator node, selected from the at least two member nodes, to update the cluster definition if the coordinator node fails to operate.

67. A system for maintaining a cluster definition for a network cluster of computing nodes having a plurality of member nodes, the system comprising:

means for coupling a shared repository to at least two member nodes of the cluster over a storage connection separate from the network cluster, the shared repository including the cluster definition;

means for accessing, by a member node, the cluster definition on the shared repository over the storage connection; and

means for directing a non-member node to access the cluster definition on the shared repository over the storage connection before establishing connectivity with the nodes of network cluster, the non-member node having access to the shared repository regardless of network connectivity with another node.