System and method for using cluster level quorum to prevent split brain scenario in a data grid cluster

• US 9,063,787 B2
• Filed: 01/17/2012
• Issued: 06/23/2015
• Est. Priority Date: 01/28/2011
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Lee, Robert H., Falco, Mark, Gleyzer, Gene, Purdy, Cameron
• Primary Examiner(s): Gaffin, Jeffrey A
• Assistant Examiner(s): Traktovenko, Ilya
• Application Number:
  US13/352,203
• Publication Number:
  US 20120197822A1
• Time in Patent Office: 1,253 Days
• Field of Search: 706/10, 706/62
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1425 by reconfiguration of node membership
• CPC Class Codes: G06F 2209/505 Clust
• CPC Class Codes: G06F 9/5061 Partitioning or combining of resources
• CPC Class Codes: H04L 41/5054 Automatic deployment of services triggered by the service manager, e.g. service implementation by automatic configuration of network components
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/51 Discovery or management thereof, e.g. service location protocol [SLP] or web services

Abstract

A system and method is described for use with a data grid cluster, which uses cluster quorum to prevent split brain scenario. The data grid cluster includes a plurality of cluster nodes, each of which runs a cluster service. Each cluster service collects and maintains statistics regarding communication flow between its cluster node and the other cluster nodes in the data grid cluster. The statistics are used to determine a status associated with other cluster nodes in the data grid cluster whenever a disconnect event happens. The data grid cluster is associated with a quorum policy, which is defined in a cache configuration file, and which specifies a time period that a cluster node will wait before making a decision on whether or not to evict one or more cluster nodes from the data grid cluster.

Claims

1. A system for using cluster quorum to prevent split brain scenario in a distributed data grid cluster, comprising:

a data grid cluster comprising a plurality of cluster nodes wherein each cluster node comprises a microprocessor;

a plurality of cluster services, wherein each cluster service runs on a particular cluster node of said plurality of cluster nodes in the data grid cluster and collects and maintains statistics regarding communication flow between the particular cluster node and other cluster nodes in the data grid cluster, and wherein the statistics are used by the data grid cluster to determine a status associated with each other cluster node in the data grid cluster when a disconnection event happens in the data grid cluster;

a cluster quorum policy defined in a cache configuration file associated with the data grid cluster, wherein the cluster quorum policy specifies a minimum number of cluster nodes required to permit a decision whether to evict one or more cluster nodes from the data grid cluster, and a time period that the data grid cluster defers to make a decision on whether or not to evict one or more cluster nodes from the data grid cluster; and

wherein, when one or more cluster nodes are detected to have been disconnected (disconnected nodes) from the data grid cluster as a results of a disconnection event,

the data grid cluster defers to make a decision on whether or not to evict the disconnected nodes from the data grid cluster for the time period specified in the cluster quorum policy,

if a connection is reestablished to the disconnected nodes prior to expiration of the time period specified in the cluster quorum policy, the data grid cluster does not evict the disconnected nodes; and

if the connection is not reestablished to the disconnected nodes prior to expiration of the time period specified in the cluster quorum policy and a number of cluster nodes remaining in the cluster after excluding the disconnected nodes is equal to at least the minimum number of cluster nodes required to permit a decision whether to evict one or more cluster nodes from the data grid cluster, the data grid cluster does evict the disconnected nodes.

2. The system according to claim 1, wherein:

each cluster service is a root cluster service, and wherein there is exactly one root cluster service running on each cluster node in the data grid cluster.

3. The system according to claim 1, wherein:

the status associated with each other cluster node in the data grid cluster is one of

a node that is definitely dead;

a node that is definitely alive; and

a node that is in question.

4. The system according to claim 1, wherein:

the data grid cluster automatically recovers from the disconnection event to provide services to users.

5. The system according to claim 1, wherein:

if the connection is not reestablished to the disconnected nodes prior to expiration of the time period specified in the cluster quorum policy and a number of cluster nodes remaining in the cluster after excluding the disconnected nodes is less than the minimum number of cluster nodes required to permit a decision whether to evict one or more cluster nodes from the data grid cluster, the cluster quorum policy can specify that user input is required to evict the disconnected nodes.

6. The system according to claim 1, further comprising:

the data grid cluster supports a pluggable interface that allows users to implement custom policies to configure the data grid cluster.

7. The system according to claim 6, wherein:

the cluster quorum policy defined in the cache management file can refer to and/or include in custom policies managed by a user application, wherein the custom policies in the user application can implement the pluggable interface supported by the data grid cluster.

8. The system according to claim 7, wherein:

the custom policies incorporate in one or more external states managed by the user application.

9. The system according to claim 8, wherein:

at least one external state is deployment topology information that describes user application deployment topology in the data grid cluster.

10. The system according to claim 1, wherein:

a cluster node in the data grid cluster is considered becoming a suspected cluster member when it has not responded to network communications and/or is in imminent danger of being disconnected from the cluster.

11. A method for using cluster level quorum in a data grid cluster comprising a plurality of cluster nodes, the method comprising:

running a cluster service on each cluster node of the plurality of clusters nodes in the data grid cluster;

collecting and maintaining, via the cluster service running on each particular cluster node, statistics regarding communication flow between the particular cluster node and other cluster nodes in the data grid cluster;

determining a status associated with each cluster node in the data grid cluster when a disconnection event happens;

providing a cluster quorum policy defined in a cache configuration file associated with the data grid cluster, wherein the cluster quorum policy specifies a minimum number of cluster nodes required to permit a decision whether to evict one or more cluster nodes from the data grid cluster, and a time period that the data grid cluster defers to make a decision on whether or not to evict one or more cluster nodes from the data grid cluster; and

in response to detecting one or more cluster nodes to have been disconnected (disconnected nodes) from the data grid cluster as a results of a disconnection event,

deferring to make a decision for the time period specified in the cluster quorum policy on whether or not to evict the disconnected nodes from the data grid cluster based on the cluster quorum policy,

if a connection is reestablished to the disconnected nodes prior to expiration of the time period specified in the cluster quorum policy, determining not to evict the disconnected nodes; and

if the connection is not reestablished to the disconnected nodes prior to expiration of the time period specified in the cluster quorum policy and a number of cluster nodes remaining in the cluster after excluding the disconnected nodes is equal to at least the minimum number of cluster nodes required to permit a decision whether to evict one or more cluster nodes from the data grid cluster, the data grid cluster does evict the disconnected nodes.

12. The method of claim 11, further comprising:

running each cluster service as a root cluster service, wherein there is exactly one root cluster service running on each cluster node in the data grid cluster.

13. The method of claim 11, wherein the status associated with each other cluster node in the data grid cluster is one of:

a node that is definitely dead;

a node that is definitely alive; and

a node that is in question;

the data grid cluster automatically recovers from the disconnection event to provide services to users.

14. The method of claim 11, further comprising:

requiring user input to evict the disconnected nodes if the connection is not reestablished to the disconnected nodes prior to expiration of the time period specified in the cluster quorum policy and a number of cluster nodes remaining in the cluster after excluding the disconnected nodes is less than the minimum number of cluster nodes required to permit a decision whether to evict one or more cluster nodes from the data grid cluster.

15. The method of claim 11, further comprising:

providing a pluggable interface that allows users to implement custom policies to configure the data grid cluster.

16. The method of claim 11, further comprising:

providing a pluggable interface that allows users to implement a custom policy to configure the data grid cluster; and

referring to said custom policy from the cluster quorum policy.

17. The method of claim 11, further comprising:

providing a pluggable interface that allows users to implement a custom policy to configure the data grid cluster wherein the custom policy incorporates one or more external states managed by a user application; and

referring to said custom policy from the cluster quorum policy.

18. The method of claim 11, further comprising:

providing a pluggable interface that allows users to implement a custom policy to configure the data grid cluster wherein the custom policy incorporates deployment topology information that describes user application deployment topology in the data grid cluster; and

referring to said custom policy from the cluster quorum policy.

19. A non-transitory machine readable medium having instructions stored thereon for using cluster quorum to prevent split brain scenario in a distributed data grid cluster comprising a plurality of cluster nodes, which instructions, when executed by a system, cause the system to perform steps comprising:

running a cluster service on each cluster node of the plurality of clusters nodes in the data grid cluster;

collecting and maintaining, via the cluster service running on each particular cluster node, statistics regarding communication flow between the particular cluster node and other cluster nodes in the data grid cluster;

determining a status associated with each cluster node in the data grid cluster when a disconnection event happens;

providing a cluster quorum policy defined in a cache configuration file associated with the data grid cluster, wherein the cluster quorum policy specifies a minimum number of cluster nodes required to permit a decision whether to evict one or more cluster nodes from the data grid cluster, and a time period that the data grid cluster defers to make a decision on whether or not to evict one or more cluster nodes from the data grid cluster; and

in response to detecting one or more cluster nodes to have been disconnected (disconnected nodes) from the data grid cluster as a results of a disconnection event,

deferring to make a decision for the time period specified in the cluster quorum policy on whether or not to evict the disconnected nodes from the data grid cluster based on the cluster quorum policy,

if a connection is reestablished to the disconnected nodes prior to expiration of the time period specified in the cluster quorum policy, determining not to evict the disconnected nodes; and

if the connection is not reestablished to the disconnected nodes prior to expiration of the time period specified in the cluster quorum policy and a number of cluster nodes remaining in the cluster after excluding the disconnected nodes is equal to at least the minimum number of cluster nodes required to permit a decision whether to evict one or more cluster nodes from the data grid cluster, the data grid cluster does evict the disconnected nodes.

20. The non-transitory machine readable medium of claim 19, having further instructions stored thereon for using cluster quorum to prevent split brain scenario in a distributed data grid cluster comprising a plurality of cluster nodes, which further instructions, when executed by a system, cause the system to perform steps comprising:

requiring user input to evict the disconnected nodes if the connection is not reestablished to the disconnected nodes prior to expiration of the time period specified in the cluster quorum policy and a number of cluster nodes remaining in the cluster after excluding the disconnected nodes is less than the minimum number of cluster nodes required to permit a decision whether to evict one or more cluster nodes from the data grid cluster.