Arrangement for operating a computer cluster

• US 8,812,632 B2
• Filed: 02/27/2013
• Issued: 08/19/2014
• Est. Priority Date: 05/11/2011
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Buendgen, Reinhard
• Primary Examiner(s): Chan, Wing F
• Assistant Examiner(s): MIAH, RAZU A
• Application Number:
  US13/779,495
• Publication Number:
  US 20130173763A1
• Time in Patent Office: 538 Days
• Field of Search: 709/220, 370218-354
• US Class Current: 709/220
• CPC Class Codes: G06F 11/1425 by reconfiguration of node membership
• CPC Class Codes: G06F 11/202 where processing functionality is redundant redundant communication control functionality G06F11/2005, redundant storage control functionality G06F11/2089
• CPC Class Codes: G06F 2209/505 Clust
• CPC Class Codes: G06F 9/5061 Partitioning or combining of resources
• CPC Class Codes: H04L 41/0668 by dynamic selection of recovery network elements, e.g. replacement by the most appropriate element after failure
• CPC Class Codes: H04L 41/08 Configuration management of networks or network elements address allocation H04L61/50
• CPC Class Codes: H04L 41/0893 Assignment of logical groups to network elements
• CPC Class Codes: H04L 43/0811 by checking connectivity

Abstract

An arrangement is disclosed for operating a computer cluster distributed over multiple locations using quorum mechanism. The cluster includes interconnected defined system instances. The arrangement includes distributing the cluster configuration to each system instance, and constantly checking system instance connectivity to determine reachable system instances. Each healthy system instance determines a subcluster of itself and all reachable system instances. Information of the distributed cluster configuration is used to calculate a corresponding subcluster configuration. Subcluster configuration information and the cluster configuration information is used for computing a new quorum decision for the corresponding subcluster, using a location quorum function considering proportions of connected locations containing reachable system instances, and weights of the reachable system instances in the locations. Workload placement decisions are made based on the new quorum granting permission to place workload to system instances in a specific subcluster, or refusing the permission, whenever the subcluster changes.

Claims

1. An apparatus for operating a computer cluster distributed over multiple locations using quorum mechanism, wherein said cluster (C) comprises a plurality of interconnected defined system instances running on a physical or virtual hardware, wherein each system instance in said cluster (C) runs a distributed cluster manager (P), wherein a cluster configuration (CC) comprises location information about each system instance defined in said cluster (C) and is distributed to each defined system instance, the apparatus comprising one or more of hardware and software, the software stored on one or more non-transitory computer readable media, the apparatus comprising:

a membership service (M) that constantly checks connectivity among said system instances defined in said cluster (C) to determine reachable system instances, wherein said membership service (M) in each healthy system instance determines a subcluster (SCx) comprising said corresponding healthy system instance and all reachable system instances said healthy system instance is connected to, using information of said distributed cluster configuration (CC), and that calculates corresponding subcluster configuration (SCC(SCx)) comprising location information about each system instance of said subcluster (SCx);

a location aware quorum computation service (Q) in said healthy system instance that uses said subcluster configuration (SCC(SCx)) information and said cluster configuration (CC) information for computing a new quorum decision for said corresponding subcluster (SCx) said healthy system instance is member of, using a location quorum function (LQF) considering proportions of connected locations containing reachable system instances, and weights of said reachable system instances in said locations; and

a workload placement software service (P) in said healthy system instance that makes workload placement decisions based on a computed new quorum granting permission to place workload to system instances in a specific subcluster (SCx), or refusing said permission, whenever said subcluster (SCx) changes.

2. The apparatus according to claim 1, wherein said location aware quorum computation service (Q) uses a majority based quorum function (MQF) considering a total number of connected healthy system instances (SCN(SCx)) and a total number of defined system instances (CN) in said cluster (C) in combination to said location quorum function (LQF) to calculate said new computed quorum for said corresponding subcluster (SCx).

3. The apparatus according to claim 2, wherein said location aware quorum computation service (Q) uses said location quorum function (LQF) to solve a tie situation of said majority based quorum function (MQF), and a tiebreaker function (TBF) to solve a tie situation of said location quorum function (LQF).

4. The apparatus according to claim 3, wherein at least one of said location quorum function (LQF) and said majority based quorum function (MQF) comprise a configurable range of quorum tie situations.

5. The apparatus according to claim 2, wherein said distributed cluster configuration (CC) comprises at least one of the following:

said total number of defined system instances in said cluster (CN), said total number of locations (SN) in said cluster (C), a total number of defined system instances in a corresponding location (CN(s)), identification information of each system instance, a first tie range information (rm) of said majority based quorum function (MQF), a second tie range information (rl) of said location quorum function (LQF), and at least one tie time out value (T)), wherein a first tie range of said majority based quorum function (MQF) is defined by said first tie range information (rm) which is configurable as (0≦

rm≦

0.5) such that said first tie range extends from (0.5−

rm) to (0.5+rm), and wherein a second tie range of said location quorum function (LQF) is defined by said second tie range information (rl) which is configurable as (0≦

rl≦

0.5) such that said second tie range extends from (0.5−

rl) to (0.5+rl).

6. The apparatus according to claim 2, wherein said majority based quorum function (MQF) calculates for each subcluster (SCx) a subcluster ratio (R(SCx)) based on a total number of healthy system instances (SCN(SCx)) in a corresponding subcluster (SCx) and said total number of defined system instances (CN) in said cluster (C), wherein said majority based quorum function (MQF) grants quorum to a corresponding subcluster (SCx) if said subcluster ratio (R(SCx)) is larger than a configurable third threshold (0.5+rm), and denies quorum, if said subcluster ratio (R(SCx)) is smaller than a configurable fourth threshold (0.5−

m), else said majority based quorum function (MQF) solves an occurring tie situation using said location quorum function (LQF), wherein said majority based quorum function (MQF) denies quorum, if said tie situation is not solved within a configurable timeout period (T).

7. The apparatus according to claim 1, wherein said location quorum function (LQF) calculates for each location of said at least one subcluster (SCx) a first location ratio (W(sx,SCx)) based on a total number of healthy system instances (SCN(SCx,sx)) in said corresponding subcluster (SCx), and a total number of defined system instances (CN(sx)) in said corresponding location.

8. The apparatus according to claim 7, wherein said location quorum function (LQF) calculates a sum (W(SCx)) of all first location ratios (W(sx,SCx)) of said at least one subcluster (SCx), wherein said location quorum function (LQF) grants quorum to a corresponding subcluster (SC2) if a ratio based on said calculated sum (W(SC2)) and a total number of locations (SN) defined in said cluster (C) is larger than a configurable first threshold (0.5+rl), and denies quorum, if said ratio based on said calculated sum (W(SC2)) and said total number of locations (SN) defined in said cluster (C) is smaller than a configurable second threshold (0.5−

rl), else said location quorum function (LQF) solves an occurring tie situation using a tiebreaker function (TBF), wherein said location quorum function (LQF) denies quorum, if said tie situation is not solved within a configurable timeout period (T).

9. The apparatus according to claim 7, wherein said location quorum function (LQM) calculates for each location of said at least one subcluster (SCx) a second location ratio (w(sx,SCx)) based on said first location ratio (W(sx,SCx)), wherein said second location ratio (w(sx,SCx)) outputs a value of “

1”

, if a value of said first location ratio (W(sx,SCx)) is larger than a value of (0.5), or if said first location ratio (W(sx,SCN)) is equal to said value of (0.5) and a corresponding subcluster (SCx) has won a tiebreaker specific to said location, else said second location ratio (W(sx,SCx)) outputs a value of “

0”

, wherein said location quorum function (LQF) calculates a sum (w(SCx)) of all second location ratios (w(sx,SCx)) of said at least one subcluster (SCx), wherein said location quorum function (LQF) grants quorum to a corresponding subcluster (SC2) if a ratio based on said calculated sum (w(SC2)) and a total number of locations (SN) defined in said cluster (C) is larger than said configurable first threshold (0.5+rl), and denies quorum, if said ratio based on said calculated sum (w(SC2)) and said total number of locations (SN) defined in said cluster (C) is smaller than a configurable second threshold (0.5−

rl), else said location quorum function (LQF) solves an occurring tie situation using a tiebreaker function (TBF), wherein said location quorum function (LQF) denies quorum, if said tie situation is not solved within a configurable timeout period (T).