METHOD AND ARRANGEMENT FOR OPERATING A COMPUTER CLUSTER

• US 20120290691A1
• Filed: 04/18/2012
• Published: 11/15/2012
• Est. Priority Date: 05/11/2011
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Buendgen, Reinhard
• Granted Patent: US 8,812,631 B2
• Time in Patent Office: Days
• US Class Current: 709/220
• CPC Class Codes: G06F 11/1425 by reconfiguration of node membership
• CPC Class Codes: G06F 11/202 where processing functionality is redundant redundant communication control functionality G06F11/2005, redundant storage control functionality G06F11/2089
• CPC Class Codes: G06F 2209/505 Clust
• CPC Class Codes: G06F 9/5061 Partitioning or combining of resources
• CPC Class Codes: H04L 41/0668 by dynamic selection of recovery network elements, e.g. replacement by the most appropriate element after failure
• CPC Class Codes: H04L 41/08 Configuration management of networks or network elements address allocation H04L61/50
• CPC Class Codes: H04L 41/0893 Assignment of logical groups to network elements
• CPC Class Codes: H04L 43/0811 by checking connectivity

Abstract

A method is disclosed for operating a computer cluster distributed over multiple locations using quorum mechanism. The cluster includes interconnected defined system instances. The method includes distributing the cluster configuration to each system instance, and constantly checking system instance connectivity to determine reachable system instances. Each healthy system instance determines a subcluster of itself and all reachable system instances. Information of the distributed cluster configuration is used to calculate a corresponding subcluster configuration. Subcluster configuration information and the cluster configuration information is used for computing a new quorum decision for the corresponding subcluster, using a location quorum function considering proportions of connected locations containing reachable system instances, and weights of the reachable system instances in the locations. Workload placement decisions are made based on the new quorum granting permission to place workload to system instances in a specific subcluster, or refusing the permission, whenever the subcluster changes.

Claims

1. A method for operating a computer cluster distributed over multiple locations using quorum mechanism, wherein said cluster (C) comprises a plurality of interconnected defined system instances running on one of a physical and a virtual hardware, and wherein a cluster configuration (CC) comprises location information about each system instance defined in said cluster (C), the method comprising:

distributing said cluster configuration (CC) to each defined system instance;

constantly checking connectivity among said system instances defined in said cluster (C) to determine reachable system instances, wherein each healthy system instance determines a subcluster (SCx) comprising itself and all reachable system instances said healthy system instance is connected to, using information of said distributed cluster configuration (CC);

calculating a corresponding subcluster configuration (SCC(SCx)) comprising location information about each system instance of said subcluster (SCx); and

using said subcluster configuration (SCC(SCx)) information and said cluster configuration (CC) information for computing a new quorum decision for said corresponding subcluster (SCx) said healthy system instance is member of, using a location quorum function (LQF) considering proportions of connected locations containing reachable system instances, and weights of said reachable system instances in said locations,

wherein workload placement decisions are made based on a computed new quorum granting permission to place workload to system instances in a specific subcluster (SCx), or refusing said permission, whenever said subcluster (SCx) changes.

2. The Method according to claim 1, wherein a majority based quorum function (MQF) considering a total number of connected healthy system instances (SCN(SCx)) and a total number of defined system instances (CN) in said cluster (C) is used in combination to said location quorum function (LQF) to calculate said new quorum for said corresponding subcluster (SCx).

3. The Method according to claim 2, wherein at least one of said location quorum function (LQF) and said majority based quorum function (MQF) comprise a configurable range of quorum tie situations.

4. The Method according to claim 2, wherein said location quorum function (LQF) is used to solve a tie situation of said majority based quorum function (MQF), and a tiebreaker function (TBF) is used to solve a tie situation of said location quorum function (LQF).

5. The Method according to claim 4, wherein at least one of said location quorum function (LQF) and said majority based quorum function (MQF) comprise a configurable range of quorum tie situations.

6. The Method of claim 2, wherein said distributed cluster configuration (CC) comprises at least one of the following:

said total number of defined system instances in said cluster (CN), said total number of locations (SN) in said cluster (C), a total number of defined system instances in a corresponding location (CN(s)), identification information of each system instance, first tie range information (rm) of said majority based quorum function (MQF), second tie range information (rl) of said location quorum function (LQF), and at least one tie time out value (T), wherein a first tie range of said majority based quorum function (MQF) is defined by said first tie range information (rm) which is configurable as (0≦

rm≦

0.5) such that said first tie range extends from (0.5−

rm) to (0.5+rm), and wherein a second tie range of said location quorum function (LQF) is defined by said second tie range information (rl) which is configurable as (0≦

rl≦

0.5) such that said second tie range extends from (0.5−

rl) to (0.5+rl).

7. The method of claim 2, wherein said majority based quorum function (MQF) calculates for each subcluster (SCx) a subcluster ratio (R(SCx)) based on a total number of healthy system instances (SCN(SCx)) in a corresponding subcluster (SCx) and said total number of defined system instances (CN) in said cluster (C), wherein said majority based quorum function (MQF) grants quorum to a corresponding subcluster (SCx) if said subcluster ratio (R(SCx)) is larger than a configurable third threshold (0.5+rm), and denies quorum, if said subcluster ratio (R(SCx)) is smaller than a configurable fourth threshold (0.5−

rm), else said majority based quorum function (MQF) solves an occurring tie situation using said location quorum function (LQF), wherein said majority based quorum function (MQF) denies quorum, if said tie situation is not solved within a configurable timeout period (T).

8. The method of claim 1, wherein said location quorum function (LQF) calculates for each location of said at least one subcluster (SCx) a first location ratio (W(sx,SCx)) based on a total number of healthy system instances (SCN(SCx,sx)) in said corresponding subcluster (SCx), and a total number of defined system instances (CN(sx)) in said corresponding location.

9. The method of claim 8, wherein said location quorum function (LQF) calculates a sum (W(SCx)) of all first location ratios (W(sx,SCx)) of said at least one subcluster (SCx), wherein said location quorum function (LQF) grants quorum to a corresponding subcluster (SC2) if a ratio based on said calculated sum (W(SC2)) and a total number of locations (SN) defined in said cluster (C) is larger than a configurable first threshold (0.5+rl), and denies quorum, if said ratio based on said calculated sum (W(SC2)) and said total number of locations (SN) defined in said cluster (C) is smaller than a configurable second threshold (0.5−

rl), else said location quorum function (LQF) solves an occurring tie situation using a tiebreaker function (TBF), wherein said location quorum function (LQF) denies quorum, if said tie situation is not solved within a configurable timeout period (T).

10. The method of claim 8, wherein said location quorum function (LQM) calculates for each location of said at least one subcluster (SCx) a second location ratio (w(sx,SCx)) based on said first location ratio (W(sx,SCx)), wherein said second location ratio (w(sx,SCx)) outputs a value of “

1”

, if a value of said first location ratio (W(sx,SCx)) is larger than a value of (0.5), or if said first location ratio (W(sx,SCN)) is equal to said value of (0.5) and a corresponding subcluster (SCx) has won a tiebreaker specific to said location, else said second location ratio (W(sx,SCx)) outputs a value of “

0”

, wherein said location quorum function (LQF) calculates a sum (w(SCx)) of all second location ratios (w(sx,SCx)) of said at least one subcluster (SCx), wherein said location quorum function (LQF) grants quorum to a corresponding subcluster (SC2) if a ratio based on said calculated sum (w(SC2)) and a total number of locations (SN) defined in said cluster (C) is larger than said configurable first threshold (0.5+rl), and denies quorum, if said ratio based on said calculated sum (w(SC2)) and said total number of locations (SN) defined in said cluster (C) is smaller than a configurable second threshold (0.5−

rl), else said location quorum function (LQF) solves an occurring tie situation using a tiebreaker function (TBF), wherein said location quorum function (LQF) denies quorum, if said tie situation is not solved within a configurable timeout period (T).

11. An apparatus for operating a computer cluster distributed over multiple locations using quorum mechanism, wherein said cluster (C) comprises a plurality of interconnected defined system instances running on a physical or virtual hardware, wherein each system instance in said cluster (C) runs a distributed cluster manager (P), wherein a cluster configuration (CC) comprises location information about each system instance defined in said cluster (C) and is distributed to each defined system instance, the apparatus comprising:

a membership service (M) that constantly checks connectivity among said system instances defined in said cluster (C) to determine reachable system instances, wherein said membership service (M) in each healthy system instance determines a subcluster (SCx) comprising said corresponding healthy system instance and all reachable system instances said healthy system instance is connected to, using information of said distributed cluster configuration (CC), and that calculates corresponding subcluster configuration (SCC(SCx)) comprising location information about each system instance of said subcluster (SCx);

a location aware quorum computation service (Q) in said healthy system instance that uses said subcluster configuration (SCC(SCx)) information and said cluster configuration (CC) information for computing a new quorum decision for said corresponding subcluster (SCx) said healthy system instance is member of, using a location quorum function (LQF) considering proportions of connected locations containing reachable system instances, and weights of said reachable system instances in said locations; and

a workload placement software service (P) in said healthy system instance that makes workload placement decisions based on a computed new quorum granting permission to place workload to system instances in a specific subcluster (SCx), or refusing said permission, whenever said subcluster (SCx) changes.

12. The arrangement according to claim 11, wherein said location aware quorum computation service (Q) uses a majority based quorum function (MQF) considering a total number of connected healthy system instances (SCN(SCx)) and a total number of defined system instances (CN) in said cluster (C) in combination to said location quorum function (LQF) to calculate said new quorum for said corresponding subcluster (SCx).

13. The arrangement according to claim 12, wherein said location aware quorum computation service (Q) uses said location quorum function (LQF) to solve a tie situation of said majority based quorum function (MQF), and a tiebreaker function (TBF) to solve a tie situation of said location quorum function (LQF), wherein at least one of said location quorum function (LQF) and said majority based quorum function (MQF) comprise a configurable range of quorum tie situations.

14. The arrangement according to claim 12, wherein said distributed cluster configuration (CC) comprises at least one of the following:

said total number of defined system instances in said cluster (CN), said total number of locations (SN) in said cluster (C), a total number of defined system instances in a corresponding location (CN(s)), identification information of each system instance, a first tie range information (rm) of said majority based quorum function (MQF), a second tie range information (rl) of said location quorum function (LQF), and at least one tie time out value (T)), wherein a first tie range of said majority based quorum function (MQF) is defined by said first tie range information (rm) which is configurable as (0≦

rm≦

0.5) such that said first tie range extends from (0.5−

rm) to (0.5+rm), and wherein a second tie range of said location quorum function (LQF) is defined by said second tie range information (rl) which is configurable as (0≦

rl≦

0.5) such that said second tie range extends from (0.5−

rl) to (0.5+rl).

15. A computer program product for operating a computer cluster distributed over multiple locations using quorum mechanism, wherein said cluster (C) comprises a plurality of interconnected defined system instances running on one or a physical and a virtual hardware, and wherein a cluster configuration (CC) comprises location information about each system instance defined in said cluster (C), the computer program product comprising a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code configured to:

distribute said cluster configuration (CC) to each defined system instance;

constantly check connectivity among said system instances defined in said cluster (C) to determine reachable system instances, wherein each healthy system instance determines a subcluster (SCx) comprising itself and all reachable system instances said healthy system instance is connected to, using information of said distributed cluster configuration (CC);

calculate a corresponding subcluster configuration (SCC(SCx)) comprising location information about each system instance of said subcluster (SCx); and

use said subcluster configuration (SCC(SCx)) information and said cluster configuration (CC) information for computing a new quorum decision for said corresponding subcluster (SCx) said healthy system instance is member of, using a location quorum function (LQF) considering proportions of connected locations containing reachable system instances, and weights of said reachable system instances in said locations,

wherein workload placement decisions are made based on a computed new quorum granting permission to place workload to system instances in a specific subcluster (SCx), or refusing said permission, whenever said subcluster (SCx) changes.

16. The computer program product of claim 15, wherein a majority based quorum function (MQF) considering a total number of connected healthy system instances (SCN(SCx)) and a total number of defined system instances (CN) in said cluster (C) is used in combination to said location quorum function (LQF) to calculate said new quorum for said corresponding subcluster (SCx).

17. The computer program product of claim 16, wherein at least one of said location quorum function (LQF) and said majority based quorum function (MQF) comprise a configurable range of quorum tie situations.

18. The computer program product of claim 16, wherein said location quorum function (LQF) is used to solve a tie situation of said majority based quorum function (MQF), and a tiebreaker function (TBF) is used to solve a tie situation of said location quorum function (LQF).

19. The computer program product of claim 16, wherein said distributed cluster configuration (CC) comprises at least one of the following said total number of defined system instances in said cluster (CN), a total number of locations (SN) in said cluster (C), a total number of defined system instances in a corresponding location (CN(s)), identification information of each system instance, first tie range information (rm) of said majority based quorum function (MQF), second tie range information (rl) of said location quorum function (LQF), and at least one tie time out value (T), wherein a first tie range of said majority based quorum function (MQF) is defined by said first tie range information (rm) which is configurable as (0≦

rm≦

0.5) such that said first tie range extends from (0.5−

rm) to (0.5+rm), and wherein a second tie range of said location quorum function (LQF) is defined by said second tie range information (rl) which is configurable as (0≦

rl≦

0.5) such that said second tie range extends from (0.5−

rl) to (0.5+rl).

20. The computer program product of claim 15, wherein said location quorum function (LQF) calculates for each location of said at least one subcluster (SCx) a first location ratio (W(sx,SCx)) based on a total number of healthy system instances (SCN(SCx,sx)) in said corresponding subcluster (SCx), and a total number of defined system instances (CN(sx)) in a corresponding location.