Using location tracking of cluster nodes to avoid single points of failure

• US 9,454,444 B1
• Filed: 03/19/2009
• Issued: 09/27/2016
• Est. Priority Date: 03/19/2009
• Status: Active Grant

• Current Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Original Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Inventors: Agarwal, Sandeep, Kong, Chio Fai Aglaia, Ramamurthy, Karthik
• Primary Examiner(s): Vicary, Keith
• Application Number:
  US12/407,237
• Time in Patent Office: 2,749 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1425 by reconfiguration of node membership
• CPC Class Codes: G06F 11/1484 involving virtual machines
• CPC Class Codes: G06F 11/20 using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
• CPC Class Codes: G06F 11/2023 Failover techniques

Abstract

Systems and methods are provided to track cluster nodes and provide high availability in a computing system. A computer system includes hosts, a cluster manager, and a cluster database. The cluster database includes entries corresponding to the hosts which identify the physical location of a corresponding host. The cluster manager uses the data to select at least two hosts and assign the selected hosts to a service group for executing an application. The cluster manager selects hosts via an algorithm that determines which hosts are least likely to share a single point of failure. The data includes a hierarchical group of location attributes describing two or more of a host'"'"'s country, state, city, building, room, enclosure, and radio frequency identifier (RFID). The location-based algorithm identifies a group of selected hosts whose smallest shared location attribute is highest in the hierarchical group. The system updates the data whenever a physical location of a host changes.

Claims

1. A computer system comprising:

a plurality of hosts;

a cluster manager; and

a cluster database, wherein the cluster database comprises a plurality of entries corresponding to the plurality of hosts, each entry including a service group identifier (ID) identifying a service group of which a corresponding host is a member and physical location data identifying a physical location of the corresponding host;

wherein in response to receiving a reconfiguration request, the cluster manager is configured to;

retrieve from the cluster database an entry corresponding to a second host of the plurality of hosts having a same service group ID as a first host of the plurality of hosts; and

substitute a third host corresponding to a service group different from a service group of the first host for either the first host or the second host, in response to determining physical location data for the first host and the second host both identify a same enclosure, said enclosure comprising one of a group of racks, a rack, and a shelf on a rack.

2. The system as recited in claim 1, wherein prior to substituting the third host, the cluster manager is configured to:

retrieve for the third host a corresponding entry in the plurality of entries with a service group ID different from a service group ID of the first host, wherein the third host is a candidate for membership in a service group identified by the service group ID of the first host; and

determine the physical location data for the third host does not identify said enclosure.

3. The system as recited in claim 1, wherein said physical location data identifying a physical location of the corresponding host comprises a hierarchical group of two or more location attributes, wherein each successive attribute is a smaller subdivision of a previous attribute, and wherein the hierarchical group comprises two or more of a country, a state, a city, a building, a room, and one of a group of racks, a rack, and a shelf of a rack.

4. The system as recited in claim 3, wherein the cluster manager is further configured to identify a group of selected hosts whose smallest shared location attribute is highest in the hierarchical group.

5. The system as recited in claim 1,

wherein one or more of the plurality of hosts comprise virtual hosts in a virtualized environment;

wherein reassignment of a virtual host during host operation leads to a dynamic change in a physical location of a corresponding host of the plurality of hosts; and

wherein in response to detecting that a physical location of a host of the plurality of hosts changed, the cluster manager is further configured to update corresponding physical location data in the cluster database.

6. The system as recited in claim 1, wherein the first host is configured to execute at least a portion of an application and the second host is configured to execute the at least a portion of the application in response to an indication that the first host has failed.

7. The system as recited in claim 1, wherein in response to detecting a request for selecting a failover host for the first host, the cluster manager is further configured to:

retrieve for at least two candidate hosts of the plurality of hosts corresponding entries in the plurality of entries;

retrieve for each other host of the plurality of hosts in the same service group as the first host corresponding entries in the plurality of entries;

determine whether physical location data in the corresponding entries for the at least two candidate hosts and physical location data in the corresponding entries for said each other host identify a same enclosure; and

select the failover host for the first host from the at least two candidate hosts such that the selected failover host for the first host has a greater geographical separation with the first host than other hosts of the at least two candidate hosts as indicated by the retrieved physical location data.

8. A computer-implemented method comprising:

a computing process accessing a cluster database comprising a plurality of entries corresponding to a plurality of hosts, each entry including a service group identifier (ID) identifying a service group of which a corresponding host of the plurality of hosts is a member and physical location data identifying a physical location of the corresponding host;

in response to receiving a reconfiguration request;

retrieving from the cluster database an entry corresponding to a second host of the plurality of hosts having a same service group ID as a first host of the plurality of hosts; and

substituting a third host corresponding to a service group different from a service group of the first host for either the first host or the second host in response to determining physical location data for the first host and the second host both identify a same enclosure, said enclosure comprising one of a group of racks, a rack, and a shelf on a rack.

9. The method as recited in claim 8, wherein prior to substituting the third host, the method further comprises:

retrieving for the third host a corresponding entry in the plurality of entries with a service group ID different from a service group ID of the first host, wherein the third host is a candidate for membership in a service group identified by the service group ID of the first host; and

determining the physical location data for the third host does not identify said enclosure.

10. The method as recited in claim 8, wherein said physical location data identifying a physical location of the corresponding host comprises a hierarchical group of two or more location attributes, wherein each successive attribute is a smaller subdivision of a previous attribute, and wherein the hierarchical group comprises two or more of a country, a state, a city, a building, a room, and one of a group of racks, a rack, and a shelf of a rack.

11. The method as recited in claim 10, wherein the method further comprises identifying a group of selected hosts whose smallest shared location attribute is highest in the hierarchical group.

12. The method as recited in claim 8,

wherein one or more of the plurality of hosts comprise virtual hosts in a virtualized environment;

wherein reassignment of a virtual host during host operation leads to a dynamic change in a physical location of a corresponding host of the plurality of hosts; and

wherein in response to detecting that a physical location of a host of the plurality of hosts changed, the method further comprises updating corresponding physical location data in the cluster database.

13. The method as recited in claim 8, the method further comprising:

the first host executing at least a portion of an application; and

the second host executing the at least a portion of the application in response to an indication that the first host has failed.

14. The method as recited in claim 8, wherein in response to detecting a request for selecting a failover host for the first host, the method further comprises:

retrieving for at least two candidate hosts of the plurality of hosts corresponding entries in the plurality of entries;

retrieving for each other host of the plurality of hosts in the same service group as the first host corresponding entries in the plurality of entries;

determining whether the physical location data in the corresponding entries for the at least two candidate hosts and physical location data in the corresponding entries for said each other host identify a same enclosure; and

selecting the failover host for the first host from the at least two candidate hosts such that the selected failover host for the first host has a greater geographical separation with the first host than other hosts of the at least two candidate hosts as indicated by the retrieved physical location data.

15. A non-transitory computer readable medium storing computer instructions that are executable by a processor to:

access a cluster database comprising a plurality of entries corresponding to a plurality of hosts, each entry including a service group identifier (ID) identifying a service group of which a corresponding host of the plurality of hosts is a member and physical location data identifying a physical location of the corresponding host;

in response to receiving a reconfiguration request;

retrieve from the cluster database an entry corresponding to a second host of the plurality of hosts having a same service group ID as a first host of the plurality of hosts; and

substitute a third host corresponding to a service group different from a service group of the first host for either the first host or the second host in response to determining physical location data for the first host and the second host both identify a same enclosure, said enclosure comprising one of a group of racks, a rack, and a shelf on a rack.

16. The computer readable medium as recited in claim 15, wherein prior to substituting the third host, the instructions are further executable to:

retrieve for the third host a corresponding entry in the plurality of entries with a service group ID different from a service group ID of the first host, wherein the third host is a candidate for membership in a service group identified by the service group ID of the first host; and

determine the physical location data for the third host does not identify said enclosure.

17. The computer readable medium as recited in claim 15, wherein said physical location data identifying a physical location of the corresponding host comprises a hierarchical group of two or more location attributes, wherein each successive attribute is a smaller subdivision of a previous attribute, and wherein the hierarchical group comprises two or more of a country, a state, a city, a building, a room, and one of a group of racks, a rack, and a shelf of a rack.

18. The computer readable medium as recited in claim 17, wherein the instructions are further executable to identify a group of selected hosts whose smallest shared location attribute is highest in the hierarchical group.

19. The computer readable medium as recited in claim 15,

wherein one or more of the plurality of hosts comprise virtual hosts in a virtualized environment;

wherein reassignment of a virtual host during host operation leads to a dynamic change in a physical location of a corresponding host of the plurality of hosts; and

wherein in response to detecting that a physical location of a host of the plurality of hosts changed, the instructions are further executable to update corresponding physical location data in the cluster database.

20. The computer readable medium as recited in claim 15, wherein the instructions are further executable to cause:

the first host to execute at least a portion of an application; and

the second host to execute the at least a portion of the application in response to an indication that the first host has failed.