Maintaining high availability of a group of virtual machines using heartbeat messages

• US 8,924,967 B2
• Filed: 04/28/2011
• Issued: 12/30/2014
• Est. Priority Date: 04/28/2011
• Status: Active Grant

• Current Assignee: Vmware LLC (Broadcom, Inc.)
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: Nelson, Michael, Farkas, Keith, Ziskind, Elisha, Rajagopal, Sridhar, Shu, Guoqiang, Passerini, Ron, Ren, Joanne
• Primary Examiner(s): Bullock, Jr., Lewis A
• Assistant Examiner(s): ALFRED, MELISSA A
• Application Number:
  US13/096,188
• Publication Number:
  US 20120278801A1
• Time in Patent Office: 1,342 Days
• Field of Search: None
• US Class Current: 718/1
• CPC Class Codes: G06F 11/0757 by exceeding a time limit, i.e. time-out, e.g. watchdogs
• CPC Class Codes: G06F 11/1438 Restarting or rejuvenating
• CPC Class Codes: G06F 9/45533 Hypervisors; Virtual machine monitors
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications

Abstract

Embodiments maintain high availability of software application instances in a fault domain. Subordinate hosts are monitored by a master host. The subordinate hosts publish heartbeats via a network and datastores. Based at least in part on the published heartbeats, the master host determines the status of each subordinate host, distinguishing between subordinate hosts that are entirely inoperative and subordinate hosts that are operative but partitioned (e.g., unreachable via the network). The master host may restart software application instances, such as virtual machines, that are executed by inoperative subordinate hosts or that cease executing on partitioned subordinate hosts.

Claims

1. A system for maintaining high availability of a plurality of virtual machines in a fault domain, the system comprising:

a memory for storing a list identifying a plurality of virtual machines within a fault domain that are to be maintained in an operating state;

a network communication interface configured to receive heartbeat messages from a plurality of hosts executing the virtual machines; and

a processor coupled to the memory and programmed to;

associate a datastore with a host by ranking a plurality of datastores based on their accessibility and discarding a datastore with an accessibility score that is below a threshold value, wherein the accessibility score is based on at least two of the following factors;

     1) quantity of hosts that have access to the datastore,

     2) whether the datastore is associated with the same storage device as another datastore, and

     3) the file system type of the datastore;

monitor the heartbeat messages received from the plurality of hosts at predetermined intervals and determine the host, within the plurality of hosts, from which the network communication interface has not received a heartbeat message within a first predetermined duration to identify the host as an unreachable host;

after the unreachable host is identified, designate the unreachable host as a dead host when no response to a previously sent ping message is received from the unreachable host; and

if the dead host has stored heartbeat data in the associated datastore within a second predetermined duration, monitor a power-on list stored in the associated datastore by the dead host, wherein the power-on list indicates virtual machines being executed by the dead host, and change the status of the dead host.

2. The system of claim 1, wherein the network communication interface is further configured to receive from a management server a host list that includes a plurality of heartbeat datastores associated with the unreachable host, and the processor is further programmed to determine whether the unreachable host has stored heartbeat data in a datastore by determining whether the unreachable host has stored heartbeat data in any of the heartbeat datastores associated with the unreachable host.

3. The system of claim 1, wherein the network communication interface is further configured to receive the list from a management server, and the processor is further programmed to execute a fault domain manager software application as a master host.

4. The system of claim 1, wherein the processor is further programmed to:

identify a datastore that stores a virtual disk image associated with the virtual machine; and

restart the virtual machine based further on determining that a file stored in the identified datastore can be exclusively locked.

5. The system of claim 1, wherein the processor is further programmed to:

restart the virtual machine when the virtual machine does not appear in the power-on list.

6. The system of claim 1, further comprising a second host other than the unreachable host, wherein the processor is programmed to restart the virtual machine by storing execution command data in a datastore that is accessible by the second host, wherein the execution command data instructs the second host to execute the virtual machine.

20. The system of claim 1, wherein the processor is programmed to:

transmit a ping message to the unreachable host and designate the unreachable host as a dead host when no response ping message corresponding to the transmitted ping message is received from the unreachable host;

after the dead host is designated, determine whether the dead host has stored heartbeat data in the associated datastore within the second predetermined duration; and

if the dead host has not stored heartbeat data in the associated datastore within the second predetermined duration, restart a virtual machine executed by the dead host.

7. A method comprising:

associating a datastore with a subordinate host by ranking a plurality of datastores based on their accessibility and discarding a datastore with an accessibility score that is below a threshold value, wherein the accessibility score is based on at least two of the following factors;

     1) quantity of hosts that have access to the datastore,

     2) whether the datastore is associated with the same storage device as another datastore, and

     3) the file system type of the datastore;

storing, by the subordinate host, heartbeat data in the associated datastore, wherein the stored heartbeat data indicates that the subordinate host is operative at the current time;

storing, by the subordinate host, a power-on list in the associated datastore, wherein the power-on list indicates software application instances for a master host to restart at a host other than the subordinate host if the subordinate host becomes inoperative;

wherein heartbeat messages received from the subordinate host is monitored at predetermined intervals and whether a heartbeat message is received from the subordinate host within a first predetermined duration is determined to identify an unreachable host;

wherein after the unreachable host is identified, the unreachable host is designated as a dead host when no response to a previously sent ping message is received from the unreachable host; and

wherein the power-on list stored in the associated datastore by the dead host is monitored and the status of the dead host is changed based on determining that the dead host has stored heartbeat data in the associated datastore within a second predetermined duration.

8. The method of claim 7, wherein the datastore is a first datastore, the method further comprising:

storing, by the subordinate host, the heartbeat data in a second datastore; and

storing, by the subordinate host, the power-on list in the second datastore.

9. The method of claim 8, further comprising:

receiving, by the subordinate host, from a management server a host list that includes a plurality of heartbeat datastores associated with the subordinate host, wherein the heartbeat datastores include the first datastore and the second datastore; and

storing, by the subordinate host, the heartbeat data and the power-on list in the first and second datastores based on the host list.

10. The method of claim 7, wherein the subordinate host is a first host, the method further comprising:

receiving, by the subordinate host, from a management server a host list that includes a version number, a plurality of host identifiers, and one or more heartbeat datastores associated with each host identifier of the plurality of host identifiers;

transmitting the version number to a second host, wherein the second host transmits to the subordinate host a request for the host list when the transmitted version number is greater than a version number included in a host list stored by the second host; and

transmitting the host list from the subordinate host to the second host in response to the request.

11. The method of claim 7, further comprising:

detecting, by the subordinate host, that a software application instance executed by the subordinate host has terminated; and

removing, by the subordinate host, the software application instance from the power-on list in the datastore.

12. The method of claim 7, wherein the subordinate host is configured to communicate with the master host via a gateway, the method further comprising:

determining, by the subordinate host, that the subordinate host is unable to communicate with the gateway;

including, by the subordinate host, an isolation notification in the power-on list;

terminating the software application instances being executed by the subordinate host; and

removing, by the subordinate host, the terminated software application instances from the power-on list.

13. The method of claim 7, further comprising:

receiving, by the subordinate host, from a management server a host list that includes a plurality of hosts and identification information associated with each host of the plurality of hosts;

comparing, by the subordinate host, identification information received from the master host with the identification information associated with the master host in the host list; and

disregarding, by the subordinate host, messages from the master host for a predetermined duration when the identification information received from the master host does not match with the identification information associated with the master host in the host list.

14. One or more non-transitory computer-readable storage media having computer-executable components comprising:

a datastore selection component that when executed causes at least one processor to;

identify a plurality of datastores to which a subordinate host has read-write access;

select from the plurality of datastores a first heartbeat datastore and a second heartbeat datastore based on a quantity of hosts that have access to the first and second heartbeat datastores; and

associate the first and second heartbeat datastores with the host by ranking a plurality of datastores based on their accessibility and discarding a datastore with an accessibility score that is below a threshold value, wherein the accessibility score is based on at least two of the following factors;

     1) quantity of hosts that have access to the datastore,

     2) whether the datastore is associated with the same storage device as another datastore, and

     3) the file system type of the datastore;

a heartbeat publication component that when executed causes at least one processor to;

store in the first and second heartbeat datastores a heartbeat message indicating that the subordinate host is operative at the current time; and

an application instance monitoring component that when executed causes at least one processor to;

determine whether a heartbeat message is received from the subordinate host at the master host within a first predetermined duration to identify an unreachable host;

after the unreachable host is identified, designate the unreachable host as a dead host when no response to a previously sent ping message is received from the unreachable host; and

if the dead host has stored heartbeat data the heartbeat message in the first heartbeat datastore or in the second heartbeat datastore within a second predetermined duration, monitor a power-on list stored in the first heartbeat datastore or in the second heartbeat datastore by the dead host, wherein the power-on list indicates virtual machines being executed by the dead host, and change the status of the dead host.

15. The non-transitory computer readable storage media of claim 14, wherein each datastore of the plurality of datastores is associated with a storage device, and the datastore selection component causes the processor to select the first and second heartbeat datastores based further on the first and second heartbeat datastores being associated with different storage devices.

16. The non-transitory computer readable storage media of claim 14, wherein datastore selection component further causes the processor to:

receive a selection of preferred datastores; and

select the first and second heartbeat datastores based further on determining that the first and second datastores are included in the preferred datastores.

17. The non-transitory computer readable storage media of claim 14, further comprising a datastore monitoring component that when executed causes at least one processor to:

detect a newly available datastore; and

after a predetermined duration, execute the datastore selection component based on the detected newly available datastore.

18. The non-transitory computer readable storage media of claim 17, wherein the datastore monitoring component further causes the processor to:

detect an impending unavailability of the first datastore; and

execute the datastore selection component based on the detected impending unavailability.

19. A method comprising:

associating a datastore with a host by ranking a plurality of datastores based on their accessibility and discarding a datastore with an accessibility score that is below a threshold value, wherein the accessibility score is based on at least two of the following factors;

     1) quantity of hosts that have access to the datastore,

     2) whether the datastore is associated with the same storage device as another datastore, and

     3) the file system type of the datastore;

monitoring heartbeat messages received from the host at predetermined intervals and determining whether a heartbeat message is received from the host within a first predetermined duration to identify an unreachable host;

after the unreachable host is identified, designate the unreachable host as a dead host when no response to a previously sent ping message is received from the unreachable host; and

if the dead host has stored heartbeat data in the associated datastore within a second predetermined duration, monitoring a power-on list stored in the associated datastore by the dead host, wherein the power-on list indicates virtual machines being executed by the dead host, and change the status of the dead host.