MAINTAINING HIGH AVAILABILITY OF A GROUP OF VIRTUAL MACHINES USING HEARTBEAT MESSAGES

• US 20120278801A1
• Filed: 04/28/2011
• Published: 11/01/2012
• Est. Priority Date: 04/28/2011
• Status: Active Grant

• Current Assignee: Vmware LLC (Broadcom, Inc.)
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: NELSON, Michael, Farkas, Keith, Rajagopal, Sridhar, Ziskind, Elisha, Shu, Guoqiang, Passerini, Ron, Ren, Joanne
• Granted Patent: US 8,924,967 B2
• Time in Patent Office: Days
• US Class Current: 718/1
• CPC Class Codes: G06F 11/0757 by exceeding a time limit, i.e. time-out, e.g. watchdogs
• CPC Class Codes: G06F 11/1438 Restarting or rejuvenating
• CPC Class Codes: G06F 9/45533 Hypervisors; Virtual machine monitors
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications

Abstract

Embodiments maintain high availability of software application instances in a fault domain. Subordinate hosts are monitored by a master host. The subordinate hosts publish heartbeats via a network and datastores. Based at least in part on the published heartbeats, the master host determines the status of each subordinate host, distinguishing between subordinate hosts that are entirely inoperative and subordinate hosts that are operative but partitioned (e.g., unreachable via the network). The master host may restart software application instances, such as virtual machines, that are executed by inoperative subordinate hosts or that cease executing on partitioned subordinate hosts.

Claims

1. A system for maintaining high availability of a plurality of virtual machines in a fault domain, the system comprising:

a memory for storing a protected virtual machine list identifying a plurality of virtual machines within a fault domain that are to be maintained in an operating state;

a network communication interface configured to receive heartbeat messages from a plurality of hosts executing the virtual machines; and

a processor coupled to the memory and programmed to;

determine a host, within the plurality of hosts, from which the network communication interface has not received a heartbeat message within a first predetermined duration to identify an unreachable host;

determine whether the unreachable host has stored heartbeat data in a datastore within a second predetermined duration; and

restart a virtual machine executed by the unreachable host based on determining that the unreachable host has not stored heartbeat data in the datastore within the second predetermined duration.

2. The system of claim 1, wherein the network communication interface is further configured to receive from a management server a host list that includes a plurality of heartbeat datastores associated with the unreachable host, and the processor is further programmed to determine whether the unreachable host has stored heartbeat data in a datastore by determining whether the unreachable host has stored heartbeat data in any of the heartbeat datastores associated with the unreachable host.

3. The system of claim 1, wherein the network communication interface is further configured to receive the protected virtual machine list from a management server, and the processor is further programmed to execute a fault domain manager software application as a master host.

4. The system of claim 1, wherein the processor is further programmed to:

identify a datastore that stores a virtual disk image associated with the virtual machine; and

restart the virtual machine based further on determining that a file stored in the identified datastore can be exclusively locked.

5. The system of claim 1, wherein the processor is further programmed to:

monitor a power-on list stored in the heartbeat datastore by the unreachable host based on determining that the unreachable host has stored heartbeat data in the datastore, wherein the power-on list indicates virtual machines being executed by the unreachable host; and

restart the virtual machine when the virtual machine does not appear in the power-on list.

6. The system of claim 1, further comprising a second host other than the unreachable host, wherein the processor is programmed to restart the virtual machine by storing execution command data in a datastore that is accessible by the second host, wherein the execution command data instructs the second host to execute the virtual machine.

7. The system of claim 1, wherein the processor is further programmed to:

transmit a ping message to the unreachable host based on not receiving a heartbeat message from the unreachable host;

identify the unreachable host as a dead host when no response ping message corresponding to the transmitted ping message is received from the unreachable host; and

restart the virtual machine based further on identifying the unreachable host as a dead host.

8. A method comprising:

transmitting, by a subordinate host, a heartbeat message to a master host, wherein the transmitted heartbeat message indicates that the subordinate host is operative at a current time;

storing, by the subordinate host, heartbeat data in a datastore, wherein the stored heartbeat data indicates that the subordinate host is operative at the current time; and

storing, by the subordinate host, a power-on list in the datastore, wherein the power-on list indicates software application instances for a master host to restart at a host other than the subordinate host if the subordinate host becomes inoperative.

9. The method of claim 8, wherein the datastore is a first datastore, the method further comprising:

storing, by the subordinate host, the heartbeat data in a second datastore; and

storing, by the subordinate host, the power-on list in the second datastore.

10. The method of claim 9, further comprising:

receiving, by the subordinate host, from a management server a host list that includes a plurality of heartbeat datastores associated with the subordinate host, wherein the heartbeat datastores include the first datastore and the second datastore; and

storing, by the subordinate host, the heartbeat data and the power-on list in the first and second datastores based on the host list.

12. The method of claim 8, further comprising:

detecting, by the subordinate host, that a software application instance executed by the subordinate host has terminated; and

removing, by the subordinate host, the software application instance from the power-on list in the datastore.

13. The method of claim 8, wherein the subordinate host is configured to communicate with the master host via a gateway, the method further comprising:

determining, by the subordinate host, that the subordinate host is unable to communicate with the gateway;

including, by the subordinate host, an isolation notification in the power-on list;

terminating the software application instances being executed by the subordinate host; and

removing, by the subordinate host, the terminated software application instances from the power-on list.

14. The method of claim 8, further comprising:

receiving, by the subordinate host, from a management server a host list that includes a plurality of hosts and a thumbprint associated with each host of the plurality of hosts;

comparing, by the subordinate host, a thumbprint received from the master host with the thumbprint associated with the master host in the host list; and

disregarding, by the subordinate host, messages from the master host for a predetermined duration when the thumbprints do not match.

11. The method of 8, wherein the subordinate host is a first host, the method further comprising:

receiving, by the subordinate host, from a management server a host list that includes a version number, a plurality of host identifiers, and one or more heartbeat datastores associated with each host identifier of the plurality of host identifiers;

transmitting the version number to a second host, wherein the second host transmits to the subordinate host a request for the host list when the transmitted version number is greater than a version number included in a host list stored by the second host; and

transmitting the host list from the subordinate host to the second host in response to the request.

15. One or more computer-readable storage media having computer-executable components comprising:

a datastore selection component that when executed causes at least one processor to;

identify a plurality of datastores to which a subordinate host has read-write access;

select from the plurality of datastores a first heartbeat datastore and a second heartbeat datastore based on a quantity of hosts that have access to the first and second heartbeat datastores; and

associate the first and second heartbeat datastores with the host; and

a heartbeat publication component that when executed causes at least one processor to;

transmit to a master host a heartbeat message indicating that the subordinate host is operative at a current time; and

store in the first and second heartbeat datastores heartbeat data indicating that the subordinate host is operative at the current time.

16. The computer readable storage media of claim 15, wherein the computer-executable components further comprise an application instance monitoring component that when executed causes at least one processor to:

determine whether a heartbeat message is received from the subordinate host at the master host;

determine whether the subordinate host has stored the heartbeat data in the first heartbeat datastore and the second heartbeat datastore when no heartbeat message is received from the subordinate host at the master host; and

restart a software application instance associated with the subordinate host when the subordinate host has not stored the heartbeat data in the first heartbeat datastore or in the second heartbeat datastore.

17. The computer readable storage media of claim 15, wherein each datastore of the plurality of datastores is associated with a storage device, and the datastore selection component causes the processor to select the first and second heartbeat datastores based further on the first and second heartbeat datastores being associated with different storage devices.

18. The computer readable storage media of claim 15, wherein datastore selection component further causes the processor to:

receive a selection of preferred datastores; and

select the first and second heartbeat datastores based further on determining that the first and second datastores are included in the preferred datastores.

19. The computer readable storage media of claim 15, further comprising a datastore monitoring component that when executed causes at least one processor to:

detect a newly available datastore; and

after a predetermined duration, execute the datastore selection component based on the detected newly available datastore.

20. The computer readable storage media of claim 19, wherein the datastore monitoring component further causes the processor to:

detect an impending unavailability of the first datastore; and

execute the datastore selection component based on the detected impending unavailability.