MAINTAINING HIGH AVAILABILITY OF A GROUP OF VIRTUAL MACHINES USING HEARTBEAT MESSAGES

• US 20150089272A1
• Filed: 12/05/2014
• Published: 03/26/2015
• Est. Priority Date: 04/28/2011
• Status: Active Grant

• Current Assignee: Vmware LLC (Broadcom, Inc.)
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: Nelson, Michael, Farkas, Keith, Ziskind, Elisha, Rajagopal, Sridhar, Shu, Guoqiang, Passerini, Ron, Ren, Joanne
• Granted Patent: US 9,558,070 B2
• Time in Patent Office: Days
• US Class Current: 714/4.1
• CPC Class Codes: G06F 11/0757 by exceeding a time limit, i.e. time-out, e.g. watchdogs
• CPC Class Codes: G06F 11/1438 Restarting or rejuvenating
• CPC Class Codes: G06F 9/45533 Hypervisors; Virtual machine monitors
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications

Abstract

Embodiments maintain high availability of software application instances in a fault domain. Subordinate hosts are monitored by a master host. The subordinate hosts publish heartbeats via a network and datastores. Based at least in part on the published heartbeats, the master host determines the status of each subordinate host, distinguishing between subordinate hosts that are entirely inoperative and subordinate hosts that are operative but partitioned (e.g., unreachable via the network). The master host may restart software application instances, such as virtual machines, that are executed by inoperative subordinate hosts or that cease executing on partitioned subordinate hosts.

Claims

1. A system for maintaining high availability of a plurality of virtual machines in a fault domain, the system comprising:

a network communication interface configured to receive heartbeat messages from a plurality of hosts executing the virtual machines; and

a processor coupled to the memory and programmed to;

associate a datastore with a host, within the plurality of hosts, based on at least one of;

     1) quantity of hosts that have access to the datastore,

     2) whether the datastore is associated with the same storage device as another datastore, and

     3) the file system type of the datastore;

determine whether the host is an unreachable host or an inoperative host based on the received heartbeat messages and heartbeat data stored in the datastore; and

restart a virtual machine that ceases executing on the unreachable host or executed by the inoperative host based on determining that the host is the unreachable host or the inoperative host.

2. The system of claim 1, wherein the processor is further programmed to associate the datastore with the host by ranking a plurality of datastores based on their accessibility scores and discarding a datastore with an accessibility score that is below a threshold value.

3. The system of claim 2, wherein the accessibility score is based on at least two of:

1) the quantity of the hosts that have access to the datastore,

     2) whether the datastore is associated with the same storage device as the another datastore, and

     3) the file system type of the datastore.

4. The system of claim 1, wherein the network communication interface is further configured to receive from a management server a host list that includes a plurality of heartbeat datastores associated with the unreachable host, and the processor is further programmed to determine whether the unreachable host has stored heartbeat data in the datastore by determining whether the unreachable host has stored heartbeat data in any of the heartbeat datastores associated with the unreachable host.

5. The system of claim 1, wherein the network communication interface is further configured to receive a protected virtual machine list identifying a plurality of virtual machines within a fault domain that are to be maintained in an operating state from a management server, and the processor is further programmed to execute a fault domain manager software application as a master host.

6. The system of claim 1, wherein the processor is further programmed to:

identify a datastore that stores a virtual disk image associated with the virtual machine; and

restart the virtual machine based further on determining that a file stored in the identified datastore can be exclusively locked.

7. The system of claim 1, wherein the processor is further programmed to:

monitor a power-on list stored in the datastore by the unreachable host based on determining that the unreachable host has stored heartbeat data in the datastore, wherein the power-on list indicates virtual machines being executed by the unreachable host; and

restart the virtual machine when the virtual machine does not appear in the power-on list.

8. The system of claim 1, further comprising a second host other than the unreachable host, wherein the processor is programmed to restart the virtual machine by storing execution command data in a datastore that is accessible by the second host, wherein the execution command data instructs the second host to execute the virtual machine.

9. The system of claim 1, wherein the processor is further programmed to:

transmit a ping message to the unreachable host based on not receiving a heartbeat message from the unreachable host;

identify the unreachable host as a dead host when no response ping message corresponding to the transmitted ping message is received from the unreachable host; and

restart the virtual machine based further on identifying the unreachable host as a dead host.

10. A method comprising:

associating a datastore with a host based on at least one of;

     1) quantity of hosts that have access to the datastore,

     2) whether the datastore is associated with the same storage device as another datastore, and

     3) the file system type of the datastore;

determining whether the host is an unreachable host or an inoperative host based on a received heartbeat message and heartbeat data stored in the datastore; and

restarting a virtual machine that ceases executing on the unreachable host or executed by the inoperative host based on determining that the host is the unreachable host or the inoperative host.

11. The method of claim 10, wherein associating the datastore with the host comprises ranking a plurality of datastores based on their accessibility scores and discarding a datastore with an accessibility score that is below a threshold value.

12. The method of claim 11, wherein the accessibility score is based on at least two of:

1) the quantity of the hosts that have access to the datastore,

     2) whether the datastore is associated with the same storage device as the another datastore, and

     3) the file system type of the datastore.

13. The method of claim 10, further comprising receiving from a management server a host list that includes a plurality of heartbeat datastores associated with the unreachable host, and determining whether the unreachable host has stored heartbeat data in the datastore by determining whether the unreachable host has stored heartbeat data in any of the heartbeat datastores associated with the unreachable host.

14. The method of claim 10, further comprising receiving a protected virtual machine list identifying a plurality of virtual machines within a fault domain that are to be maintained in an operating state from a management server, and executing a fault domain manager software application as a master host.

15. The method of claim 10, further comprising:

identifying a datastore that stores a virtual disk image associated with the virtual machine; and

restarting the virtual machine based further on determining that a file stored in the identified datastore can be exclusively locked.

16. The method of claim 10, further comprising:

monitoring a power-on list stored in the datastore by the unreachable host based on determining that the unreachable host has stored heartbeat data in the datastore, wherein the power-on list indicates virtual machines being executed by the unreachable host; and

restarting the virtual machine when the virtual machine does not appear in the power-on list.

17. The method of claim 10, further comprising restarting the virtual machine by storing execution command data in a datastore that is accessible by a second host, wherein the execution command data instructs the second host to execute the virtual machine.

18. The method of claim 10, further comprising:

transmitting a ping message to the unreachable host based on not receiving a heartbeat message from the unreachable host;

identifying the unreachable host as a dead host when no response ping message corresponding to the transmitted ping message is received from the unreachable host; and

restarting the virtual machine based further on identifying the unreachable host as a dead host.

19. A computer readable non-transitory storage medium storing instructions which when executed by a computer cause the computer to perform a method, the method comprising:

associating a datastore with a host based on at least one of;

     1) quantity of hosts that have access to the datastore,

     2) whether the datastore is associated with the same storage device as another datastore, and

     3) the file system type of the datastore;

determining whether the host is an unreachable host or an inoperative host based on a received heartbeat message and heartbeat data stored in the datastore; and

restarting a virtual machine that ceases executing on the unreachable host or executed by the inoperative host based on determining that the host is the unreachable host or the inoperative host.

20. The computer readable non-transitory storage medium of claim 19, wherein associating the datastore with the host comprises ranking a plurality of datastores based on their accessibility scores and discarding a datastore with an accessibility score that is below a threshold value.

21. The computer readable non-transitory storage medium of claim 20, wherein the accessibility score is based on at least two of:

1) the quantity of the hosts that have access to the datastore,

     2) whether the datastore is associated with the same storage device as the another datastore, and

     3) the file system type of the datastore.

22. The computer readable non-transitory storage medium of claim 19, the method further comprising receiving from a management server a host list that includes a plurality of heartbeat datastores associated with the unreachable host, and determining whether the unreachable host has stored heartbeat data in the datastore by determining whether the unreachable host has stored heartbeat data in any of the heartbeat datastores associated with the unreachable host.

23. The computer readable non-transitory storage medium of claim 19, the method further comprising receiving a protected virtual machine list identifying a plurality of virtual machines within a fault domain that are to be maintained in an operating state from a management server, and executing a fault domain manager software application as a master host.

24. The computer readable non-transitory storage medium of claim 19, the method further comprising:

identifying a datastore that stores a virtual disk image associated with the virtual machine; and

restarting the virtual machine based further on determining that a file stored in the identified datastore can be exclusively locked.

25. The computer readable non-transitory storage medium of claim 19, the method further comprising:

monitoring a power-on list stored in the datastore by the unreachable host based on determining that the unreachable host has stored heartbeat data in the datastore, wherein the power-on list indicates virtual machines being executed by the unreachable host; and

restarting the virtual machine when the virtual machine does not appear in the power-on list.

26. The computer readable non-transitory storage medium of claim 19, the method further comprising further comprising restarting the virtual machine by storing execution command data in a datastore that is accessible by a second host, wherein the execution command data instructs the second host to execute the virtual machine.

27. The computer readable non-transitory storage medium of claim 19, the method further comprising:

transmitting a ping message to the unreachable host based on not receiving a heartbeat message from the unreachable host;

identifying the unreachable host as a dead host when no response ping message corresponding to the transmitted ping message is received from the unreachable host; and

restarting the virtual machine based further on identifying the unreachable host as a dead host.