Storage handling for fault tolerance in virtual machines

• US 7,840,839 B2
• Filed: 09/09/2008
• Issued: 11/23/2010
• Est. Priority Date: 11/06/2007
• Status: Active Grant

• Current Assignee: Vmware LLC (Broadcom, Inc.)
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: Tucker, Andrew, Lowe, Eric, Scales, Daniel J., Nelson, Michael
• Primary Examiner(s): Chu; Gabriel L
• Application Number:
  US12/207,292
• Publication Number:
  US 20090119538A1
• Time in Patent Office: 805 Days
• Field of Search: None
• US Class Current: 714/13
• CPC Class Codes: G06F 11/14 Error detection or correction of the data by redundancy in operation G06F11/16 takes precedence
• CPC Class Codes: G06F 11/1683 at instruction level
• CPC Class Codes: G06F 11/1695 which are operating with time diversity
• CPC Class Codes: G06F 11/2033 switching over of hardware resources
• CPC Class Codes: G06F 11/2038 with a single idle spare processing component
• CPC Class Codes: G06F 11/2046 where the redundant components share persistent storage G06F11/2043 takes precedence
• CPC Class Codes: G06F 2009/45579 I/O management, e.g. providing access to device drivers or storage
• CPC Class Codes: G06F 2201/815 Virtual middleware or OS functionality using virtual machines to implement generic software techniques for error detection or fault masking G06F11/1484
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects

Abstract

A fault-tolerant virtualized computer system comprises a primary host executing a primary virtual machine (VM) and a backup host executing a backup VM. In one embodiment, each VM includes a virtual disk mapped to a shared disk image on physical disk accessible to both hosts. A virtual disk IO request is received by virtualization software executing on the backup host from a backup VM. When the virtual disk IO request is a read request, the virtual disk IO request is mapped to a physical read IO request of a virtual disk image stored on a physical disk, the physical read IO request is issued to the physical disk. However, when the virtual disk IO request is a write request, it is assumed to be completed without mapping the virtual disk IO request or issuing a mapped physical IO request to the physical disk.

Claims

1. A computer readable storage medium for providing shared storage architecture in a fault-tolerant virtualized computer system, the fault-tolerant virtualized computer system comprising a primary host executing a primary virtual machine (VM) and a backup host executing a backup VM, the computer readable storage medium embodying machine instructions causing the backup host to implement a method, the method comprising:

receiving a virtual disk IO request from the backup VM, the virtual disk IO request being identical to a corresponding virtual disk IO request issued by the primary VM;

when the virtual disk IO request is a read request, mapping the virtual disk IO request to a physical read IO request of a virtual disk image stored on a physical disk, and issuing the physical read IO request to the physical disk;

when the virtual disk IO request is a write request, assuming a completed status of the disk IO request without mapping the virtual disk IO request or issuing a mapped physical IO request to the physical disk;

receiving a log entry from a primary host, the log entry containing a completion interrupt and execution point for the corresponding virtual disk IO request issued by the primary VM; and

replaying the completion interrupt in the backup VM at an execution point in execution of the backup VM, the execution point corresponding to the execution point identified in the log entry.

2. The computer readable storage medium of claim 1, wherein the method further comprises:

responsive to failure of the primary VM, identifying a set of virtual disk IO requests having corresponding disk IO requests generated by the primary VM that were in flight at the time the failover occurred but for which a mapped physical IO request has not been issued to the physical disk, the set being identified as each virtual disk IO request that has been received from the backup VM but for which no completion interrupt log entry has been received, less ones of the virtual disk IO requests for which a mapped physical IO request has been issued to the physical disk;

mapping each virtual disk IO request of the set of virtual disk IO requests to mapped physical disk IO requests; and

issuing the mapped physical disk IO requests to the physical disk.

3. The computer readable storage medium of claim 1, wherein the execution point identified by the log entry is reached by the backup VM prior to receipt of a completion interrupt for the physical IO request, and the method further comprises:

suspending the backup virtual machine until the completion interrupt is received.

4. A computer readable storage medium for providing a fault-tolerant virtualized computer system, the fault-tolerant virtualized computer system comprising a primary host executing a primary virtual machine (VM) and a backup host executing a backup VM, the computer readable storage medium embodying machine instructions causing the backup host to implement a method, the method comprising:

receiving a virtual IO request from the backup VM, the virtual IO request being identical to a corresponding virtual IO request issued by the primary VM;

receiving a log entry from a primary host, the log entry containing a completion interrupt and execution point for the corresponding virtual IO request issued by the primary VM,

determining whether the log entry received from the primary host indicates that an error status was returned when processing the corresponding virtual IO request;

in response to a determination that the log entry indicates that the error status was returned, modifying virtual memory of the backup VM or a second virtual disk of the backup VM with the current state of a first virtual memory or a first virtual disk provided by the primary virtual machine in the log entry.

5. The computer readable storage medium of claim 4, wherein the method further comprises:

mapping the virtual IO request to a physical IO request of a physical device that is mapped to a virtual device accessible by the backup VM, and issuing the physical IO request to the physical device;

receiving a completion interrupt in response to the issuing of the physical IO request;

determining whether the completion interrupt contains an IO error status;

determining whether the IO error status of the completion interrupt is different from an IO completion status indicated in the log entry, the IO completion status being one of a plurality of IO error statuses or a successful status; and

terminating the backup virtual machine when the IO error status of the completion interrupt is different from the IO completion status indicated in the log entry.

6. The computer readable storage medium of claim 5, wherein the method further comprises:

reissuing the physical IO request prior to terminating the backup virtual machine; and

terminating the backup virtual machine only when the reissued physical IO request returns the error status.

7. The computer readable storage medium of claim 5, wherein the method further comprises:

starting a third VM to provide fault tolerance to the primary VM.

8. A computer readable storage medium for providing shared storage architecture in a fault-tolerant virtualized computer system, the fault-tolerant virtualized computer system comprising a primary host executing a primary virtual machine (VM) and a backup host executing a backup VM, wherein the primary VM has a virtual disk mapped to a disk image on a physical disk by virtualization software executing on the primary host and the backup VM has a virtual disk mapped to the disk image on the physical disk by virtualization software executing on the backup host, the computer readable storage medium embodying machine instructions causing the primary host to implement a method, the method comprising:

for each disk read IO request received from the primary VM, entering an accessed range of a virtual disk supported by the shared storage device in a disk range data structure, the accessed range defining a set of locations on the virtual disk of the primary VM that are targeted by the disk read IO request; and

for each disk write IO request received from the primary VM, determining if the disk write IO request is directed to the disk range entered in the disk range data structure and if so, delaying the disk write operation until the disk read operation is completed by the backup virtual machine and the accessed range is removed from the disk range data structure.

9. The computer readable storage medium of claim 8, wherein the method further comprises:

installing a trace on one or more memory pages targeted by direct memory access (DMA) when satisfying the disk read IO request;

removing the trace on completion of the DMA; and

responsive to triggering of the trace by a guest operating system running on the primary VM accessing the memory pages, suspending the primary VM until the direct memory access is completed and the trace is removed.

10. A computer readable storage medium for providing a fault-tolerant virtualized computer system, the fault-tolerant virtualized computer system comprising a primary host executing a primary virtual machine (VM) and a backup host executing a backup VM, the computer readable medium embodying machine instructions causing the primary host to implement a method, the method comprising:

receiving a first virtual device IO request from the primary VM, the first virtual device IO request being at least one of a read or a write to addressable memory of the virtual device;

in response to the receiving, entering an accessed range of the device in a range data structure, the accessed range being an identification of the portions of the addressable memory identified by the first virtual device IO request;

mapping the first virtual device IO request from the primary VM to a physical device IO request and issuing the physical device IO request to a physical device that is mapped to the virtual device;

receiving a second virtual device IO request form the primary VM, the second virtual device IO request being received after the receipt of the first virtual device IO request;

determining whether the second virtual device IO request is directed to the accessed range of the device entered in the range data structure; and

delaying issuing a physical IO operation mapped from the second virtual device IO request until the first physical IO request is completed.

11. The computer readable storage medium of claim 10, wherein the device is a virtual disk mapped to disk shared by the primary host and the backup hosts.

12. The computer readable storage medium of claim 10, wherein the device is a virtual memory.

13. The computer readable storage medium of claim 10, wherein the first virtual device IO request includes a scatter-gather array having a first scatter-gather entry and a second scatter-gather entry, and the method further comprises:

entering an accessed range of a first virtual memory corresponding to the first scatter-gather entry in a memory range data table;

determining whether the second scatter-gather entry is directed to the accessed range of the device;

responsive to determining that the second scatter-gather entry is directed to the accessed range, adding contents corresponding to the entered accessed range of the first virtual memory in a log entry; and

sending the log entry to the backup host.

14. A method for deterministically accessing storage resources in a fault-tolerant system wherein a first virtual machine (VM) executes on a first host and a second VM executes on a second host, the second VM providing fault tolerance for the first VM, the storage resources comprising a first virtual disk accessible by the first VM and a second virtual disk accessible by the second VM, the method comprising:

executing virtualization software on the first host, the virtualization software being logically interposed between and interfacing with the first VM and the first host, wherein the virtualization software;

receives a virtual disk IO request issued by the first VM to the virtual disk;

maps the virtual disk IO request to a corresponding physical disk IO request, the physical disk IO request defining an IO access of a physical disk, the physical disk being accessible by the first host;

enters an accessed range of the physical disk IO request in a range data structure;

determines from the range data structure whether the physical disk IO request is directed to a conflicting accessed range of a pending physical disk IO request;

delays issuing the physical disk IO request when the physical disk IO request is directed to the conflicting accessed range, the physical disk IO request being delayed until the pending physical disk IO request is completed and the accessed range of the device is removed from the range data structure; and

issues the physical disk IO request to a first physical disk when the physical disk IO request is not directed to a conflicting accessed range;

sends the log entries containing information regarding each nondeterministic event associated with the first VM to the second host so that the second host can inject corresponding nondeterministic events into the VM machine and the second VM can replicate the execution of the first VM.

15. The method of claim 14, wherein:

virtualization software executing on the second host maps the second virtual disk to physical disk such that a single virtual disk image on the physical disk is accessed by the first VM and the second VM.

16. The method of claim 15, wherein the virtualization software executing on the second host maps virtual disk read IO requests issued by the second VM to the physical disk and marks virtual disk write IO requests as completed without mapping the virtual disk write IO requests to the physical disk.

17. The method of claim 16, wherein the virtualization software executing on the second host, in response to a failover from the first VM to the second VM, maps virtual disk write IO requests to physical disk IO requests and issues the physical disk IO requests to the physical disk for corresponding physical disk write IO requests that were generated by the first host and were in flight on the first host at the time the failover occurred.

18. The method of claim 14, further comprising:

sending, in response to a physical IO request on the first host returning an error status, a current state of at least a portion of a first virtual memory or the first virtual disk targeted by the physical IO request to the second VM in the log entry, the sending being performed by the virtualization software executing on the first host; and

replicating the state sent by the first host on the second host, the replicating being performed after the second host completes a physical IO request that corresponds to the physical IO request on the first host, the replicating being performed by virtualization software executing on the second host.