Virtual machine fault tolerance

• US 8,201,169 B2
• Filed: 06/15/2009
• Issued: 06/12/2012
• Est. Priority Date: 06/15/2009
• Status: Active Grant

• Current Assignee: Vmware LLC (Broadcom, Inc.)
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: Venkitachalam, Ganesh, Weissman, Boris, Scales, Daniel J., Malyugin, Vyacheslav, Sheldon, Jeffrey W., Xu, Min, Jain, Rohit
• Primary Examiner(s): An, Meng
• Assistant Examiner(s): Lu, Kevin X
• Application Number:
  US12/484,640
• Publication Number:
  US 20100318991A1
• Time in Patent Office: 1,093 Days
• Field of Search: 718/1, 714/2, 714 11- 13, 714/20
• US Class Current: 718/1
• CPC Class Codes: G06F 11/0712 in a virtual computing platform, e.g. logically partitioned systems
• CPC Class Codes: G06F 11/0766 Error or fault reporting or storing
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/1658 Data re-synchronization of a redundant component, or initial sync of replacement, additional or spare unit
• CPC Class Codes: G06F 2009/45579 I/O management, e.g. providing access to device drivers or storage
• CPC Class Codes: G06F 2009/45591 Monitoring or debugging support
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: G06F 2201/815 Virtual
• CPC Class Codes: G06F 2201/82 Solving problems relating to consistency
• CPC Class Codes: G06F 9/45533 Hypervisors; Virtual machine monitors
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues

Abstract

In a computer system running a primary virtual machine (VM) on virtualization software on a primary virtualized computer system (VCS) and running a secondary VM on virtualization software on a secondary VCS, a method for the secondary VM to provide quasi-lockstep fault tolerance for the primary VM includes: as the primary VM is executing a workload, virtualization software in the primary VCS is: (a) causing predetermined events to be recorded in an event log, (b) keeping output associated with the predetermined events pending, and (c) sending the log entries to the virtualization software in the secondary VCS; as the secondary VM is replaying the workload, virtualization software in the secondary VCS is: (a) sending acknowledgements indicating that log entries have been received; (b) when the virtualization software encounters one of the predetermined events, searching the log entries to determine whether a log entry corresponding to the same event was received from the primary VCS, and if so, comparing data associated with the predetermined event produced by the secondary VM with that of the primary VM; if there is a match, the virtualization software in the secondary VCS transmitting an acknowledgement to the virtualization software in the primary VCS; one of the virtualization software in the primary or secondary VCS dropping the event and the other dispatching the output; and if there is no match, performing a checkpoint resynchronization.

Claims

1. In a computer system running at least a primary virtual machine (VM) on virtualization software on a primary virtualized computer system (VCS) and running a secondary VM on virtualization software on a secondary VCS, a computer implemented method for the secondary VM to provide quasi-lockstep fault tolerance for the primary VM comprises:

as the primary VM is executing a workload, virtualization software in the primary VCS is;

(a) causing predetermined events to be recorded in an event log, (b) keeping output associated with the predetermined events pending, and (c) sending the log entries to the virtualization software in the secondary VCS;

as the secondary VM is replaying the workload, virtualization software in the secondary VCS is;

(a) sending acknowledgements indicating that log entries have been received;

(b) when the virtualization software encounters one of the predetermined events, searching the log entries to determine whether a log entry corresponding to the same event was received from the primary VCS, and if so, comparing data associated with the predetermined event produced by the secondary VM with that of the primary VM;

if there is a match, the virtualization software in the secondary VCS transmitting an acknowledgement to the virtualization software in the primary VCS;

one of the virtualization software in the primary or secondary VCS dropping the event and the other dispatching the output; and

if there is no match, performing a checkpoint resynchronization;

wherein at least one of the predetermined events relates to virtual disk I/O, and virtual disks are shared; and

wherein the secondary VM output is not sent to the virtual disks after a match is found.

2. The computer implemented method of claim 1 wherein:

at least one of the predetermined events is an externally visible event; and

one of the virtualization software in the primary VCS or secondary VCS dropping the event and the other dispatching the output comprises dispatching externally visible output.

3. The computer implemented method of claim 2 wherein the externally visible event is an I/O event.

4. In a computer system running at least a first virtual machine (VM) and a second VM on virtualization software, a computer implemented method for the second VM to provide quasi-lockstep fault tolerance for the first VM comprises:

recording predetermined operations of the first VM to log entries and communicating the log entries to the second VM until the first VM reaches a first externally visible output;

replaying the log entries at the second VM until the second VM reaches a second externally visible output; and

responsive to the second externally visible output diverging from the first externally visible output, the first VM initiating a checkpoint resynchronization process and the second VM completing the checkpoint resynchronization process for the second VM to restore checkpointed states of the first VM and be synchronized with the first VM, wherein the checkpoint resynchronization process comprises;

(a) pausing progress of the first VM, or preventing the first VM'"'"'s checkpointed state from being overwritten before the second VM restores the checkpointed state of the first VM, wherein pausing includes stopping I/O completions from being posted to the first VM and stopping guest operating system instructions from being executed at the first VM;

(b) serializing an emulation state of the first VM, storing the emulation state of the first VM in a serialized file, and sending the serialized file to the second VM;

(c) restoring execution at the second VM based on the serialized state of the first VM; and

(d) reissuing pending I/O operations at the second VM.

5. The method of claim 4, further comprising:

keeping the first externally visible output pending in the first VM.

6. The method of claim 5, further comprising:

determining whether the second externally visible output becomes same as the first externally visible output within a predetermined amount of time.

7. The method of claim 4, further comprising:

responsive to a fault in the first VM, the second VM taking over operation of the first VM.

8. The method of claim 4, further comprising:

responsive to a fault in the second VM, the first VM initiating a checkpoint resynchronization process and a third VM completing the checkpoint resynchronization process to restore checkpointed states of the first VM and be synchronized with the first VM.

9. The method of claim 4, further comprising reducing occurrences of the second externally visible output diverging from the first externally visible output, said reducing occurrences including:

recording virtual non-determinism of the first VM except memory access by different virtual processors of the first VM to the log entries, for replay of the log entries by the second VM.

10. The method of claim 4, further comprising reducing occurrences of the second externally visible output diverging from the first externally visible output, said reducing occurrences including:

allowing the second virtual machine to continue replay of the log entries even after control divergence in network packets within the virtualized computer system, until a physical output packet diverges between the first virtual machine and the second virtual machine.

11. The method of claim 4, further comprising reducing occurrences of the second externally visible output diverging from the first externally visible output, said reducing occurrences including:

suppressing the second externally visible output at the second VM; and

allowing I/O operation from external sources to be performed at the second VM, until physical output divergence between the first VM and the second VM.

12. The method of claim 4, further comprising reducing occurrences of the second externally visible output diverging from the first externally visible output, said reducing occurrences including:

allowing I/O operations to continue in the second VM even if I/O packets diverge between the first VM and the second VM, until actual payload data in the I/O packets diverge between the first VM and the second VM.

13. The method of claim 4, further comprising reducing occurrences of the second externally visible output diverging from the first externally visible output, said reducing occurrences including:

scheduling virtual processors of the first VM accessing a shared memory region at different times, for replay of the log entries including a scheduling plan of the virtual processors by the second VM.

14. The method of claim 4, further comprising reducing occurrences of the second externally visible output diverging from the first externally visible output, said reducing occurrences including:

tracking access by a virtual processor of the first VM to a memory page not belonging to the virtual processor, for replay of the log entries including the tracked access by the second VM.

15. The method of claim 4, further comprising reducing occurrences of the second externally visible output diverging from the first externally visible output, said reducing occurrences including:

adding read values from access to a shared memory region by virtual processors not corresponding to the first VM to the log entries, for replay of the read values at the second VM based on the log entries.

16. The method of claim 4, further comprising reducing occurrences of the second externally visible output diverging from the first externally visible output, said reducing occurrences including:

adding an order of lock acquisition by a guest operating system of the first VM to the log entries, for replay of the order of lock acquisitions at the second VM based on the log entries.

17. In a computer system running at least a primary virtual machine (VM) on virtualization software on a primary virtualized computer system (VCS) and running a secondary VM on virtualization software on a secondary VCS, a computer implemented method for the secondary VM to provide quasi-lockstep fault tolerance for the primary VM comprises:

as the primary VM is executing a workload, virtualization software in the primary VCS is;

(a) causing predetermined events to be recorded in an event log, (b) keeping output associated with the predetermined events pending, and (c) sending the log entries to the virtualization software in the secondary VCS;

as the secondary VM is replaying the workload, virtualization software in the secondary VCS is;

(a) sending acknowledgements indicating that log entries have been received;

(b) when the virtualization software encounters one of the predetermined events, searching the log entries to determine whether a log entry corresponding to the same event was received from the primary VCS, and if so, comparing data associated with the predetermined event produced by the secondary VM with that of the primary VM;

if there is a match, the virtualization software in the secondary VCS transmitting an acknowledgement to the virtualization software in the primary VCS;

one of the virtualization software in the primary or secondary VCS dropping the event and the other dispatching the output; and

if there is no match, performing a checkpoint resynchronization;

wherein the predetermined events relate to virtual disk I/O, and the virtual disks are not shared; and

wherein the secondary VM output is sent to the second VM'"'"'s virtual disk after a match is found.