Efficient suspend and resume of instances

• US 9,645,847 B1
• Filed: 06/08/2015
• Issued: 05/09/2017
• Est. Priority Date: 06/08/2015
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Roth, Gregory Branchek
• Primary Examiner(s): Nguyen, Phillip H
• Application Number:
  US14/733,834
• Time in Patent Office: 701 Days
• Field of Search: 718 1, 718104
• CPC Class Codes: G06F 2009/45575 Starting, stopping, suspending or resuming virtual machine instances
• CPC Class Codes: G06F 9/45533 Hypervisors; Virtual machine monitors
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects
• CPC Class Codes: G06F 9/461 Saving or restoring of program or task context
• CPC Class Codes: G06F 9/4868 with creation or replication
• CPC Class Codes: G06F 9/505 considering the load
• CPC Class Codes: G06F 9/52 Program synchronisation; Mutual exclusion, e.g. by means of semaphores

Abstract

A virtual machine is suspended as a result of receiving a request for suspending a plurality of virtual machines, and, for each virtual machine of the plurality, obtaining a first snapshot of an exemplary virtual machine, obtaining a second snapshot of the virtual machine, determining a set of changes based on the second snapshot and the first snapshot, storing the set of changes in persistent storage in association with the virtual machine, and terminating the virtual machine. The system and method further including receiving a request for restoring the plurality of virtual machines to corresponding previous states, and, for each virtual machine of the plurality, obtaining the first snapshot of the exemplary virtual machine, obtaining the second snapshot of the virtual machine, and restoring the virtual machine to a previous state based on the second snapshot.

Claims

1. A computer-implemented method, comprising:

under the control of one or more computer systems configured with executable instructions,

receiving a first request whose fulfillment involves suspending operations of a fleet of virtual machines of a customer of a computing resource service provider;

as a result of receiving the first request, for each virtual machine of the fleet of virtual machines;

preventing the virtual machine from receiving information from an external source that causes the virtual machine to perform work;

obtaining a first set of local changes at least in part from persistent storage, the first set of local changes representing at least in part the differences between a base snapshot of an exemplary virtual machine image and a first snapshot of the virtual machine;

capturing a second snapshot of the virtual machine;

determining a second set of local changes based at least in part on differences between the second snapshot and the first snapshot;

storing the first set of local changes and the second set of local changes; and

terminating the virtual machine;

receiving a second request whose fulfillment involves restoring former states of the fleet of virtual machines; and

as a result of receiving the second request, for each virtual machine of the fleet of virtual machines;

obtaining the first set of local changes, the second set of local changes, and the base snapshot from persistent storage;

obtaining the first snapshot at least in part from the base snapshot and the first set of local changes;

obtaining the second snapshot at least in part from the first snapshot and the second set of local changes; and

instantiating the virtual machine to a former state based at least in part on the second snapshot.

2. The computer-implemented method of claim 1, wherein the fleet of virtual machines were instantiated from a common software image containing contents and structure of a storage volume.

3. The computer-implemented method of claim 1, wherein preventing the virtual machine from receiving work includes removing the virtual machine from a group of virtual machines being load-balanced.

4. The computer-implemented method of claim 1, wherein:

the virtual machine is instantiated to the former state on a different physical computing system than a physical computing system the virtual machine was running on before the virtual machine was terminated.

5. The computer-implemented method of claim 4, wherein:

the first request is received as a result of an increased demand for virtual machines of an on-demand virtual computer system service of a computing resource service provider; and

the second request is received as a result of reduced demand for the virtual machines of the on-demand virtual computer system service.

6. The computer-implemented method of claim 1, wherein the virtual machine is instantiated to the former state on the same physical computing system the virtual machine was running on before the virtual machine was terminated.

7. A system, comprising:

one or more processors; and

memory including instructions that, when executed by the one or more processors, cause the system to;

store a first set of local changes relative to a base snapshot and an initial snapshot;

receive a first request whose fulfillment involves suspending operations of a first virtual machine provided by a computing resource service provider;

as a result of receiving the first request;

capture a second snapshot of a state of the first virtual machine;

determine a second set of local changes based at least in part on differences between the second snapshot and a derived snapshot derived from the first set of local changes and the base snapshot;

store the second set of local changes; and

terminate the first virtual machine;

receive a second request whose fulfillment involves instantiating a second virtual machine and configuring the second virtual machine to the state of the first virtual machine; and

as a result of receiving the second request;

derive a derived second snapshot based at least in part on the second set of local changes and the first set of local changes;

instantiate the second virtual machine; and

configure the second virtual machine to the state of the first virtual machine based at least in part on the derived second snapshot.

8. The system of claim 7, wherein the first and second virtual machines are members of a fleet of virtual machines having a common configuration but individual states.

9. The system of claim 7, wherein the first and second virtual machines are instantiated for use by an entity different from the computing resource service provider.

10. The system of claim 7, wherein a size of the set of local changes is less than twenty-five percent of a size of the base snapshot.

11. The system of claim 7, wherein:

the virtual machine is a virtual machine of an on-demand virtual computer system service of a computing resource service provider; and

the first request is associated at least in part on a reduction in demand for virtual computing resources from the on-demand virtual computer system service.

12. The system of claim 7, wherein the state of the first virtual machine corresponds to a state of operation of the first virtual machine at a time after the first request is received.

13. The system of claim 7, wherein the virtual machine is restored to a different physical computing system than a physical computing system the virtual machine was running on before the virtual machine was suspended.

14. The system of claim 7, wherein the virtual machine was instantiated from a base software image from which at least one other virtual machine is derived.

15. The system of claim 7, wherein the snapshot is a snapshot of one or more of contents of local memory of the virtual machine or contents of block-level storage allocated to the virtual machine at a time after receiving a request to suspend the virtual machines.

16. The system of claim 7, wherein the virtual machine is restored to a physical computing system that is the same physical computing system the virtual machine was running on before the virtual machine was suspended.

17. A non-transitory computer-readable storage medium having stored thereon executable instructions that, when executed by one or more processors of a computer system, cause the computer system to at least:

receive a first request to suspend a set of virtual machines associated with an account of a customer of a computing resource service provider; and

for each virtual machine of the set of virtual machines;

obtain a first snapshot of an exemplary virtual machine;

obtain a second snapshot of the virtual machine of the set of virtual machines;

determine a first set of changes based at least in part on the second snapshot and the first snapshot;

store the first set of changes in persistent storage in association with the virtual machine of the set of virtual machines;

obtain a third snapshot of the virtual machine of the set of virtual machines;

determine a second set of changes based at least in part on the third snapshot and the second snapshot;

store the second set of changes in persistent storage in association with the virtual machine of the set of virtual machines; and

terminate the virtual machine of the set of virtual machines;

receive a second request to restore the set of virtual machines; and

in response to receiving the second request for each virtual machine of the set of virtual machines;

obtain the first set of changes and the second set of changes;

derive the third second snapshot based at least in part on the second set of changes relative to the second snapshot and the first set of changes relative to the first snapshot; and

instantiate and configure the virtual machine based at least in part on the third snapshot.

18. The non-transitory computer-readable storage medium of claim 17, wherein:

the set of virtual machines are virtual machines of an on-demand virtual computer system service of a computing resource service provider; and

the request to suspend the set of virtual machines is associated at least in part on an increase in demand for virtual computing resources from the on-demand virtual computer system service.

19. The non-transitory computer-readable storage medium of claim 17, wherein the executable instructions that, prior to obtaining the second snapshot, cause the computer system to remove the virtual machine from a group of virtual machines being load-balanced.

20. The non-transitory computer-readable storage medium of claim 17, wherein the executable instructions that, prior to obtaining the second snapshot, cause the computer system to isolate the virtual machine from one or more networks.

21. The non-transitory computer-readable storage medium of claim 17, wherein at least a portion of memory of the virtual machine is immutable.

22. The non-transitory computer-readable storage medium of claim 17, wherein the set of virtual machines were instantiated from a common base image.

23. The non-transitory computer-readable storage medium of claim 17, wherein the virtual machine is restored to a physical computing system that is the same physical computing system the virtual machine was running on before the virtual machine was suspended.