Live migration of virtual machines that use externalized memory pages

• US 9,342,346 B2
• Filed: 07/27/2014
• Issued: 05/17/2016
• Est. Priority Date: 07/27/2014
• Status: Active Grant

• Current Assignee: Mellanox Technologies Limited (NVIDIA Corporation)
• Original Assignee: Stratoscale Ltd.
• Inventors: Maislos, Ariel, Bogner, Etay, Ben-Yehuda, Muli, Matichin, Shlomo, Hudzia, Benoit Guillaume Charles, Freiman, Rom
• Primary Examiner(s): DORAIS, CRAIG C
• Application Number:
  US14/341,813
• Publication Number:
  US 20160026489A1
• Time in Patent Office: 660 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 12/1009 using page tables, e.g. page table structures
• CPC Class Codes: G06F 12/1072 Decentralised address translation, e.g. in distributed shared memory systems
• CPC Class Codes: G06F 15/167 using a common memory, e.g. mailbox
• CPC Class Codes: G06F 2009/4557 Distribution of virtual machine instances; Migration and load balancing
• CPC Class Codes: G06F 2009/45583 Memory management, e.g. access or allocation
• CPC Class Codes: G06F 2009/45595 Network integration; Enabling network access in virtual machine instances
• CPC Class Codes: G06F 2212/151 Emulated environment, e.g. virtual machine
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects

Abstract

A method includes running a Virtual Machine (VM) on a first compute node in a plurality of compute nodes that communicate with one another over a communication network. The VM is migrated from the first compute node to a second compute node in the plurality by generating, for memory pages accessed by the VM, page transfer state of one or more local memory pages that are accessed locally on the first compute node, and of one or more externalized memory pages whose access is not confined to the first node. Based on the page transfer state, the migrated VM is provided with access to the memory pages, including both the local and the externalized memory pages, on the second compute node.

Claims

1. A method, comprising:

running a Virtual Machine (VM) on a first compute node in a plurality of compute nodes that communicate with one another over a communication network; and

migrating the VM from the first compute node to a second compute node in the plurality, by;

generating, for memory pages accessed by the VM, page transfer state of one or more local memory pages that are accessed locally on the first compute node, and of one or more externalized memory pages whose access is not confined to the first node;

based on the page transfer state, providing for the migrated VM access to the memory pages, including both the local and the externalized memory pages, on the second compute node; and

transferring from the first compute node to the second compute node contents or unique identifiers of the local memory pages, further transferring from the first compute node to the second compute node unique identifiers of the externalized memory pages after stopping the VM on the first compute node, and transferring remaining memory pages after starting a migrated instance of the VM on the second compute node.

2. The method according to claim 1, wherein generating the page transfer state comprises assigning for each memory page a state selected from a group of states consisting of:

a DIRTY state for a local page that has changed or for an externalized page that has been retrieved back to the first compute node after the page has been transferred to the second compute node;

an EXTERNALIZED state for a page that resides outside the first compute node; and

a TRANSFERRED state for a page that has been successfully transferred to the second compute node.

3. The method according to claim 1, wherein transferring the remaining memory pages comprises requesting the remaining memory pages in response to page-fault events occurring in the second compute node.

4. The method according to claim 1, wherein transferring the remaining memory pages comprises transferring the remaining memory pages in a background process.

5. The method according to claim 1, wherein migrating the VM comprises transferring at least part of the page transfer state from the first compute node to the second compute node before stopping the VM.

6. The method according to claim 1, wherein migrating the VM comprises transferring at least some of the contents or unique identifiers of the local memory pages, and unique identifiers of the externalized memory pages, using at least one of a push background process in the first compute node and a pull background process in the second compute node.

7. A method, comprising:

running a Virtual Machine (VM) on a first compute node in a plurality of compute nodes that communicate with one another over a communication network; and

migrating the VM from the first compute node to a second compute node in the plurality, by;

generating, for memory pages accessed by the VM, page transfer state of one or more local memory pages that are accessed locally on the first compute node, and of one or more externalized memory pages whose access is not confined to the first node;

based on the page transfer state, providing for the migrated VM access to the memory pages, including both the local and the externalized memory pages, on the second compute node;

transferring from the first compute node to the second compute node contents or unique identifiers of the local memory pages, and unique identifiers of the externalized memory pages,

wherein migrating the VM comprises transferring at least some of the memory pages while the VM is running on the first compute node, and transferring remaining externalized pages before starting a migrated instance of the VM on the second compute node.

8. A system comprising a plurality of compute nodes that communicate with one another over a communication network, including at least first and second compute nodes comprising respective processors, wherein the processors are configured to run a Virtual Machine (VM) on the first compute node and to migrate the VM from the first compute node to the second compute node, by:

generating, for memory pages accessed by the VM, page transfer state of one or more local memory pages that are accessed locally on the first compute node, and of one or more externalized memory pages whose access is not confined to the first node; and

based on the page transfer state, providing for the migrated VM access to the memory pages, including both the local and the externalized memory pages, on the second compute node,

wherein the processors are further configured to transfer from the first compute node to the second compute node contents or unique identifiers of the local memory pages, and to further transfer from the first compute node to the second compute node unique identifiers of the externalized memory pages after stopping the VM on the first compute node, and to transfer remaining memory pages after starting a migrated instance of the VM on the second compute node.

9. The system according to claim 8, wherein the processors are configured to assign for each memory page a state selected from a group of states consisting of:

a DIRTY state for a local page that has changed or for an externalized page that has been retrieved back to the first compute node after the page has been transferred to the second compute node;

an EXTERNALIZED state for a page that resides outside the first compute node; and

a TRANSFERRED state for a page that has been successfully transferred to the second compute node.

10. The system according to claim 8, wherein the processors are configured to request the remaining memory pages in response to page-fault events occurring in the second compute node.

11. The system according to claim 8, wherein the processors are configured to transfer the remaining memory pages in a background process.

12. The system according to claim 8, wherein the processors are configured to transfer at least part of the page transfer state from the first compute node to the second compute node before stopping the VM.

13. The system according to claim 8, wherein the processors are configured to migrate the VM by transferring at least some of the contents or unique identifiers of the local memory pages, and unique identifiers of the externalized memory pages, using at least one of a push background process in the first compute node and a pull background process in the second compute node.

14. A system comprising a plurality of compute nodes that communicate with one another over a communication network, including at least first and second compute nodes comprising respective processors, wherein the processors are configured to run a Virtual Machine (VM) on the first compute node and to migrate the VM from the first compute node to the second compute node, by:

generating, for memory pages accessed by the VM, page transfer state of one or more local memory pages that are accessed locally on the first compute node, and of one or more externalized memory pages whose access is not confined to the first node; and

based on the page transfer state, providing for the migrated VM access to the memory pages, including both the local and the externalized memory pages, on the second compute node, wherein the processors are further configured to transfer from the first compute node to the second compute node contents or unique identifiers of the local memory pages, and unique identifiers of the externalized memory pages, to transfer at least some of the memory pages while the VM is running on the first compute node, and to transfer remaining externalized pages before starting a migrated instance of the VM on the second compute node.

15. A computer software product, the product comprising a tangible non-transitory computer-readable medium in which program instructions are stored, which instructions, when read by processors of first and second compute nodes that communicate with one another over a communication network, cause the processors to run a Virtual Machine (VM) on the first compute node and to migrate the VM from the first compute node to the second compute node, by:

generating, for memory pages accessed by the VM, page transfer state of one or more local memory pages that are accessed locally on the first compute node, and of one or more externalized memory pages whose access is not confined to the first node;

based on the page transfer state, providing for the migrated VM access to the memory pages, including both the local and the externalized memory pages, on the second compute node; and

transferring from the first compute node to the second compute node contents or unique identifiers of the local memory pages, further transferring from the first compute node to the second compute node unique identifiers of the externalized memory pages after stopping the VM on the first compute node, and transferring remaining memory pages after starting a migrated instance of the VM on the second compute node.