Post-copy migration of a group of virtual machines that share memory

• US 10,552,230 B2
• Filed: 05/23/2016
• Issued: 02/04/2020
• Est. Priority Date: 11/18/2014
• Status: Active Grant

• Current Assignee: Red Hat Israel Ltd. (International Business Machines Corporation)
• Original Assignee: Red Hat Israel Ltd. (International Business Machines Corporation)
• Inventors: Tsirkin, Michael S., Gilbert, David A.
• Primary Examiner(s): Lee, James J
• Assistant Examiner(s): Chu Joy-Davila, Jorge A
• Application Number:
  US15/162,277
• Publication Number:
  US 20160266940A1
• Time in Patent Office: 1,352 Days
• CPC Class Codes: G06F 2009/4557 Distribution of virtual machine instances; Migration and load balancing
• CPC Class Codes: G06F 2209/5021 Priority
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects
• CPC Class Codes: G06F 9/5016 the resource being the memory
• CPC Class Codes: G06F 9/5088 involving task migration

Abstract

A hypervisor of a source host receives a request to migrate a group of virtual machines that provide network function virtualization support (NFV) from the source host to a destination host. The hypervisor of the source host determines that a first virtual machine of the group of virtual machines being migrated to the destination host shares a memory space on the source host with the group of virtual machines on the source host. Upon receiving a request from a second virtual machine of the group of virtual machines on the source host to access a first memory page of the shared memory space on the source host that has been migrated to the destination host, the hypervisor of the source host initiates migration of the second virtual machine to the destination host.

Claims

1. A method comprising:

determining, by a processing device executing a hypervisor on a source host, that a group of virtual machines undergoing live migration from the source host to a destination host share a memory space on the source host, wherein at least one shared memory page in the shared memory space is migrated simultaneously with the virtual machines undergoing live migration;

receiving, by the processing device executing the hypervisor on the source host, from a virtual machine of the group of virtual machines, a request to access a first shared memory page of the shared memory space on the source host;

determining that the first shared memory page of the shared memory space on the source host has been migrated by determining that a valid bit of a page table entry corresponding to the first memory page in a memory page table of a host operating system on the source host indicates that the first memory page has been designated as not present on the source host; and

responsive to determining that the first shared memory page of the shared memory space on the source host has been migrated to the destination host;

stopping, by the hypervisor of the source host, execution of the virtual machine on the source host, and

migrating, by the hypervisor of the source host, the virtual machine to the destination host, wherein a hypervisor of the destination host is configured to receive a request to access the first shared memory page from the virtual machine on the destination host and to grant access to the first shared memory page.

2. The method of claim 1, further comprising:

receiving, by the hypervisor of the source host, a request from the destination host for a second shared memory page of the shared memory space on the source host;

migrating, by the hypervisor of the source host, the second shared memory page of the shared memory space on the source host to the destination host; and

designating the second memory page of the shared memory space on the source host as not present.

3. The method of claim 2, further comprising:

responsive to determining that a number of shared memory pages of the shared memory space remaining on the source host meets a predetermined threshold condition, notifying, by the hypervisor of the source host, the destination host that shared memory transfer has completed.

4. The method of claim 1, further comprising:

starting, by the hypervisor of the destination host, the first virtual machine on the destination host;

receiving, by the hypervisor of the destination host, a request from the first virtual machine for a second shared memory page of the shared memory space;

accessing, by the hypervisor of the destination host, a page table entry for the second shared memory page within a memory page table of the destination host;

determining, by the hypervisor of the destination host, whether the second shared memory page is missing from the destination host in view of the page table entry for the second shared memory page; and

responsive to determining that memory space is missing from the destination host;

pausing the execution of the first virtual machine;

retrieving, by the hypervisor of the destination host, the second shared memory page from the source host; and

resuming, by the hypervisor of the destination host, execution of the first virtual machine.

5. The method of claim 4, wherein retrieving the second shared memory page comprises:

sending a request to the source host for the second shared memory page;

designating the second shared memory page on the destination host as having been requested;

receiving the second shared memory page from the source host; and

designating the second shared memory page on the destination host as present by updating the page table entry for the second shared memory page in the memory page table of the destination host.

6. The method of claim 4, wherein retrieving the second shared memory page comprises:

sending a request to the source host for the second shared memory page;

starting a timer upon sending the request to the source host for the second shared memory;

detecting an expiration of the timer prior to receiving the second shared memory page from the source host;

resending the request to the source host for the second shared memory page; and

receiving the second shared memory page from the source host.

7. A computing apparatus, comprising:

a memory; and

a processing device, operatively coupled to the memory, to execute a hypervisor on a source host to;

determine that a group of virtual machines undergoing live migration from the source host to a destination host share a memory space on the source host, wherein at least one shared memory page in the shared memory space is migrated simultaneously with the virtual machines undergoing live migration;

receive from a virtual machine of the group of virtual machines, a request to access a first shared memory page of the shared memory space on the source host;

determine that the first shared memory page of the shared memory space on the source host has been migrated, wherein the processing device to determine that a valid bit of a page table entry corresponding to the first memory page in a memory page table of a host operating system on the source host indicates that the first memory page has been designated as not present on the source host; and

responsive to determining that the first shared memory page of the shared memory space on the source host has been migrated to the destination host;

stop, by the hypervisor of the source host, execution of the virtual machine on the source host, and

migrate, by the hypervisor of the source host, the virtual machine to the destination host, wherein a hypervisor of the destination host is configured to receive a request to access the first shared memory page from the virtual machine on the destination host and to grant access to the first shared memory page.

8. The apparatus of claim 7, wherein the processing device is further to:

receive, by the hypervisor of the source host, a request from the destination host for a second shared memory page of the shared memory space on the source host;

migrate, by the hypervisor of the source host, the second shared memory page of the shared memory space on the source host to the destination host; and

designate, by the hypervisor of the source host, the second shared memory page of the shared memory space on the source host as not present.

9. The apparatus of claim 8, wherein the processing device is further to:

responsive to determining that a number of shared memory pages of the shared memory space remaining on the source host meets a predetermined threshold condition;

notify, by the hypervisor of the source host, the destination host that shared memory transfer has completed.

10. The apparatus of claim 7, wherein the virtual machine of the source host is one of a group of virtual machines being migrated to the destination host, and wherein the shared memory space is shared by the group of virtual machines.

11. The apparatus of claim 7, wherein the hypervisor of the destination host is to:

start the virtual machine on the destination host;

receive a request from the virtual machine for a second shared memory page of the shared memory space;

access a page table entry for the second shared memory page within a memory page table of the destination host;

determine whether the second shared memory page is missing from the destination host in view of the page table entry for the second shared memory page; and

responsive to determining that the second shared memory page of the shared memory space is missing from the destination host;

pause the execution of the virtual machine,

retrieve the second shared memory page from the source host, and

resume execution of the virtual machine.

12. The apparatus of claim 11, wherein to retrieve the second shared memory page, the hypervisor of the destination host is to:

send a request to the source host for second shared memory page;

designate the second shared memory page on the destination host as having been requested;

receive the second shared memory page from the source host; and

designate the second shared memory page on the destination host as present by updating the page table entry for the second shared memory page in the memory page table of the destination host.

13. The apparatus of claim 11, wherein to retrieve the second shared memory page, the hypervisor of the destination host is to:

send a request to the source host for second shared memory page;

start a timer upon sending the request to the source host for the second shared memory page;

detect an expiration of the timer prior to receiving the second shared memory page from the source host;

resend the request to the source host for the second shared memory page; and

receive the second shared memory page from the source host.

14. A non-transitory computer readable storage medium, having instructions stored therein, which when executed by a processing device of a computer system, cause the processing device to:

determine, by the processing device executing a hypervisor on a source host, that a group of virtual machines undergoing live migration from the source host to a destination host share a memory space on the source host, wherein at least one shared memory page in the shared memory space is migrated simultaneously with the virtual machines undergoing live migration;

receive, by the processing device executing the hypervisor on the source host, from a virtual machine of the group of virtual machines, a request to access a first shared memory page of the shared memory space on the source host;

determine, by the processing device executing the hypervisor on the source host, that the first shared memory page of the shared memory space on the source host has been migrated wherein the instructions to cause the processing device to determine that a valid bit of a page table entry corresponding to the first memory page in a memory page table of a host operating system on the source host indicates that the first memory page has been designated as not present on the source host; and

responsive to determining that the memory space on the source host has been migrated to the destination host;

stop, by the hypervisor of the source host, execution of the virtual machine on the source host, and

migrate, by the hypervisor of the source host, the virtual machine to the destination host, wherein a hypervisor of the destination host is configured to receive a request to access the first shared memory page from the virtual machine on the destination host and to grant access to the first shared memory page.

15. The non-transitory computer readable storage medium of claim 14, wherein the processing device is further to:

receive, by the hypervisor of the source host, a request from the destination host for a second shared memory page of the shared memory space on the source host;

migrate, by the hypervisor of the source host, the second shared memory page of the shared memory space on the source host to the destination host; and

designate, by the hypervisor of the source host, the second shared memory page of the shared memory space on the source host as not present.

16. The non-transitory computer readable storage medium of claim 14, wherein the hypervisor of the destination host is further to:

start the first virtual machine on the destination host;

receive a request from the first virtual machine for a second shared memory page of the shared memory space;

access a page table entry for the second shared memory page within a memory page table of the destination host;

determine whether the second shared memory page is missing from the destination host in view of the page table entry for the second shared memory page; and

responsive to determining that the second shared memory page of the shared memory space is missing from the destination host;

pause the execution of the first virtual machine,

retrieve the second shared memory page from the source host, and resume execution of the first virtual machine.

17. The non-transitory computer readable storage medium of claim 16, wherein to retrieve the second shared memory page, the hypervisor of the destination host is to:

send a request to the source host for the second shared memory page;

designate the second shared memory page on the destination host as having been requested;

receive the second shared memory page from the source host; and

designate the second shared memory page on the destination host as present by updating the page table entry for the second shared memory page in the memory page table of the destination host.

18. The non-transitory computer readable storage medium of claim 16, wherein to retrieve the second shared memory page, the hypervisor of the destination host is to:

send a request to the source host for second shared memory page;

start a timer upon sending the request to the source host for the second shared memory page;

detect an expiration of the timer prior to receiving the second shared memory page from the source host;

resend the request to the source host for the second shared memory page; and

receive the second shared memory page from the source host.