System and Method for Improving Memory Locality of Virtual Machines

• US 20090313445A1
• Filed: 06/11/2008
• Published: 12/17/2009
• Est. Priority Date: 06/11/2008
• Status: Active Grant

• Current Assignee: Vmware LLC (Broadcom, Inc.)
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: Pandey, Vivek, Agesen, Ole, Waldspurger, Carl, Garthwaite, Alex, Venkatasubramanian, Rajesh
• Granted Patent: US 8,037,280 B2
• Time in Patent Office: Days
• US Class Current: 711/162
• CPC Class Codes: G06F 2009/4557 Distribution of virtual machine instances; Migration and load balancing
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects
• CPC Class Codes: G06F 9/4856 resumption being on a different machine, e.g. task migration, virtual machine migration G06F9/5088 takes precedence
• CPC Class Codes: G06F 9/5033 considering data affinity

Abstract

A system and related method of operation for migrating the memory of a virtual machine from one NUMA node to another. Once the VM is migrated to a new node, migration of memory pages is performed while giving priority to the most utilized pages, so that access to these pages becomes local as soon as possible. Various heuristics are described to enable different implementations for different situations or scenarios.

Claims

1. A method for migrating a virtual machine and memory pages from a first node to a second node, comprising:

a. migrating the virtual machine from the first node to the second node;

b. monitoring memory access requests of the virtual machine and identifying priority pages for migration from the first node to the second node;

c. initiating page migration process and migrating at least a subset of the priority pages from the first node to the second node.

2. The method of claim 1, wherein steps b. and c. are repeated until occurrence of at least one event selected from:

a set number of steps b. and c. has been repeated;

fewer than a predetermined number of priority pages have been identified;

a set amount of time period has passed;

all of the pages have been migrated from the first node to the second node;

or

the virtual machine has been migrated away from the second node.

3. The method of claim 2, further comprising, upon termination of repeating steps b. and c., migrating any remaining pages according to one of:

random selection or fragmentation minimizing selection or sequential selection.

4. The method of claim 1, wherein step b. comprises monitoring hidden page faults and identifying faulting pages residing on the first node as priority pages.

5. The method of claim 4, wherein step a. further comprises discarding one of shadow page tables or second level page tables upon migrating the virtual machine to the second node.

6. The method of claim 5, further comprising discarding one of shadow page tables or second level page tables upon each completion of step c.

7. The method of claim 1, wherein in step c., a predetermined number of N pages is migrated and wherein:

when the number of priority pages comprises less than N, the method proceeds by migrating all the priority pages and selecting further pages from the first node; and

when the number of priority pages comprises more than N, the method proceeds by migrating only N priority pages.

8. The method of claim 7, wherein N may be dynamically varied.

9. The method of claim 1, wherein step b. comprises clearing access bits of memory page table entries (PTE) residing on the first node and thereafter identifying as a priority page each page having its corresponding PTE access bit set by an access operation.

10. The method of claim 9, further comprising clearing access bits of memory page table entries (PTE) residing on the first node after each completion of step c.

11. The method of claim 1, wherein step b. comprises clearing the P-bits of shadow page table entries that translate a guest linear address to a host physical address on pages residing on the first node.

12. The system of claim 1, wherein in step c. migration is initiated when:

a predetermined time period has elapsed;

or

the number of priority pages reaches a predetermined threshold.

28. The method of claim 1, further comprising, for each one of the pages corresponding to memory access requests, designating neighboring pages as priority pages.

13. A machine executable process for migrating a virtual machine and memory pages from a first node to a second node, comprising:

a. migrating the virtual machine from the first node to the second node;

b. setting a time period for periodic migration of pages from the first node to the second node and, during each period performing the steps;

b.1. monitoring memory access requests of the virtual machine and identifying priority pages corresponding to memory access requests mapped to pages on the first node;

b.2. when the time period has elapsed, initiating page migration process and migrating priority pages from the first node to the second node; and

,

b.3. determining whether the process should revert to step b.1. or terminate.

14. The process of claim 13, further comprising, when the process terminates at step b.3., performing page migration according to a secondary procedure.

15. The process of claim 14, wherein the secondary procedure comprises at least one of, random migration, minimizing defragmentation migration, and migrating pages having A-bit set.

16. The process of claim 13, wherein step b.1. comprises monitoring hidden page faults and identifying faulting pages residing on the first node as priority pages.

17. The process of claim 16, further comprising arranging the priority pages in a queue or a stack.

18. The process of claim 17, wherein step a. further comprises discarding one of shadow page tables or second level page tables upon migrating the virtual machine to the second node.

19. The process of claim 13, wherein step b.1. comprises clearing access bits of memory page table entries (PTE) residing on the first node and thereafter identifying each page having its corresponding PTE access bit set due to access operation as priority page.

29. The process of claim 13, wherein step b.1., further comprises for each one of the pages corresponding to memory access requests, designating neighboring pages as priority pages.

20. A computerized virtual machine comprising:

virtualized hardware;

virtual machine monitor;

a plurality of virtual pages;

page mapping tables storing mapping of virtual page numbers to guest physical page numbers;

a prioritizer listing priority pages having priority for migration.

21. The virtual machine of claim 20, wherein the prioritizer comprises a queue or a stack.

22. The virtual machine of claim 20, further comprising hidden page fault identifier, which provides faulting page identification to the prioritizer.

23. The virtual machine of claim 20, further comprising an access bit identifier that provides page identification to the prioritizer according to the access bit setting corresponding to the virtual pages.

24. A computer readable storage medium holding computer software for executing on the destination computer, the software embodying a method comprising:

upon receiving migration instruction for migrating a virtual machine from a first node to a second node, performing the steps;

a. migrating the virtual machine to the second node;

b. discarding page tables comprising mapping of virtual pages to machine pages;

c. setting a time period for periodic migration of pages from the first node to the second node and, during each period performing the steps;

c.1. whenever a memory access generates a hidden page fault, storing identity of faulting page addressed by the page call;

c.2. when the time period has elapsed, initiating page migration process and migrating N pages from the first node to the second node, wherein N comprises a natural number and the N pages include at least the faulting pages;

c.3. determining whether the process should revert to step c.1. or terminate.

25. The computer readable storage medium claim 24, wherein during at least one iteration of steps c.1. to c.3. the N pages comprise the faulting pages and randomly selected pages.

26. A computer readable storage medium holding computer software for executing on the destination computer, the software embodying a method comprising:

upon receiving migration instruction for migrating a virtual machine from a first node to a second node, performing the steps;

a. migrating the virtual machine to the second node;

b. setting a time period for periodic migration of pages from the first node to the second node and, during each period performing the steps;

b.1. clearing access bit of all pages;

b.2. when the time period has elapsed, checking the access bit of all pages, and listing accessed pages having their access bit set to indicate occurrence of page access since last clear;

b.3. initiating page migration process and migrating N pages from the first node to the second node, wherein N comprises a natural number and the N pages include at least the accessed pages;

b.4. determining whether the process should revert to step b.1. or terminate.

27. The computer readable storage medium of claim 26, wherein during at least one iteration of steps b.1. to b.4. the N pages comprise the accessed pages and randomly selected pages.