Online classification of memory pages based on activity level represented by one or more bits

• US 9,032,398 B2
• Filed: 07/12/2010
• Issued: 05/12/2015
• Est. Priority Date: 07/12/2010
• Status: Active Grant

• Current Assignee: Vmware LLC (Broadcom, Inc.)
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: Ahmad, Irfan, Waldspurger, Carl A., Garthwaite, Alexander Thomas, Tati, Kiran, Lu, Pin
• Primary Examiner(s): Lee, Thomas
• Assistant Examiner(s): Prifti, Aurel
• Application Number:
  US12/834,662
• Publication Number:
  US 20120011504A1
• Time in Patent Office: 1,765 Days
• Field of Search: 711/165, 711/170, 711/5, 711/159, 711/156, 711/105, 711/6, 711/171, 709/224, 713/323, 713/320, 713/160, 370/329, 718/1, 714/45
• US Class Current: 718/1
• CPC Class Codes: G06F 12/08 in hierarchically structured memory systems, e.g. virtual memory systems
• CPC Class Codes: G06F 12/10 Address translation

Abstract

Activity level of memory pages is classified in virtual machine environment, so that processes such as live VM migration and checkpointing, among others, can be carried out more efficiently. The method includes the steps of scanning page table entries of hypervisor-managed page tables continuously over repeating scan periods to determine whether memory pages have been accessed or not, and for each memory page, determining an activity level of the memory page based on whether the memory page has been accessed or not since a prior scan and storing the activity level of the memory page. The activity level of the memory page may be represented by one or more bits of its page table entry and may be classified as having at least two states ranging from hot to cold.

Claims

1. A method of classifying an activity level of memory pages that are accessed by one or more virtual machines executing in a host machine, comprising:

setting a scan rate that defines how frequently page table entries of hypervisor-managed page tables corresponding to the memory pages are to be scanned;

scanning the page table entries over multiple scan periods at the scan rate, wherein the page table entries are scanned once during a scan period; and

for each of the memory pages and at each scan period in the multiple scan periods, determining an activity level of the memory page using a finite state machine that includes a plurality of states, wherein, at each scan period, the finite state machine determines access information for the memory page at each scan period based on whether the memory page has been accessed since a prior scan period, the memory page being determined to have been accessed according to a setting of an accessed or dirty bit of the scanned page table entry of the memory page, and determines a state transition for the memory page to another state based on a current state of the memory page and the access information for the memory page as determined for each scan period; and

storing the activity level of the memory page.

2. The method according to claim 1, wherein the activity level of the memory page is one of high and low.

3. The method according to claim 2, wherein the activity level of the memory page is one of high, low, and at least one intermediate level between high and low.

4. The method according to claim 1, wherein the activity level of the memory page is determined based on whether the memory page has been accessed since a prior scan period and a currently stored activity level of the memory page.

5. The method according to claim 4, wherein the currently stored activity level of the memory page is changed from low to an intermediate level if the memory page has been accessed since the prior scan period, and from high to an intermediate level if the memory page has not been accessed since the prior scan period.

6. The method according to claim 5, wherein the currently stored activity level of the memory page is changed from an intermediate level to high if the memory page has been accessed since the prior scan period, and from an intermediate level to low if the memory page has not been accessed since the prior scan period.

7. The method according to claim 1, wherein the scan rate is set with respect to real time.

8. The method according to claim 1, wherein the memory pages are associated with a virtual machine and the scan rate is set with respect to a virtual execution time of the virtual machine.

9. The method according to claim 1, wherein the scan rate is dynamically adjustable.

10. The method according to claim 1, wherein the page table entries are scanned in order from physical page number 0 to physical page number N−

1, where N is the number of memory pages scanned during one scan period.

11. The method according to claim 1, wherein the activity level of the memory page is stored in the page table entry corresponding to the memory page.

12. The method according to claim 1, wherein the page table entries are scanned according to an order defined in a data structure.

13. The method according to claim 12, wherein the data structure includes one or more linked-list data structures.

14. The method according to claim 13, wherein the memory pages having a first activity level have page table entries referenced in a first linked-list data structure and the memory pages having a second activity level have page table entries referenced in a second linked-list data structure.

15. The method according to claim 12, wherein the data structure includes a linked-list data structure, and each entry in the linked-list data structure corresponds to a memory page and includes an indication of the currently stored activity level of the memory page and a pointer to the next page table entry to be scanned.

16. The method according to claim 1, wherein the activity level of a memory page is determined by the finite state machine examining whether the accessed bit of the page table entry corresponding to the memory page is set.

17. The method according to claim 16, further comprising:

clearing the accessed bit of the page table entry corresponding to the memory page after determining the activity level of the memory page.

18. The method according to claim 17, further comprising:

clearing any cached state of the accessed bit of the page table entry corresponding to the memory page after determining the activity level of the memory page.

19. The method according to claim 1, wherein the activity level of a memory page is determined by the finite state machine examining whether the dirty bit of the page table entry corresponding to the memory page is set.

20. The method according to claim 19, further comprising:

clearing the dirty bit of the page table entry corresponding to the memory page after determining the activity level of the memory page.

21. The method according to claim 20, further comprising:

clearing any cached state of the dirty bit of the page table entry corresponding to the memory page after determining the activity level of the memory page.

22. The method according to claim 1, wherein the activity level of the memory page is determined based on whether the memory page has been read from or written to the prior scan period.

23. The method according to claim 1, wherein the activity level of the memory page is determined based on whether the memory page has been written to since the prior scan period.

24. The method according to claim 1, wherein functions of scanning the page table entries over multiple scan periods at the scan rate and determining, for each of the memory pages and at each scan period, the activity level of the memory page, are implemented in hardware, and said scanning and said determining are performed by said hardware.

25. The method of claim 14, wherein the memory pages having the first activity level have page tables entries referenced in the first linked-list data structure and are scanned at a first scan rate, and the memory pages having the second activity level have page tables entries referenced in the second linked-list data structure and are scanned at a second scan rate.

26. The method of claim 1, wherein:

the state transition for the memory page is limited to a set of pre-defined transitions from the current state to another state in the plurality of states, and

each state in the plurality of states has a different set of pre-defined transitions.

27. A computer system including one or more virtual machines running therein, and a hypervisor for managing page tables that provide mappings from guest physical memory pages to machine memory pages, wherein the hypervisor is programmed to:

scan entries of the pages tables once over repeating scan periods in accordance with a scan rate and determine an activity level of a machine memory page corresponding to the scanned page table entry using a finite state machine that includes a plurality of states, wherein, at each scan period, the finite state machine determines access information for the memory page at each scan period based on whether an access or dirty bit of the scanned page table entry of the memory page is set and determines a state transition to another state for the machine memory page based on a current state of the machine memory page and the access information for the machine memory page as determined whether the access or dirty bit of the scanned page table entry is set for each scan period; and

store the activity level of the machine memory page.

28. The computer system of claim 27, wherein the hypervisor is programmed with the finite state machine for determining the activity level of the machine memory page.

29. The computer system of claim 28, wherein the activity level of the machine memory page is one of a low state, a high state, and first, second, and third intermediate states, and the finite state machine transitions the current state of the machine memory page to the another state based on whether an access or dirty bit of a scanned page table entry corresponding to the machine memory page is set.

30. The computer system of claim 28, wherein if the access or dirty bit is set, the state transitions include low to second intermediate, first intermediate to high, second intermediate to third intermediate, and third intermediate to high, and if the access or dirty bit is not set, the state transitions include high to third intermediate, third intermediate to first intermediate, second intermediate to first intermediate, and first intermediate to low.

31. A non-transitory computer-readable storage medium comprising instructions, which when executed in a computer system including a hypervisor for managing page tables that provide mappings from guest physical memory pages to machine memory pages, causes the computer system to carry out the steps of:

scanning the page table entries of hypervisor-managed page tables over multiple scan periods according to a pre-defined scan rate, wherein the page table entries are scanned once during a scan period; and

for each of the memory pages and at each scan period in the multiple scan periods, determining an activity level of the memory page using a finite state machine that includes a plurality of states, wherein, at each scan period, the finite state machine determines access information for the memory page at each scan period based on whether the memory page has been accessed since a prior scan period, the memory page being determined to have been accessed according to a setting of an accessed or dirty bit of the scanned page table entry of the memory page, and determines a state transition for the memory page to another state based on a current state of the memory page and the access information for the memory page as determined for each scan period; and

storing the activity level of the memory page.

32. The non-transitory computer-readable storage medium of claim 31, wherein the activity level of the memory page is determined based on whether the memory page has been accessed since the prior scan period and the current state which is a currently stored activity level of the memory page.

33. The non-transitory computer-readable storage medium of claim 32, wherein the activity level of the memory page is one of high and low.

34. The non-transitory computer-readable storage medium of claim 32, wherein the activity level of the memory page is one of high, low, and at least one intermediate level between high and low.

35. The non-transitory computer-readable storage medium of claim 31, wherein the activity level of the memory page is stored in the page table entry corresponding to the memory page.

36. The non-transitory computer-readable storage medium of claim 31, wherein the memory pages having a first activity level are referenced in a first data structure and the memory pages having a second activity level are referenced in a second data structure, and the step of storing the activity level of the memory page includes moving the reference of the memory page between the first and second data structures.