Consistent and efficient mirroring of nonvolatile memory state in virtualized environments where dirty bit of page table entries in non-volatile memory are not cleared until pages in non-volatile memory are remotely mirrored

• US 10,223,026 B2
• Filed: 09/30/2013
• Issued: 03/05/2019
• Est. Priority Date: 09/30/2013
• Status: Active Grant

• Current Assignee: Vmware LLC (Broadcom, Inc.)
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: Subrahmanyam, Pratap, Venkatasubramanian, Rajesh
• Primary Examiner(s): Lo, Ann J
• Assistant Examiner(s): Yew, Chie
• Application Number:
  US14/042,288
• Publication Number:
  US 20150095585A1
• Time in Patent Office: 1,982 Days
• CPC Class Codes: G06F 11/14 Error detection or correction of the data by redundancy in operation G06F11/16 takes precedence
• CPC Class Codes: G06F 11/1484 involving virtual machines
• CPC Class Codes: G06F 11/1666 where the redundant component is memory or memory area
• CPC Class Codes: G06F 11/20 using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups

Abstract

Updates to nonvolatile memory pages are mirrored so that certain features of a computer system, such as live migration of applications, fault tolerance, and high availability, will be available even when nonvolatile memory is local to the computer system. Mirroring may be carried out when a cache flush instruction is executed to flush contents of the cache into nonvolatile memory. In addition, mirroring may be carried out asynchronously with respect to execution of the cache flush instruction by retrieving content that is to be mirrored from the nonvolatile memory using memory addresses of the nonvolatile memory corresponding to target memory addresses of the cache flush instruction.

Claims

1. In a computer system having one or more processors, volatile memory, nonvolatile memory, and a processor cache, a method of mirroring updates to the nonvolatile memory, comprising:

determining that an instruction to flush the processor cache into the nonvolatile memory has been issued, wherein the processor cache contains cached data and memory addresses of both the volatile memory and the non-volatile memory, and a memory space of the volatile memory and the nonvolatile memory is logically partitioned into pages;

in response to the issuance of the flush instruction when dirty contents of pages of the nonvolatile memory are cached in cache lines of the processor cache such that a dirty indication in each of page table entries of the pages of the nonvolatile memory is set, writing the dirty contents of the pages of the nonvolatile memory that are cached in the cache lines of the processor cache into the pages of the nonvolatile memory at corresponding memory addresses of the nonvolatile memory stored in the cache lines; and

transmitting at least the cache lines of the processor cache that have been written into the nonvolatile memory to a remote computer system,

wherein after said writing and prior to said transmitting, not clearing the dirty indication in each of the pages table entries of the pages of the nonvolatile memory, and responsive to said transmitting, clearing the dirty indication in each of the page table entries of the pages of the nonvolatile memory.

2. The method of claim 1, wherein upon completion of said writing and in response thereto, a processing thread that carries out said transmitting is executed.

3. The method of claim 1, wherein said writing and said transmitting are executed in independent processing threads.

4. The method of claim 3, further comprising:

halting an application that issued the flush instruction; and

after said transmitting, resuming the application.

5. The method of claim 4, wherein the processing thread that carries out said transmitting is executed periodically.

6. The method of claim 5, further comprising:

each time the processing thread completes execution, changing a counter value, wherein the application is resumed according to the changed counter value.

7. The method of claim 1, wherein the processor cache includes a plurality of clean or dirty cache lines each configured to store the cache data and address corresponding to the memory addresses of the non-volatile memory.

8. The method of claim 7, wherein said transmitting includes:

identifying the pages of the nonvolatile memory that are indicated as having been modified; and

transmitting at least the modified portions of the identified pages of the non-volatile memory to the remote computer system.

9. The method of claim 8, wherein said transmitting includes transmitting the modified portions of the identified pages of the nonvolatile memory to the remote computer system without transmitting unmodified portions of the identified pages of the nonvolatile memory to the remote computer system.

10. The method of claim 7, further comprising:

each time a clean cache line of the processor cache is updated with the cached data and a memory address of the memory addresses of the nonvolatile memory, storing the memory address of the memory addresses of the nonvolatile memory in a buffer,

wherein said transmitting includes reading memory addresses stored in the buffer, reading data from the nonvolatile memory at the memory addresses stored in the buffer and transmitting the data read from the nonvolatile memory at the memory addresses stored in the buffer to the remote system.

11. The method of claim 10, wherein said transmitting is triggered when the buffer becomes full.

12. The method of claim 10, wherein said transmitting is triggered periodically.

13. The method of claim 1, further comprising:

upon said writing, storing memory addresses of the nonvolatile memory at which the dirty contents are written, in a buffer,

wherein said transmitting includes reading the memory addresses stored in the buffer, reading the dirty contents from the nonvolatile memory at the memory addresses stored in the buffer, and transmitting the dirty contents read from the nonvolatile memory at the memory addresses stored in the buffer to the remote computer system.

14. The method of claim 13, wherein said transmitting is triggered when the buffer becomes full.

15. The method of claim 13, wherein said transmitting is triggered periodically.

16. A non-transitory computer readable medium comprising instructions to be executed in a computer system having one or more processors, volatile memory, nonvolatile memory, and a processor cache to perform a method of mirroring updates to the nonvolatile memory, comprising the steps of:

determining that an instruction to flush the processor cache into the nonvolatile memory has been issued, wherein the processor cache contains cached data and memory addresses of both the volatile memory and the non-volatile memory, and a memory space of the volatile memory and the nonvolatile memory is logically partitioned into pages;

in response to the issuance of the flush instruction when dirty contents of pages of the nonvolatile memory are cached in cache lines of the processor cache such that a dirty indication in each of page table entries of the pages of the nonvolatile memory is set, writing the dirty contents of the pages of the nonvolatile memory that are cached in the cache lines of the processor cache into the pages of the nonvolatile memory at corresponding memory addresses of the nonvolatile memory stored in the cache lines; and

transmitting at least the cache lines of the processor cache that have been written into the nonvolatile memory to a remote computer system,

wherein after said writing and prior to said transmitting, not clearing the dirty indication in each of the pages table entries of the pages of the nonvolatile memory, and responsive to said transmitting, clearing the dirty indication in each of the page table entries of the pages of the nonvolatile memory.

17. The non-transitory computer readable medium of claim 16, wherein upon completion of said writing and in response thereto, a processing thread that carries out said transmitting is executed.

18. The non-transitory computer readable medium of claim 16, wherein said writing and said transmitting are executed in independent processing threads.

19. A computer system comprising:

one or more processors;

volatile memory;

nonvolatile memory; and

a processor cache, wherein the processor cache contains cached data and memory addresses of both the volatile memory and the non-volatile memory, and a memory space of the volatile memory and the nonvolatile memory is logically partitioned into pages, and

wherein the one or more processors are programmed to execute a method of mirroring updates to the nonvolatile memory, comprising the steps of;

determining that an instruction to flush the processor cache into the nonvolatile memory has been issued;

in response to the issuance of the flush instruction when dirty contents of pages of the nonvolatile memory are cached in cache lines of the processor cache such that a dirty indication in each of page table entries of the pages of the nonvolatile memory is set, writing the dirty contents of the pages of the nonvolatile memory that are cached in the cache lines of the processor cache into the pages of the nonvolatile memory at corresponding memory addresses of the nonvolatile memory stored in the cache lines; and

transmitting at least the cache lines of the processor cache that have been written into the nonvolatile memory to a remote computer system,

wherein after said writing and prior to said transmitting, not clearing the dirty indication in each of the pages table entries of the pages of the nonvolatile memory, and responsive to said transmitting, clearing the dirty indication in each of the page table entries of the pages of the nonvolatile memory.

20. The computer system of claim 19, wherein the volatile memory includes a buffer for storing memory addresses of the nonvolatile memory at which dirty lines of the processor cache are stored.