Reliable distributed messaging using non-volatile system memory

• US 9,740,606 B1
• Filed: 11/01/2013
• Issued: 08/22/2017
• Est. Priority Date: 11/01/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: McKelvie, Samuel James, Gupta, Anurag Windlass
• Primary Examiner(s): Bertram, Ryan
• Assistant Examiner(s): Blust, Jason
• Application Number:
  US14/070,362
• Time in Patent Office: 1,390 Days
• Field of Search: None
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 12/0238 Memory management in non-volatile memory, e.g. resistive RAM or ferroelectric memory
• CPC Class Codes: G06F 12/0868 Data transfer between cache memory and other subsystems, e.g. storage devices or host systems
• CPC Class Codes: G06F 12/0871 Allocation or management of cache space
• CPC Class Codes: G06F 2212/1024 Latency reduction
• CPC Class Codes: G06F 2212/1032 Reliability improvement, data loss prevention, degraded operation etc
• CPC Class Codes: G06F 2212/152 Virtualized environment, e.g. logically partitioned system
• CPC Class Codes: G06F 2212/205 Hybrid memory, e.g. using both volatile and non-volatile memory
• CPC Class Codes: G06F 2212/311 In host system
• CPC Class Codes: G06F 2212/656 Address space sharing
• CPC Class Codes: G06F 3/0607 by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device
• CPC Class Codes: G06F 3/0655 Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices

Abstract

Methods and apparatus for reliable distributed messaging are described. A computer system includes a system memory coupled to one or more processors. The system memory comprises at least a non-volatile portion. A particular location within the non-volatile portion is designated as a target location to which a sender module participating in a communication protocol is granted write permission. A receiver module participating in the communication protocol, subsequent to a failure event that results in a loss of data stored in a volatile portion of the system memory, reads a data item written by the sender program at the target location prior to the failure event. The receiver module performs an operation based on contents of the data item.

Claims

1. A system, comprising:

one or more processors, and

a system memory for the one or more processors, wherein the system memory is linked to the one or more processors and comprises one or more non-volatile memory modules and one or more volatile memory modules;

a persistent storage device having a higher access latency than the system memory;

wherein the system memory comprises program instructions that when executed on the one or more processors;

implement a shared-memory based communication protocol for communications between at least a source program and at least a destination program in accordance with which the source program is granted at least a write permission to one or more target locations within the system memory, and the destination program is granted at least a read permission to the one or more target locations;

wherein the shared-memory based communication protocol allows the source program or the destination program to request, in a system call to an operating system, whether each of the one or more target locations are to be memory mapped in a volatile memory space or a non-volatile memory space;

map, by the operating system in response to the system call, the one or more target locations into the memory space requested in the system call, wherein the volatile memory space is mapped to at least one of the one or more volatile memory modules, and the non-volatile memory space is mapped to at least one of the one or more non-volatile memory modules; and

write, by the source program, a data item to a particular target location, and the particular target location is mapped to a non-volatile memory module of the one or more non-volatile memory modules;

read, by the destination program from the particular target location, subsequent to a failure event that results in a loss of data stored in the one or more volatile modules of the system memory, the data item written by the source program at the particular target location prior to the failure event; and

perform, by the destination program, one or more operations based at least in part on contents of the data item.

2. The system as recited in claim 1, wherein the one or more non-volatile memory modules of the system memory comprise a non-volatile dual inline memory module (NV-DIMM).

3. The system as recited in claim 1, wherein the source program is executed at a different computing device than the destination program, and wherein the source program transmits the data item to the particular target location via a network link.

4. The system as recited in claim 1, wherein the communication protocol comprises an RDMA (remote direct memory access) protocol.

5. The system as recited in claim 1, wherein the communication protocol comprises a store-and-forward messaging protocol, wherein the destination program is configured to transmit at least a portion of the data item to a different destination program.

6. A method, comprising:

performing, by one or more computing devices comprising one or more respective persistent storage devices and one or more respective processors linked to a respective system memory comprising one or more non-volatile memory modules and one or more volatile memory modules, the system memory having a lower access latency than the respective persistent storage device;

implementing a shared-memory based communication protocol for communications between at least a sender module and at least a receiver module in accordance with which the sender module is granted at least a write permission to one or more target locations within the system memory, and the receiver module is granted at least a read permission to the one or more target locations;

wherein the shared-memory based communication protocol allows the sender module or the receiver module to request, in a system call to an operating system, whether each of the one or more target locations are to be memory mapped in a volatile memory space or a non-volatile memory space;

mapping, by the operating system in response to the system call, the one or more target locations into the memory space requested in the system call, wherein the volatile memory space is mapped to at least one of the one or more volatile memory modules and the non-volatile memory space is mapped to at least one of the one or more non-volatile memory modules;

writing, by the sender module, a data item to a particular target location, wherein the particular target location is mapped to a non-volatile memory module of the one or more non-volatile memory modules;

reading, by the receiver module from the target location in accordance with the communication protocol, subsequent to a failure event that results in a loss of data stored in a volatile portion of the system memory, the data item written by the sender module at the particular location in accordance with the communication protocol prior to the failure event; and

performing, by the receiver module, one or more operations based at least in part on contents of the data item.

7. The method as recited in claim 6, wherein the non-volatile portion of the system memory comprises resistive memory.

8. The method as recited in claim 6, wherein the non-volatile portion of the system memory comprises phase-change memory.

9. The method as recited in claim 6, wherein the system memory is part of a particular computing device of the one or more computing devices, and wherein the sender module is executed at a different computing device than the particular computing device, the method further comprising performing, by the one or more computing devices:

receiving the data item at the particular computing device over a network link from the different computing device.

10. The method as recited in claim 6, wherein the communication protocol comprises an RDMA (remote direct memory access) protocol.

11. The method as recited in claim 6, wherein the communication protocol comprises a store-and-forward messaging protocol, wherein performing said one or more operations based at least in part on contents of the data item comprises transmitting the data item to a different receiver module.

12. The method as recited in claim 6, wherein the receiver module comprises an application running in user mode, further comprising performing, at the one or more computing devices:

generating, in response to a system call, a memory map of a subset of the non-volatile portion of the system memory within an address space of the application.

13. The method as recited in claim 6, wherein the receiver module comprises a component of one of:

(a) a Domain Name System (DNS) service, or (b) a directory service.

14. The method as recited in claim 6, further comprising performing, by the one or more computing devices:

providing a write acknowledgement to the sender module;

wherein said performing the one or more operations is asynchronous with respect to said providing the write acknowledgment.

15. The method as recited in claim 6, further comprising performing, by the one or more computing devices:

setting, by the receiver module, a read-complete indicator in the non-volatile portion of the system memory to indicate to the sender module that at least a portion of the target location is available for writing a different data item.

16. The method as recited in claim 6, further comprising performing, by the one or more computing devices:

disabling, subsequent to the failure event, write access to at least a sub-portion of the non-volatile portion of the system memory until the data item has been read by the receiver module.

17. A non-transitory computer-accessible storage medium storing program instructions that execute on one or more computing devices comprising one or more processors that are linked to system memory, the system memory comprising one or more non-volatile memory modules and one or more volatile memory modules, the system memory having a lower access latency than a persistent storage device of the system, the program instructions executable on the one or more computing devices to:

implement a shared-memory based communication protocol for communications between at least a sender module and at least a receiver module in accordance with which the sender module is granted at least a write permission to one or more target locations within the system memory and the receiver module is granted at least a read permission to the one or more target locations,

wherein the shared-memory based communication protocol allows the sender module or the receiver module to request, in a system call to an operating system, whether each of the one or more target locations are to be memory mapped in a volatile memory space or a non-volatile memory space;

map, by the operating system in response to the system call, the one or more target locations into the memory space requested in the system call, wherein the volatile memory space is mapped to at least one of the one or more volatile memory modules and the non-volatile memory space is mapped to at least one of the one or more non-volatile memory modules;

write, by the sender module, a data item to a particular location, wherein the particular location is mapped to a non-volatile memory module of the one or more non-volatile memory modules;

read, by the receiver module from the particular location in accordance with the communication protocol, subsequent to a failure event that results in a loss of data stored in the volatile memory space of the system memory, the data item written by the sender module in accordance with the communication protocol at the particular location prior to the failure event; and

perform, by the receiver module, one or more operations based at least in part on contents of the data item.

18. The non-transitory computer-accessible storage medium as recited in claim 17, wherein the non-volatile portion of the system memory comprises racetrack memory.

19. The non-transitory computer-accessible storage medium as recited in claim 17, wherein the program instructions, when executed on the one or more computing devices:

receive the data item over a network link.

20. The non-transitory computer-accessible storage medium as recited in claim 17, wherein the communication protocol comprises an RDMA (remote direct memory access) protocol.

21. The non-transitory computer-accessible storage medium as recited in claim 17, wherein the communication protocol comprises a store-and-forward messaging protocol, wherein to perform said one or more operations based at least in part on contents of the data item, the program instructions when executed on the one or more computing devices transmit the data item to a different receiver module.