Direct access persistent memory shared storage

• US 10,223,326 B2
• Filed: 07/31/2013
• Issued: 03/05/2019
• Est. Priority Date: 07/31/2013
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Frank, Richard, Kaimalettu, Arun
• Primary Examiner(s): Winder, Patrice L
• Assistant Examiner(s): de Rodriguez, Angela Widhalm
• Application Number:
  US13/956,096
• Publication Number:
  US 20150039712A1
• Time in Patent Office: 2,043 Days
• Field of Search: 709212, 709213, 709214, 709215
• CPC Class Codes: G06F 13/28 using burst mode transfer, e.g. direct memory access DMA, cycle steal G06F13/32 takes precedence
• CPC Class Codes: G06F 15/167 using a common memory, e.g. mailbox
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/133 Protocols for remote procedure calls [RPC]

Abstract

Techniques are described for providing one or more remote nodes with direct access to persistent random access memory (PRAM). In an embodiment, registration information is generated for a remote direct access enabled network interface controller (RNIC). The registration information associates an access key with a target region in PRAM. The access key is sent to a remote node of the one or more nodes. The RNIC may subsequently receive a remote direct memory access (RDMA) message from the remote node that includes the access key. In response to the RDMA message, the RNIC performs a direct memory access within the target region of PRAM.

Claims

1. A method for providing one or more database servers with direct access to at least part of a database stored persistently in a storage server, the method comprising:

generating, for a remote direct access memory enabled network interface controller (RNIC) of said storage server, registration information that associates an access key with a target region in persistent random access memory (PRAM) of said storage server;

storing the registration information in a cache that is associated with the RNIC;

sending the access key to a database server of the one or more database servers;

receiving, by the RNIC, a remote direct memory access (RDMA) message from the database server that includes the access key;

wherein the RDMA message identifies, based at least on the access key, a memory location within the target region where a data object within the database is persistently stored;

determining, by the RNIC using the registration information and based at least in part on the access key, an address within the target region in PRAM;

performing, by the RNIC, a direct memory access at the address within the target region of PRAM without communicating with a storage controller for the storage server while performing said direct memory access.

2. The method of claim 1, wherein the RDMA message includes a data payload and identifies, based on the access key, a memory location within the target region in PRAM for persistently storing the data payload.

3. The method of claim 2, the method further comprising:

receiving, from the database server by the RNIC, an RDMA read message that identifies a final portion of the data payload;

returning, to the database server by the RNIC, an RDMA read response message that includes the final portion of the data payload;

wherein the RDMA read response message confirms that the data payload has been persistently stored.

4. The method of claim 1, the method further comprising returning, by the RNIC, the data object to the database server in an RDMA read response message;

wherein the RDMA read response message references a buffer in the database server where the data object should be placed.

5. The method of claim 1,

wherein the RDMA message is a first RDMA message that identifies, based on the access key, a first memory location within the target region of PRAM;

wherein the direct memory access is to the first memory location within the target region of PRAM;

the method further comprising;

receiving, by the RNIC, a second RDMA message from the database server that includes the access key and identifies a second memory location within the target region of PRAM;

performing, by the RNIC, a second direct memory access to the second memory location within the target region of PRAM.

6. The method of claim 1,

wherein the direct memory access is a first direct memory access,

wherein the registration information associates a second access key with the target region in PRAM,

the method further comprising:

sending the second access key to a second database server of the one or more database servers;

receiving, at the RNIC, a second RDMA message from the second database server that includes the second access key;

in response to receiving, at the RNIC, the second RDMA message from the second database server that includes the second access key, performing, by the RNIC, a second direct memory access within the target region of PRAM.

7. The method of claim 6, wherein the second direct memory access is to said memory location in the target region of PRAM accessed by the first direct memory access.

19. The method of claim 1, wherein determining, by the RNIC using the registration information and based at least in part on the access key, an address within the target region in PRAM includes determining, by the RNIC using the registration information and based at least in part on the access key and the memory location, an address within the target region in PRAM.

8. One or more non-transitory computer-readable media storing sequences of instructions that, in response to being executed by one or more processors, cause:

generating, for a remote direct access memory enabled network interface controller (RNIC) of a storage server, registration information that associates an access key with a target region in persistent random access memory (PRAM) of the storage server;

wherein the PRAM stores at least part of a database;

storing the registration information in a cache that is associated with the RNIC;

sending the access key to a database server of one or more database servers;

receiving, at the RNIC, a remote direct memory access (RDMA) message from the database server that includes the access key;

wherein the RDMA message identifies, based at least on the access key, a memory location within the target region where a data object within the database is persistently stored;

determining, by the RNIC using the registration information and based at least in part on the access key, an address within the target region in PRAM;

performing, by the RNIC for the database server, a direct memory access at the address within the target region of PRAM without communicating with a storage controller for the storage server while performing said direct memory access.

9. The non-transitory computer-readable media of claim 8, wherein the RDMA message includes a data payload and identifies, based on the access key, a memory location within the target region in PRAM for persistently storing the data payload.

10. The non-transitory computer-readable media of claim 9, said sequences of instructions including instructions that, in response to being executed by one or more processors, cause:

receiving, from the database server at the RNIC, an RDMA read message that identifies a final portion of the data payload;

returning, to the database server by the RNIC, an RDMA read response message that includes the final portion of the data payload;

wherein the RDMA read response message confirms that the data payload has been persistently stored.

11. The non-transitory computer-readable media of claim 8, said sequences of instructions including instructions that, in response to being executed by one or more processors, cause:

returning, by the RNIC, the data object to the database server in an RDMA read response message;

wherein the RDMA read response message references a buffer in the database server where the data object should be placed.

12. The non-transitory computer-readable media of claim 8,

wherein the RDMA message is a first RDMA message that identifies, based on the access key, a first memory location within the target region of PRAM;

wherein the direct memory access is to the first memory location within the target region of PRAM;

wherein said sequences of instructions include instructions that, in response to being executed by one or more processors, cause;

receiving, by the RNIC, a second RDMA message from the database server that includes the access key and identifies a second memory location within the target region of PRAM;

performing, by the RNIC, a second direct memory access to the second memory location within the target region of PRAM.

13. The non-transitory computer-readable media of claim 8,

wherein direct memory access is a first direct memory access,

wherein the registration information associates a second access key with the target region in PRAM,

wherein said sequences of instructions include instructions that, in response to being executed by one or more processors, cause:

sending the second access key to a second database server of the one or more database servers;

receiving, at the RNIC, a second RDMA message from the second database server that includes the second access key;

in response to receiving, at the RNIC, the second RDMA message from the second database server that includes the second access key, performing, by the RNIC, a second direct memory access within the target region of PRAM.

14. The non-transitory computer-readable media of claim 13, wherein the second direct memory access is to the memory location in the target region of PRAM accessed by the first direct memory access.

20. The one or more non-transitory computer-readable media storing instructions of claim 8, wherein determining, by the RNIC using the registration information and based at least in part on the access key, an address within the target region in PRAM includes determining, by the RNIC using the registration information and based at least in part on the access key and the memory location, an address within the target region in PRAM.

15. A storage server for providing one or more database servers with direct access to at least part of a database stored persistently by the storage server, the storage server comprising:

one or more central processing units (CPUs);

persistent random access memory (PRAM) of a storage server;

wherein the PRAM stores at least part of a database;

a remote direct access memory enabled network interface controller (RNIC) configured to;

store registration information that associates an access key with a target region in PRAM;

send the access key to a database server of the one or more database servers;

receive a remote direct memory access (RDMA) message from the database server that includes the access key;

wherein the RDMA message identifies, based at least on the access key, a memory location within the target region where a data object within the database is persistently stored;

determine, using the registration information and based at least in part on the access key, an address the target region in PRAM;

perform, for the database server, a direct memory access at the address within the target region in PRAM without communicating with a storage controller for the storage server while performing said direct memory access.

a bus interconnecting the one or more CPUs, the PRAM, and the RNIC.

16. The storage server of claim 15, wherein RNIC receives the RDMA message and performs the direct memory access without interacting with the one or more CPUs.

17. The storage server of claim 15, wherein the RDMA message includes a data payload and identifies, based on the access key, a memory location within the target region in PRAM for persistently storing the data payload.

18. The storage server of claim 15, wherein the registration information associates a second access key with the target region in PRAM, the RNIC is further configured to:

send the second access key to a second database server of the one or more database servers;

receive a second RDMA message from the second database server that includes the second access key;

in response to receiving the second RDMA message from the second database server that includes the second access key, performing a second direct memory access within the target region of PRAM.

21. The storage server of claim 15, wherein the RNIC is configured to determine, using the registration information and based at least in part on the access key, an address the target region in PRAM by being configured to determine, by the RNIC using the registration information and based at least in part on the access key and the memory location, an address within the target region in PRAM.