System and method for a shared write address protocol over a remote direct memory access connection

• US 9,544,243 B2
• Filed: 03/31/2014
• Issued: 01/10/2017
• Est. Priority Date: 04/03/2009
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Lentini, James
• Primary Examiner(s): Baturay, Alicia
• Application Number:
  US14/230,861
• Publication Number:
  US 20140214998A1
• Time in Patent Office: 1,016 Days
• Field of Search: 709/217
• US Class Current: 1/1
• CPC Class Codes: H04L 47/70 Admission control; Resource allocation
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

The present invention provides a system and method for a shared write address protocol (SWAP) that is implemented over a remote direct memory address (RDMA) connection. Each party to a connection establishes a flow control block that is accessible to its partner via a RDMA READ operation. The novel protocol operates so that each module needs to have one outstanding RDMA READ operation at a time, i.e., to obtain the current flow control information from its partner. In operation, if data to be transmitted is less than or equal to a buffer size, an INLINE message data structure of the SWAP protocol is utilized to send the data to be target. However, if the data is greater than the buffer size, a second determination is made as to whether sufficient space exists in the message pool for the data. If insufficient space exists, the sender will wait until sufficient space exists before utilizing a novel WRITE operation of the SWAP protocol to transmit the data.

Claims

1. A method comprising:

issuing, by a computing device, a read operation to a flow control block of another computing device, via a remote direct memory access (RDMA) connection with the another computing device, to retrieve a transaction identifier of a last processed response of the another computing device;

determining, by the computing device, when sufficient space exists in a message pool on the another computing device to store data to be transmitted to the another computing device based on the retrieved transaction identifier and a size of the data; and

issuing, by the computing device, a write operation via the RDMA connection to transmit the data to the another computing device, when the determining indicates that sufficient space exists in the message pool on the another computing device to store the data.

2. The method of claim 1 further comprising

determining, by the computing device, when the size of data is less than or equal to an inline message size;

and

transmitting, by the computing device, the data over the RDMA connection using an inline message data structure, when the determining indicates that the size of the data is less than or equal to the inline message size.

3. The method of claim 1 further comprising:

determining, by the computing device, when sufficient transactions have been processed by the another computing device to free an amount of space in the message pool greater than or equal to the size of the data, when the determining indicates that sufficient space does not exist in the message pool on the another computing device to store the data; and

transmitting, by the computing device, the data to the another computing device, when the determining indicates that sufficient transactions have been processed by the second node to free an amount of space in the message pool greater than or equal to the size of the data.

4. The method of claim 1 further comprising receiving, by the computing device and from the another computing device, configuration information for the RDMA connection comprising at least a location of the flow control block.

5. A non-transitory machine readable medium having stored thereon instructions for performing a method comprising machine executable code which when executed by at least one machine, causes the machine to:

issue a read operation to a flow control block of another computing device, via a remote direct memory access (RDMA) connection with the another computing device, to retrieve a transaction identifier of a last processed response of the another computing device;

determine when sufficient space exists in a message pool on the another computing device to store data to be transmitted to the another computing device based on the retrieved transaction identifier and a size of the data; and

issue a write operation via the RDMA connection to transmit the data to the another computing device, when the determining indicates that sufficient space exists in the message pool on the another computing device to store the data.

6. The non-transitory computer readable medium of claim 5 wherein the processor is further configured to execute the machine executable code to further cause the processor to:

determine, by the computing device, when the size of data is less than or equal to an inline message size; and

transmit the data over the RDMA connection using an inline message data structure, when the determining indicates that the size of the data is less than or equal to the inline message size.

7. The non-transitory computer readable medium of claim 5 wherein the processor is further configured to execute the machine executable code to further cause the processor to:

determine when sufficient transactions have been processed by the another computing device to free an amount of space in the message pool greater than or equal to the size of the data, when the determining indicates that sufficient space does not exist in the message pool on the another computing device to store the data;

and

transmit the data to the another computing device, when the determining indicates that sufficient transactions have been processed by the second node to free an amount of space in the message pool greater than or equal to the size of the data.

8. The non-transitory computer readable medium of claim 5 wherein the machine executable code when executed by the machine further causes the machine to receive, from the another computing device, configuration information for the RDMA connection comprising at least a location of the flow control block.

9. A computing device comprising:

a memory containing machine readable medium comprising machine executable code having stored thereon instructions for performing a method of optimizing communications over remote direct memory access (RDMA) connections; and

a processor coupled to the memory, the processor configured to execute the machine executable code to cause the processor to;

issue a read operation to a flow control block of another computing device, via an RDMA connection with the another computing device, to retrieve a transaction identifier of a last processed response of the another computing device;

determine when sufficient space exists in a message pool on the another computing device to store data to be transmitted to the another computing device based on the retrieved transaction identifier and a size of the data; and

issue a write operation via the RDMA connection to transmit the data to the another computing device, when the determining indicates that sufficient space exists in the message pool on the another computing device to store the data.

10. The computing device of claim 9, wherein the processor is further configured to execute the machine executable code to further cause the processor to:

determine when the size of data is less than or equal to an inline message size; and

transmit the data over the RDMA connection using an inline message data structure, when the determining indicates that the size of the data is less than or equal to the inline message size.

11. The computing device of claim 9, wherein the processor is further configured to execute the machine executable code to further cause the processor to:

determine when sufficient transactions have been processed by the another computing device to free an amount of space in the message pool greater than or equal to the size of the data, when the determining indicates that sufficient space does not exist in the message pool on the another computing device to store the data; and

transmit the data to the another computing device, when the determining indicates that sufficient transactions have been processed by the second node to free an amount of space in the message pool greater than or equal to the size of the data.

12. The computing device of claim 9, wherein the processor is further configured to execute the machine executable code to further cause the processor to receive, from the another computing device, configuration information for the RDMA connection comprising at least a location of the flow control block.