Transporting operations of arbitrary size over remote direct memory access

• US 9,331,955 B2
• Filed: 06/29/2011
• Issued: 05/03/2016
• Est. Priority Date: 06/29/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Talpey, Thomas M., Kramer, Gregory Robert, Kruse, David Matthew
• Primary Examiner(s): Vu, Viet
• Assistant Examiner(s): Belcher, Herman
• Application Number:
  US13/172,757
• Publication Number:
  US 20130007180A1
• Time in Patent Office: 1,770 Days
• Field of Search: 709/212
• US Class Current: 1/1
• CPC Class Codes: G06F 16/183 Provision of network file services by network file servers, e.g. by using NFS, CIFS network file access protocols H04L67/1097
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0656 Data buffering arrangements
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 1/1835 Buffer management
• CPC Class Codes: H04L 47/39 Credit based
• CPC Class Codes: H04L 47/722 at the destination endpoint, e.g. reservation of terminal resources or buffer space
• CPC Class Codes: H04L 49/9057 Arrangements for supporting packet reassembly or resequencing
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

The embodiments described herein generally relate to a protocol for implementing data operations, e.g., a version of SMB, atop RDMA transports. In embodiments, systems and methods use the protocol definition, which specifies new messages for negotiating an RDMA connection and for transferring SMB2 data using the negotiated communication. A new protocol message may include new header information to determine message size, number of messages, and other information for sending the SMB2 data over RDMA. The header information is used to accommodate differences in message size requirements between RDMA and SMB2. The SMB Direct protocol allows SMB2 data to be fragmented into multiple individual RDMA messages that a receiver may then logically concatenate into a single SMB2 request or SMB2 response. The SMB Direct protocol also may allow SMB2 to transfer application data via efficient RDMA direct placement and to signal the application data'"'"'s availability when the transfer is complete.

Claims

1. A computer-implemented method for establishing a connection that exchanges data using server message block (SMB/SMB2) over remote direct memory access (RDMA), the method comprising:

Receiving, by a receiver, a first SMB Direct Data Transfer packet, wherein the first SMB Direct Data Transfer packet comprises a RemainingDataLength field and SMB2 data, wherein the RemainingDataLength field is equal to a number of bytes of a fragmented message that the receiver has yet to receive;

determining if the RemainingDataLength field is non-zero;

when the RemainingDataLength field is zero;

processing the SMB2 data in the first SMB Direct Data Transfer packet;

when the RemainingDataLength field is non-zero;

allocating a reassembly buffer to the connection;

copying the SMB2 data from the first SMB Direct Data Transfer packet into the reassembly buffer;

receiving at least one other SMB Direct Data Transfer packet; and

copying the SMB2 data from the at least one other SMB Direct Data Transfer packet into the reassembly buffer.

2. The method of claim 1, wherein the SMB Direct Data Transfer packet comprises one or more fields from the group consisting of:

CreditsRequested, CreditsGranted, DataOffset, and DataLength.

3. The method of claim 2, wherein the CreditsGranted field delineates how many credits were provided to a sender of the SMB Direct Data Transfer packet.

4. The method of claim 1, wherein a credit is a value for the number of RDMA messages that may be sent by the sender.

5. The method of claim 1, further comprising:

determining if the at least one other SMB Direct Data Transfer packet is a last packet;

if the at least one other SMB Direct Data Transfer packet is the last packet, processing the SMB2 data in the reassembly buffer;

if the at least one other SMB Direct Data Transfer packet is not the last packet, repeating receiving at least one other SMB Direct Data Transfer packet and copying the SMB2 data from the at least one other SMB Direct Data Transfer packet into the reassembly buffer.

6. The method of claim 5, wherein the reassembly buffer includes data from at least two SMB Direct Data Transfer packets.

7. The method of claim 5, wherein the determining if the first SMB Direct Data Transfer packet is a last packet comprises determining if the RemainingDataLength equals zero.

8. The method of claim 1, further comprising:

pre-posting a receive of a SMB Direct Negotiate Request;

setting a negotiation request expiration timer;

determining if the SMB Direct Negotiate Request was received before the negotiation request expiration timer expired;

if the SMB Direct Negotiate Request was not received before the negotiation request expiration timer expired, cancelling the receive of the SMB Direct Negotiate Request;

if the SMB Direct Negotiate Request was received before the negotiation request expiration timer expired, receiving the SMB Direct Negotiate Request, wherein the SMB Direct Negotiate Request requests to establish the connection to send SMB2 data over RDMA;

validating the SMB Direct Negotiate Request;

processing the SMB Direct Negotiate Request;

in response to processing the SMB Direct Negotiate Request, generating a SMB Direct Negotiate Response, wherein the SMB Direct Negotiate Response allows the connection and assigns at least one credit to a sender of the SMB Direct Negotiate Request to send the SMB Direct Data Transfer packet;

sending the SMB Direct Negotiate Response.

9. The method of claim 8, wherein the SMB Direct Negotiate Request comprises one or more fields from the group consisting of:

MinVersion, MaxVersion, CreditsRequested, PreferredSendSize, MaxReceiveSize, and MaxSMB2MessageSize.

10. The method of claim 9, wherein the SMB Direct Negotiate Response comprises one or more fields from the group consisting of:

MinVersion, MaxVersion, PreferredVersion, CreditsRequested, CreditsGranted, Status, PreferredSendSize, MaxReceiveSize, and MaxSMB2MessageSize.

11. A computer readable storage device storing computer-executable instructions that when executed by a processor perform a method for exchanging data using data operations over remote direct memory access (RDMA), the method comprising:

establishing a connection with a server;

negotiating a connection with the server, wherein the negotiation establishes a maximum number of bytes the server will receive in the connection;

determining a number of packets to send data associated with the connection to the server;

determining if fragmentation of the data is to be used;

when fragmentation of the data is not to be used;

sending a first protocol packet to the server with the data;

when fragmentation of the data is to be used;

initializing first bytes within a transmission buffer to send as data in the first protocol packet;

setting a DataLength field and a RemainingDataLength field in the first protocol packet, wherein the RemainingDataLength field is equal to a number of bytes of a fragmented message that the server has yet to receive;

sending the first protocol packet to the server;

for at least a second protocol packet, instructions to repeat, comprising;

initializing bytes within the transmission buffer to send as data in the at least a second protocol packet;

setting the RemainingDataLength field in the at least a second protocol packet; and

sending the at least a second protocol packet to the server.

12. The computer readable storage device of claim 11, wherein the determining if fragmentation is to be used comprises determining if the number of packets to send the data is greater than one.

13. The computer readable storage device of claim 12, wherein the determining a number of packets to send data associated with the connection to the server comprises determining if the maximum number of bytes the server will receive in the connection is less than a number of bytes of data to be sent.

14. The computer readable storage device of claim 11, wherein negotiating the connection with the server comprises:

pre-posting a receive;

constructing a protocol negotiate request;

sending the protocol negotiate request to the server;

setting a negotiate expiration timer;

determining if a response to the protocol negotiate request is received before the negotiate expiration timer expires;

if a response to the protocol negotiate request is not received before the negotiate expiration timer expires, resending the protocol negotiate response;

if a response to the protocol negotiate request is received before the negotiate expiration timer expires, receiving a protocol negotiate response;

validating the protocol negotiate response; and

processing the protocol negotiate response, wherein the protocol negotiate response provides at least one credit to the sender to send the protocol packet.

15. The computer readable storage device of claim 14, further comprising:

if enough credits exist for sending the at least a second protocol packet, sending the at least a second protocol packet;

if enough credits do not exist to send the at least a second protocol packet, requesting at least one more credit from the server;

determining if enough credits are received;

if enough credits are not received, terminating the connection; and

if enough credits are received, sending the at least a second protocol packet.

16. A system configured to exchange data using server message block (SMB/SMB2) over remote direct memory access (RDMA), the system comprising:

a RDMA network interface card (RNIC) operable to transfer data by a RDMA message;

memory operable to store computer program instructions executable by a processor; and

the processor, in communication with the RNIC and the memory, operable to execute a kernel, the kernel comprising;

a first protocol client operable to;

communicate fragmented SMB data in two or more RDMA messages, wherein each RDMA message comprises a RemainingDataLength field, and wherein the RemainingDataLength field is equal to a number of bytes of a fragmented message that a receiver has yet to receive; and

create a receive buffer to exchange credits, wherein the credits determine a number of RDMA messages that a sender may send to a receiver.

17. The system of claim 16, further comprising an RDMA Interface in communication with the first protocol client, the RDMA Interface operable to open an RDMA connection through the RNIC.

18. The system of claim 17, further comprising:

a second protocol client operable to;

create an SMB2 read/write request to instruct a server to perform an RDMA direct data transfer;

send the SMB2 read/write request to the first protocol client; and

wherein the first protocol client is further operable to;

encapsulate the SMB2 read/write request in a first protocol Data Transfer packet; and

send the first protocol Data Transfer packet to the server.

19. The system of claim 18, wherein the SMB2 read/write request includes fields that identify RDMA channel descriptor information, and wherein the first protocol is the SMB Direct protocol.

20. The system of claim 19, wherein the RDMA channel descriptor information includes steering information, and wherein the server determines the steering information and performs an RDMA write or an RDMA read of unencoded application data to or from memory addresses identified by the steering information.