Apparatus and method for out-of-order placement and in-order completion reporting of remote direct memory access operations

• US 20070208820A1
• Filed: 02/17/2006
• Published: 09/06/2007
• Est. Priority Date: 02/17/2006
• Status: Abandoned Application

• Current Assignee: Intel Corporation
• Original Assignee: NetEffect, Inc. (Intel Corporation)
• Inventors: Makhervaks, Vadim, Keels, Kenneth
• Application Number:
  US11/357,445
• Publication Number:
  US 20070208820A1
• Time in Patent Office: Days
• US Class Current: 709/212
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 69/16 Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
• CPC Class Codes: H04L 69/166 IP fragmentation; TCP segmentation

Abstract

A mechanism for performing RDMA operations over a network fabric. Apparatus includes transaction logic to process work queue elements, and to accomplish the RDMA operations over a TCP/IP interface between first and second servers. The transaction logic has out-of-order segment range record stores and a protocol engine. The out-of-order segment range record stores maintains parameters associated with one or more out-of-order segments, the one or more out-of-order segments having been received and corresponding to one or more RDMA messages that are associated with the work queue elements. The protocol engine is coupled to the out-of-order segment range record stores and is configured to access the parameters to enable in-order completion tracking and reporting of the one or more RDMA messages.

Claims

1. An apparatus, for performing remote direct memory access (RDMA) operations between a first server and a second server over a network fabric, the apparatus comprising:

transaction logic, configured to process work queue elements, and configured to accomplish the RDMA operations over a TCP/IP interface between the first and second servers, wherein said work queue elements reside within first host memory corresponding to the first server, said transaction logic comprising;

out-of-order segment range record stores, configured to maintain parameters associated with one or more out-of-order segments, said one or more out-of-order segments having been received and corresponding to one or more RDMA messages that are associated with said work queue elements; and

a protocol engine, coupled to said out-of-order segment range record stores, configured to access said parameters to enable in-order completion tracking and reporting of said one or more RDMA messages.

2. The apparatus as recited in claim 1, wherein said transaction logic causes data corresponding to said one or more out-of-order segments to be transferred from second host memory into said first host memory, wherein said second host memory corresponds to the second server, and wherein said data is placed in said first host memory prior to completion of said one or more RDMA messages.

3. The apparatus as recited in claim 1, wherein said protocol engine employs said parameters to generate completion queue elements that correspond to said work queue elements

4. The apparatus as recited in claim 1, wherein said network fabric comprises a point-to-point fabric.

5. The apparatus as recited in claim 1, wherein said network fabric comprises one or more 1-Gigabit Ethernet links.

6. The apparatus as recited in claim 1, wherein said network fabric comprises one or more 10-Gigabit Ethernet links.

7. The apparatus as recited in claim 1, wherein said transaction logic is embodied within a network adapter corresponding to the first server.

8. The apparatus as recited in claim 1, wherein said out-of-order segment range record stores comprises one or more records, and wherein one of said one or more records corresponds to a TCP connection context.

9. The apparatus as recited in claim 8, wherein said one or more records comprise a local memory.

10. The apparatus as recited in claim 8, wherein each of said one or more records is dynamically bound to said corresponding TCP connection context upon receipt of an associated out-of-order segment.

11. The apparatus as recited in claim 8, wherein each of said one or more records can track information associated with a plurality of SACK blocks for said corresponding TCP connection.

12. The apparatus as recited in claim 11, wherein said plurality of SACK blocks comprises four SACK blocks.

13. The apparatus as recited in claim 11, wherein each of said one or more records are associated with said corresponding TCP connection, and wherein each one of said one or more records comprises fields, and wherein said protocol engine employs said fields to determine completion of said one or more RDMA messages.

14. The apparatus as recited in claim 13, wherein said fields comprise:

a first field, configured to indicate a number of received out-of-order segments with last flag asserted.

15. The apparatus as recited in claim 14, wherein said number of received out-of-order segments corresponds to an RDMA read request message.

16. The apparatus as recited in claim 14, wherein said number of received out-of-order segments corresponds to an RDMA write message.

17. The apparatus as recited in claim 14, wherein said number of received out-of-order segments corresponds to a send message.

18. The apparatus as recited in claim 14, wherein said number of received out-of-order segments corresponds to an RDMA read response message.

19. An apparatus, for performing remote direct memory access (RDMA) operations between a first server and a second server over a network fabric, the apparatus comprising:

a first network adapter, configured to access work queue elements, and configured to transmit framed protocol data units (FPDUs) corresponding to the RDMA operations over a TCP/IP interface between the first and second servers, wherein the RDMA operations are responsive to said work queue elements, and wherein said work queue elements are provided within first host memory corresponding to the first server, said first network adapter comprising;

out-of-order segment range record stores, configured to maintain parameters associated with one or more out-of-order segments in a corresponding record, said one or more out-of-order segments having been received and corresponding to one or more RDMA messages that are associated with said work queue elements; and

a protocol engine, coupled to said out-of-order segment range record stores, configured to access said record to enable in-order completion tracking and reporting of said one or more RDMA messages.

a second network adapter, configured to receive said FPDUs, and configured to transmit said one or more RDMA messages, whereby said RDMA operations are accomplished without error.

20. The apparatus as recited in claim 19, wherein said FPDUs direct that data corresponding to the RDMA operations be transferred from second host memory into said first host memory, wherein said second host memory corresponds to the second server.

21. The apparatus as recited in claim 19, wherein said protocol engine employs parameters stored within said record to generate completion queue elements that correspond to said work queue elements.

22. The apparatus as recited in claim 19, wherein said out-of-order segment range record stores comprises a plurality of records, each corresponding to a particular TCP connection, and each comprising one or more fields, said corresponding record being one of said plurality of records.

23. The apparatus as recited in claim 22, wherein said plurality of records comprises a local memory.

24. The apparatus as recited in claim 22, wherein each of said plurality of records is dynamically bound to a corresponding TCP connection upon receipt of an associated out-of-order segment.

25. The apparatus as recited in claim 22, wherein said each of said plurality of records comprises fields, and wherein said protocol engine employs said fields to determine completion of said one or more RDMA messages.

26. The apparatus as recited in claim 20, wherein said fields comprise:

a first field, configured to indicate a number of received out-of-order segments with last flag asserted.

27. A method for performing remote direct memory access (RDMA) operations between a first server and a second server over a network fabric, the method comprising:

processing work queue elements, wherein the work queue elements reside within a work queue that is within first host memory corresponding to the first server; and

accomplishing the RDMA operations over a TCP/IP interface between the first and second servers, wherein said accomplishing comprises;

maintaining out-of-order segment range record parameters associated with the work queue element in a local out-of-order segment range record; and

accessing the parameters to enable in-order completion reporting for an associated RDMA message having received and placed out-of-order segments.

28. The method as recited in claim 27, wherein said accomplishing further comprises:

transferring data corresponding to the RDMA operations from second host memory into the first host memory, wherein the second host memory corresponds to the second server.

29. The method as recited in claim 27, wherein said accomplishing further comprises:

employing the parameters to generate completion queue elements that correspond to the work queue elements.

30. The method as recited in claim 27, wherein said maintaining comprises:

dynamically binding the local out-of-order segment range record to a corresponding TCP connection upon receipt of a first out-of-order segment.

31. The method as recited in claim 27, wherein said maintaining comprises storing the parameters in fields, and wherein said accessing comprises employing said fields to determine if the associated RDMA message has been completely received in order.

32. The method as recited in claim 27, wherein the fields comprise:

a first field, configured to indicate a number of received out-of-order segments with last flag asserted.

33. The method as recited in claim 32, wherein said number of received out-of-order segments corresponds to an RDMA read request message.

34. The method as recited in claim 32, wherein said number of received out-of-order segments corresponds to an RDMA write message.

35. The method as recited in claim 32, wherein said number of received out-of-order segments corresponds to a send message.

36. The method as recited in claim 32, wherein said number of received out-of-order segments corresponds to an RDMA read response message.

37. The method as recited in claim 27, wherein the out-of-order segment range record parameters correspond to one or more out-of-order segment ranges.

38. The method as recited in claim 27, wherein the one or more out-of-order segment ranges comprise four out-of-order segment ranges.