Apparatus and method for packet transmission over a high speed network supporting remote direct memory access operations

• US 20060230119A1
• Filed: 12/22/2005
• Published: 10/12/2006
• Est. Priority Date: 04/08/2005
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: NetEffect, Inc. (Intel Corporation)
• Inventors: Gross, Tristan T., Hausauer, Brian S., Keels, Kenneth G., Wandler, Shaun V.
• Granted Patent: US 8,458,280 B2
• Time in Patent Office: Days
• US Class Current: 709/216
• CPC Class Codes: H04L 47/6265 past bandwidth allocation
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A mechanism for performing remote direct memory access (RDMA) operations between a first server and a second server over an Ethernet fabric. The RDMA operations are initiated by execution of a verb according to a remote direct memory access protocol. The verb is executed by a CPU on the first server. The apparatus includes transaction logic that is configured to process a work queue element corresponding to the verb, and that is configured to accomplish the RDMA operations over a TCP/IP interface between the first and second servers, where the work queue element resides within first host memory corresponding to the first server. The transaction logic includes transmit history information stores and a protocol engine. The transmit history information stores maintains parameters associated with said work queue element. The protocol engine is coupled to the transmit history information stores and is configured to access the parameters to enable retransmission of one or more TCP segments corresponding to the RDMA operations.

Claims

1. An apparatus, for performing remote direct memory access (RDMA) operations between a first server and a second server over an Ethernet fabric, the RDMA operations being initiated by execution of a verb according to a remote direct memory access protocol, the verb being executed by a CPU on the first server, the apparatus comprising:

transaction logic, configured to process a work queue element corresponding to the verb, and configured to accomplish the RDMA operations over a TCP/IP interface between the first and second servers, wherein said work queue element resides within first host memory corresponding to the first server, said transaction logic comprising;

transmit history information stores, configured to maintain parameters associated with said work queue element; and

a protocol engine, coupled to said transmit history information stores, configured to access said parameters to enable retransmission of one or more TCP segments corresponding to the RDMA operations.

2. The apparatus as recited in claim 1, wherein said transaction logic causes data corresponding to the RDMA operations to be transferred from second host memory into said first host memory, wherein said second host memory corresponds to the second server.

3. The apparatus as recited in claim 1, wherein said transaction logic causes data corresponding to the RDMA operations to be transferred from said first host memory into second host memory, wherein said second host memory corresponds to the second server.

4. The apparatus as recited in claim 1, wherein said protocol engine employs said parameters to generate a completion queue element that corresponds to said work queue element.

5. The apparatus as recited in claim 1, wherein said Ethernet fabric comprises a point-to-point fabric.

6. The apparatus as recited in claim 1, wherein said Ethernet fabric comprises one or more 1-Gigabit Ethernet links.

7. The apparatus as recited in claim 1, wherein said Ethernet fabric comprises one or more 10-Gigabit Ethernet links.

8. The apparatus as recited in claim 1, wherein said transaction logic is embodied within a network adapter corresponding to the first server.

9. The apparatus as recited in claim 8, wherein said network adapter comprises a printed circuit board.

10. The apparatus as recited in claim 8, wherein said network adapter comprises an integrated circuit.

11. The apparatus as recited in claim 8, wherein the first server comprises a blade server, and wherein said network adapter is embodied within said blade server.

12. The apparatus as recited in claim 8, wherein said network adapter is integrated into a memory controller corresponding to the first server.

13. The apparatus as recited in claim 1, further comprising:

host interface logic, coupled to said transaction logic, configured to transfer data corresponding to the RDMA transactions to or from said first host memory.

14. The apparatus as recited in claim 13, wherein said host interface logic comprises PCI-X logic, and wherein PCI-X transactions are employed to transfer said data.

15. The apparatus as recited in claim 13, wherein said host interface logic comprises PCI Express logic, and wherein PCI Express transactions are employed to transfer said data.

16. The apparatus as recited in claim 1, wherein said transmit history information stores comprises one or more FIFO buffers, and wherein one of said one or more FIFO buffers corresponds to a work queue, and wherein said work queue comprises said work queue element.

17. The apparatus as recited in claim 16, wherein said one or more FIFO buffers comprise a local memory.

18. The apparatus as recited in claim 16, wherein each of said one or more FIFO buffers is dynamically bound to a corresponding work queue.

19. The apparatus as recited in claim 16, wherein each of said one or more FIFO buffers comprises one or more entries.

20. The apparatus as recited in claim 19, wherein one of said one or more entries corresponds to said work queue element, and wherein said one of said one or more entries comprises fields, and wherein said protocol engine employs said fields to rebuild one or more TCP segments for retransmission over the Ethernet fabric.

21. The apparatus as recited in claim 20, wherein said fields comprise:

a first field, configured to indicate a send message sequence number for the RDMA transactions;

a second field, configured to indicate a read message sequence number for the RDMA transactions;

a third field, configured to indicate a TCP start sequence number for the RDMA transactions; and

a fourth field, configured to indicate a TCP final sequence number for the RDMA transactions.

22. The apparatus as recited in claim 19, wherein one of said one or more entries corresponds to said work queue element, and wherein said one of said one or more entries comprises fields, and wherein said protocol engine employs said fields to determine that a completion queue element is to be written to a completion queue corresponding to said work queue, and wherein said completion queue element corresponds to said work queue element.

23. The apparatus as recited in claim 22, wherein said fields comprise:

a first field, configured to indicate a TCP final sequence number for the RDMA transactions.

24. An apparatus, for performing remote direct memory access (RDMA) operations between a first server and a second server over an Ethernet fabric, the RDMA operations being initiated by execution of a verb according to a remote direct memory access protocol, the verb being executed by a CPU on the first server, the apparatus comprising:

a first network adapter, configured to access a work queue element responsive to execution of the verb, and configured to transmit framed protocol data units (FPDUs) corresponding to the RDMA operations over a TCP/IP interface between the first and second servers, wherein the RDMA operations are responsive to said work queue element, and wherein said work queue element is provided within first host memory corresponding to the first server, said first network adapter comprising;

transmit history information stores, configured to maintain parameters associated with said work queue element in a corresponding FIFO buffer entry; and

a protocol engine, coupled to said transmit history information stores, configured to access said FIFO buffer entry to enable retransmission of one or more TCP segments corresponding to a subset of said FPDUs; and

a second network adapter, configured to receive said FPDUs, wherein reception of said FPDUs includes receiving said one or more TCP segments, whereby said RDMA operations are accomplished without error.

25. The apparatus as recited in claim 24, wherein said FPDUs direct that data corresponding to the RDMA operations be transferred from second host memory into said first host memory, wherein said second host memory corresponds to the second server.

26. The apparatus as recited in claim 24, wherein said FPDUs comprise data corresponding to the RDMA operations, and wherein said data is transferred from said first host memory into second host memory, wherein said second host memory corresponds to the second server.

27. The apparatus as recited in claim 24, wherein said protocol engine employs said parameters to generate a completion queue element that corresponds to said work queue element.

28. The apparatus as recited in claim 24, wherein the first and second servers comprise first and second blade servers, respectively, and wherein said first and second network adapters are respectively embodied within said first and second blade servers.

29. The apparatus as recited in claim 24, wherein said first network adapter further comprises:

host interface logic, operatively coupled to said first host memory, configured to transfer data within one or more of said FPDUs to or from said first host memory.

30. The apparatus as recited in claim 29, wherein said host interface logic comprises PCI-X logic, and wherein PCI-X transactions are employed to transfer said data.

31. The apparatus as recited in claim 29, wherein said host interface logic comprises PCI Express logic, and wherein PCI Express transactions are employed to transfer said data.

32. The apparatus as recited in claim 24, wherein said transmit history information stores comprises a plurality of FIFO buffers, each corresponding to a particular work queue, and each comprising one or more entries.

33. The apparatus as recited in claim 32, wherein said plurality of FIFO buffers comprises a local memory.

34. The apparatus as recited in claim 32, wherein each of said plurality of FIFO buffers is dynamically bound to a corresponding work queue.

35. The apparatus as recited in claim 24, wherein said FIFO buffer entry comprises fields, and wherein said protocol engine employs said fields to rebuild said one or more TCP segments for retransmission over the Ethernet fabric.

36. The apparatus as recited in claim 35, wherein said fields comprise:

a first field, configured to indicate a send message sequence number for said FPDUs;

a second field, configured to indicate a read message sequence number for said FPDUs;

a third field, configured to indicate a TCP start sequence number for said FPDUs; and

a fourth field, configured to indicate a TCP final sequence number for said FPDUs.

37. The apparatus as recited in claim 24, wherein said FIFO buffer entry comprises fields, and wherein said protocol engine employs said fields to determine that a completion queue element is to be written to a completion queue corresponding to said work queue, and wherein said completion queue element corresponds to said work queue element.

38. The apparatus as recited in claim 37, wherein said fields comprise:

a first field, configured to indicate a TCP final sequence number for the RDMA transactions.

39. An apparatus, for performing remote direct memory access (RDMA) operations between a first server and a second server over an Ethernet fabric, the RDMA operations being initiated by execution of a verb according to a remote direct memory access protocol, the verb being executed by a CPU on the first server, the apparatus comprising:

transaction logic, configured to process a work queue element corresponding to the verb, and configured to accomplish the RDMA operations over a TCP/IP interface between the first and second servers, wherein said work queue element resides within first host memory corresponding to the first server, said transaction logic comprising;

transmit history information stores, configured to maintain a maximum upper level protocol data unit (MULPDU) size that was in effect at the time of transmission of a PDU associated with said work queue element; and

a protocol engine, coupled to said transmit history information stores, configured to access said MULPDU to enable retransmission of one or more TCP segments corresponding to the RDMA operations.

40. The apparatus as recited in claim 39, wherein said transaction logic is embodied within a network adapter corresponding to the first server.

41. The apparatus as recited in claim 39, further comprising:

host interface logic, coupled to said transaction logic, configured to transfer data corresponding to the RDMA transactions to or from said first host memory.

42. The apparatus as recited in claim 39, wherein said transmit history information stores comprises one or more FIFO buffers, and wherein one of said one or more FIFO buffers corresponds to a work queue, and wherein said work queue comprises said work queue element.

43. The apparatus as recited in claim 42, wherein each of said one or more FIFO buffers comprises one or more entries.

44. The apparatus as recited in claim 43, wherein one of said one or more entries corresponds to said work queue element, and wherein said one of said one or more entries comprises fields, and wherein said protocol engine employs said fields to rebuild one or more TCP segments for retransmission over the Ethernet fabric.

45. A method for performing remote direct memory access (RDMA) operations between a first server and a second server over an Ethernet fabric, the RDMA operations being initiated by execution of a verb according to a remote direct memory access protocol, the verb being executed by a CPU on the first server, the method comprising:

processing a work queue element corresponding to the verb, wherein the work queue element resides within a work queue that is within first host memory corresponding to the first server; and

accomplishing the RDMA operations over a TCP/IP interface between the first and second servers, wherein said accomplishing comprises;

maintaining parameters associated with the work queue element in a local FIFO buffer entry; and

accessing the parameters to enable retransmission of one or more TCP segments corresponding to the RDMA transactions.

46. The method as recited in claim 45, wherein said accomplishing further comprises:

transferring data corresponding to the RDMA operations from second host memory into the first host memory, wherein the second host memory corresponds to the second server.

47. The method as recited in claim 45, wherein said accomplishing further comprises:

transferring data corresponding to the RDMA operations from the first host memory into second host memory, wherein the second host memory corresponds to the second server.

48. The method as recited in claim 45, wherein said accomplishing further comprises:

employing the parameters to generate a completion queue element that corresponds to the work queue element.

49. The method as recited in claim 45, wherein said maintaining comprises:

dynamically binding the local FIFO buffer entry to the work queue element.

50. The method as recited in claim 45, wherein said processing comprises:

via host interface logic, transferring data corresponding to the RDMA operations to or from the first host memory.

51. The method as recited in claim 50, wherein the host interface logic comprises PCI-X logic, and wherein PCI-X transactions are employed to transfer the data.

52. The method as recited in claim 50, wherein the host interface logic comprises PCI Express logic, and wherein PCI Express transactions are employed to transfer the data.

53. The method as recited in claim 45, wherein said maintaining comprises storing the parameters in fields, and wherein said accessing comprises employing said fields to rebuild one or more TCP segments for retransmission over the Ethernet fabric.

54. The method as recited in claim 53, wherein said fields comprise:

a first field, configured to indicate a send message sequence number for the RDMA operations;

a second field, configured to indicate a read message sequence number for the RDMA operations;

a third field, configured to indicate a TCP start sequence number for the RDMA operations; and

a fourth field, configured to indicate a TCP final sequence number for the RDMA operations.

55. The method as recited in claim 45, wherein said maintaining comprises storing the parameters in fields, and wherein said accessing comprises employing said fields to determine that a completion queue element is to be written to a completion queue corresponding to the work queue, and wherein the completion queue element corresponds to the work queue element.

56. The method as recited in claim 55, wherein said fields comprise:

a first field, configured to indicate a TCP final sequence number for the RDMA operations.