Pipelined processing of RDMA-type network transactions

• US 8,078,743 B2
• Filed: 02/17/2006
• Issued: 12/13/2011
• Est. Priority Date: 02/17/2006
• Status: Expired due to Fees

• Current Assignee: Intel Corporation
• Original Assignee: Intel-Ne, Inc
• Inventors: Sharp, Robert O., Keels, Kenneth G., Hausauer, Brian S., Rose, Eric
• Primary Examiner(s): BURGESS, BARBARA N
• Application Number:
  US11/356,493
• Publication Number:
  US 20070226750A1
• Time in Patent Office: 2,125 Days
• Field of Search: 709/250, 709/232, 709/213, 709/230, 709/216, 709/217, 709/212, 709/214, 709/219, 370/395.52, 370/392, 370/412, 370/463, 370/428, 370/389, 719/312, 711/206, 711/203
• US Class Current: 709/230
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A computer system such as a server pipelines RNIC interface (RI) management/control operations such as memory registration operations to hide from network applications the latency in performing RDMA work requests caused in part by delays in processing the memory registration operations and the time required to execute the registration operations themselves. A separate QP-like structure, called a control QP (CQP), interfaces with a control processor (CP) to form a control path pipeline, separate from the transaction pipeline, which is designated to handle all control path traffic associated with the processing of RI control operations. This includes memory registration operations (MR OPs), as well as the creation and destruction of traditional QPs for processing RDMA transactions. Once the MR OP has been queued in the control path pipeline of the adapter, a pending bit is set which is associated with the MR OP. Processing of an RDMA work request in the transaction pipeline that has engendered the enqueued MR OP is permitted to proceed as if the processing of the MR OP has already been completed. If the work request gets ahead of the MR OP, the associated pending bit being set will notify the adapter'"'"'s work request transaction pipeline to stall (and possibly reschedule) completion of the work request until the processing of the MR OP for that memory region is complete. When the memory registration process for the memory region is complete, the associated pending bit is reset and the adapter transaction pipeline is permitted to continue processing the work request using the newly registered memory region.

Claims

1. An apparatus comprising:

a remote direct memory access (RDMA) adapter to be comprised in a server and to be coupled in the server to a plurality of host processors and to a host memory via a host interface, one of the host processors to initiate an RDMA transaction to be implemented by the adapter, the adapter to be coupled via physical interfaces to a network, the adapter including an offload engine, a local adapter memory interface, and protocol modules associated with respective protocols, one of the protocol modules to implement iWARP RDMA protocol layers and RDMA direct data placement, another of the protocol modules being an iSCSI protocol module to provide hardware acceleration to iSCSI protocol processing, the adapter to selectively permit data received from the host processors and the network to be provided to the protocol modules and the offload engine, the adapter to be programmed in support of an RDMA memory transaction requested by an RDMA transaction request from an application, a first pipeline portion of the adapter to enqueue a pending memory registration operation to register a memory region specified by the RDMA memory transaction, a second pipeline portion of the adapter to enqueue a work request specifying access to the memory region, processing of the work request to proceed as if the pending memory registration operation has already been completed unless the adapter is notified to stall completion of the work request until after memory registration operation processing is complete, the host memory including a plurality of memory regions, the adapter including a control operation portion having the first pipeline portion, a work request portion having the second pipeline portion, and a memory region table, a program being to queue a control operation of the RDMA transaction request in the first pipeline portion and to set a pending notification in the memory region table associated with the control operation and a relevant memory region of the plurality of memory regions, the program also being to queue the work request of the RDMA transaction request in the second pipeline portion, the control operation portion being to reset the pending notification as a result of completion of the control operation, and the work request portion being to delay processing of the work request if the pending notification is set and to complete processing of the work request if the pending notification is reset.

2. The apparatus of claim 1, wherein:

the local adapter memory interface is coupled to local adapter memory; and

the host interface is comprised in the adapter and is coupled to a chipset, the chipset coupling the host processors to the adapter and to the host memory.

3. The apparatus of claim 2, further comprising:

a transaction switch to selectively permit the data to be provided to the protocol modules, the offload engine, and the local adapter memory interface.

4. The apparatus of claim 1, wherein:

the host memory includes a control queue pair; and

queuing of the control operation includes posting the control operation in the control queue pair.

5. The apparatus of claim 4, wherein:

the adapter also includes a register to receive entries for the memory region table; and

setting of the pending notification also includes performing a register write operation to the register.

6. The apparatus of claim 5, wherein:

the adapter also includes a doorbell to receive notification that the control operation has been posted; and

the queuing of the control operation comprising ringing the doorbell.

7. The apparatus of claim 6, wherein:

delaying the processing of the work request comprises rescheduling the work request in the second pipeline portion.

8. The apparatus of claim 6, wherein:

the second pipeline portion is a transmit pipeline.

9. A method comprising:

coupling in a server a remote direct memory access (RDMA) adapter comprised in the server to a plurality of host processors and to a host memory via a host interface, one of the host processors to initiate an RDMA transaction to be implemented by the adapter, the adapter to be coupled via physical interfaces to a network, the adapter including an offload engine, a local adapter memory interface, and protocol modules associated with respective protocols, one of the protocol modules to implement iWARP RDMA protocol layers and RDMA direct data placement, another of the protocol modules being an iSCSI protocol module to provide hardware acceleration to iSCSI protocol processing, the adapter to selectively permit data received from the host processors and the network to be provided to the protocol modules and the offload engine, the adapter to be programmed in support of an RDMA memory transaction requested by an RDMA transaction request from an application, a first pipeline portion of the adapter to enqueue a pending memory registration operation to register a memory region specified by the RDMA memory transaction, a second pipeline portion of the adapter to enqueue a work request specifying access to the memory region, processing of the work request to proceed as if the pending memory registration operation has already been completed unless the adapter is notified to stall completion of the work request until after memory registration operation processing is complete, the host memory including a plurality of memory regions, the adapter including a control operation portion having the first pipeline portion, a work request portion having the second pipeline portion, and a memory region table, a program being to queue a control operation of the RDMA transaction request in the first pipeline portion and to set a pending notification in the memory region table associated with the control operation and a relevant memory region of the plurality of memory regions, the program also being to queue the work request of the RDMA transaction request in the second pipeline portion, the control operation portion being to reset the pending notification as a result of completion of the control operation, and the work request portion being to delay processing of the work request if the pending notification is set and to complete processing of the work request if the pending notification is reset.

10. The method of claim 9, wherein:

the local adapter memory interface is coupled to local adapter memory;

the host interface is comprised in the adapter and is coupled to a chipset, the chipset coupling the host processors to the adapter and to the host memory.

11. The method of claim 10, wherein:

a transaction switch is to selectively permit the data to be provided to the protocol modules, the offload engine, and the local adapter memory interface.

12. The method of claim 9, wherein:

the host memory includes a control queue pair; and

queuing of the control operation includes posting the control operation in the control queue pair.

13. The method of claim 12, wherein:

the adapter also includes a register to receive entries for the memory region table; and

setting of the pending notification also includes performing a register write operation to the register.

14. The method of claim 13, wherein:

the adapter also includes a doorbell to receive notification that the control operation has been posted; and

the queuing of the control operation comprising ringing the doorbell.

15. The method of claim 14, wherein:

delaying the processing of the work request comprises rescheduling the work request in the second pipeline portion.

16. The method of claim 14, wherein:

the second pipeline portion is a transmit pipeline.