Network interface adapter with shared data send resources

• US 8,051,212 B2
• Filed: 12/04/2001
• Issued: 11/01/2011
• Est. Priority Date: 04/11/2001
• Status: Active Grant

• Current Assignee: Mellanox Technologies Limited (NVIDIA Corporation)
• Original Assignee: Mellanox Technologies Limited (NVIDIA Corporation)
• Inventors: Crupnicoff, Diego, Shachar, Ariel, Shainer, Gilad, Shnitman, Margarita, Izhaki, Ram, Levenvirth, Dafna, Koren, Benny, Shenhav, Yael, Gutman, Aviram, Kagan, Michael, Bloch, Gil
• Primary Examiner(s): DIVECHA, KAMAL B
• Application Number:
  US10/000,456
• Publication Number:
  US 20020152327A1
• Time in Patent Office: 3,619 Days
• Field of Search: 709212-217, 709/250, 709/249
• US Class Current: 709/250
• CPC Class Codes: G06F 13/1605 based on arbitration arbitration in handling access to a common bus or bus system G06F13/36
• CPC Class Codes: H04L 49/90 Buffering arrangements

Abstract

A network interface adapter includes an outgoing packet generator, adapted to generate an outgoing request packet for delivery to a remote responder responsive to a request submitted by a host processor and a network output port, coupled to transmit the outgoing request packet over a network to the remote responder. A network input port receives an incoming response packet from the remote responder, in response to the outgoing request packet sent thereto, as well as an incoming request packet sent by a remote requester. An incoming packet processor receives and processes both the incoming response packet and the incoming request packet, and causes the outgoing packet generator, responsive to the incoming request packet, to generate, in addition to the outgoing request packet, an outgoing response packet for transmission to the remote requester.

Claims

1. A network interface adapter, implemented on at least one integrated circuit chip and comprising:

a host interface, for coupling to a host processor;

an outgoing packet generator, operative to generate an outgoing request packet for delivery to a remote responder responsive to a request submitted by the host processor via the host interface;

a network output port, coupled to the outgoing packet generator to receive the outgoing request packet from the outgoing packet generator and to transmit the outgoing request packet over a network to the remote responder;

a network input port, for coupling to the network to receive an incoming response packet from the remote responder, in response to the outgoing request packet sent thereto, and further to receive an incoming request packet sent by a remote requester; and

an incoming packet processor, coupled to the network input port to receive and process both the incoming response packet and the incoming request packet, and further coupled to cause the outgoing packet generator, responsive to the incoming request packet, to generate, in addition to the outgoing request packet, an outgoing response packet for transmission via the network output port to the remote requester;

wherein the outgoing request packet comprises an outgoing write request packet containing write data taken from a system memory accessible via the host interface;

wherein the outgoing response packet comprises an outgoing read response packet containing read data taken from the system memory in response to the incoming request packet;

wherein the incoming request packet comprises an incoming RDMA read request packet specifying data to be read from a system memory that is external to the network interface adapter and that is accessible via the host interface;

wherein the incoming packet processor is operative to write a quasi-WQE to a first memory location, in the system memory, indicating the data to be read from the system memory responsive to the incoming RDMA read request packet;

wherein the outgoing packet generator is operative to read the quasi-WQE from the first memory location and, responsive thereto, to read the indicated data and to generate the outgoing response packet containing the indicated data;

wherein the outgoing packet generator comprises a hardware gather engine, which is coupled to gather both the write data and the read data from the system memory for inclusion in the respective outgoing request and response packets via a commonly shared data flow path from the system memory to the network output port; and

wherein to submit the request, the host processor writes a request descriptor indicative of the write data to a second memory location, and wherein the hardware gather engine is operative to read information from the quasi-WQE and from the request descriptor and to gather the read data and the write data responsive thereto.

2. An adapter according to claim 1, wherein the outgoing packet generator comprises a plurality of schedule queues, and is operative to generate the outgoing request packet and the outgoing response packet responsive to respective entries placed in the schedule queues of the plurality of schedule queues.

3. An adapter according to claim 2, wherein the network input and output ports are operative to receive and send the incoming and outgoing packets, respectively, over a plurality of transport service instances, and

wherein the outgoing request packet and the outgoing response packet are associated with respective transport service instances among the plurality of transport service instances, and

wherein the outgoing packet generator is operative to assign the transport service instances of the plurality of transport service instances to the schedule queues of the plurality of schedule queues based on service parameters of the transport service instances of the plurality of transport service instances, and to place the entries in the schedule queues of the plurality of schedule queues corresponding to the transport service instances, of the plurality of transport service instances, with which the incoming and outgoing packets are associated.

4. An adapter according to claim 3, wherein the outgoing packet generator comprises:

one or more execution engines, which are operative to generate the outgoing request packet and the outgoing response packet responsive to a list of work items respectively associated with each of the transport service instances of the plurality of transport service instances; and

a scheduler, which is coupled to select the entries from the plurality of schedule queues and to assign the transport service instances of the plurality of transport service instances to the one or more execution engines for execution of the work items responsive to the service parameters.

5. An adapter according to claim 3, wherein the transport service instances of the plurality of transport service instances comprise queue pairs.

6. An adapter according to claim 2, wherein the outgoing packet generator comprises one or more doorbell registers, to which the host processor and the incoming packet processor write in order to place the entries in the schedule queues of the plurality of schedule queues.

7. An adapter according to claim 2, wherein the incoming request packet comprises an incoming write request packet carried over the network on a reliable transport service, and wherein responsive to the incoming write request packet, the incoming packet processor is operative to add an entry to the entries placed in the schedule queues of the plurality of schedule queues, such that responsive to the entry, the outgoing packet generator generates an acknowledgment packet.

8. An adapter according to claim 1, wherein the incoming packet processor is configured so that when it receives an incoming write request packet containing write data to be written to a system memory accessible via the host interface after receiving the incoming RDMA read request packet, it conveys the write data to the host interface without waiting for execution of the quasi-WQE.

9. An adapter according to claim 1, wherein the incoming packet processor is configured so that when it receives an incoming write request packet containing write data to be written to a system memory accessible via the host interface before receiving the incoming RDMA read request packet, it prevents execution of the quasi-WQE until the write data have been written to the system memory.

10. An adapter according to claim 1, wherein the outgoing packet generator is operative, upon generating the outgoing request packet, to notify the incoming packet processor to await the incoming response packet and to write a completion message to the host interface when the awaited incoming response packet is received.

11. An adapter according to claim 1, wherein the incoming RDMA read request packet is one of a plurality of incoming RDMA read request packets, and wherein the incoming packet processor is operative to write a list of corresponding quasi-WQEs to the first memory location, each said quasi-WQE indicating the data to be read from the system memory responsive to the corresponding incoming RDMA read request packet, responsive to which the outgoing packet generator is operative to generate a sequence of corresponding outgoing response packets.

12. An adapter according to claim 11, wherein the network input and output ports are operative to receive and send the incoming and outgoing packets, respectively, over a plurality of transport service instances, and wherein the incoming packet processor is operative to prepare the list of the quasi-WQEs for each of the transport service instances of the plurality of transport service instances as a part of a response database held for the plurality of the transport service instances in common.

13. An adapter according to claim 12, wherein the transport service instances of the plurality of transport service instances comprise queue pairs.

14. An adapter according to claim 1, wherein the request comprises a write request, which is submitted by the host processor by generating a request descriptor indicating further data to be read from the system memory for inclusion in the outgoing request packet, and wherein the output packet generator is operative to read the request descriptor and, responsive thereto, to generate the outgoing request packet as a write request packet containing the indicated further data.

15. A method for coupling a host processor to a network, comprising:

generating an outgoing request packet for delivery to a remote responder using an outgoing packet generator of a network interface adapter that is implemented on at least one integrated circuit chip, responsive to a request submitted by the host processor;

transmitting the outgoing request packet from the outgoing packet generator over the network to the remote responder;

receiving an incoming response packet from the remote responder, in response to the outgoing request packet sent thereto, using an incoming packet processor of the network interface adapter;

receiving an incoming request packet sent by a remote requester using the incoming packet processor; and

coupling the incoming packet processor to the outgoing packet generator to cause the outgoing packet generator to generate, responsive to the incoming request packet, in addition to the outgoing request packet, an outgoing response packet for transmission via the network to the remote requester;

wherein generating the outgoing request packet comprises generating an outgoing write request packet containing write data taken from a system memory that is external to the network interface adapter and that is associated with the host processor;

wherein coupling the incoming packet processor to the outgoing packet generator comprises generating, using the outgoing packet generator, an outgoing read response packet containing read data taken from the system memory in response to the incoming request packet;

wherein receiving the incoming request packet comprises receiving an incoming RDMA read request packet specifying data to be read from a system memory associated with the host processor,

wherein coupling the incoming packet processor comprises writing, in response to the incoming RDMA read request packet, a quasi-WQE to a first memory location of the system memory indicating the data to be read therefrom, and causing the outgoing packet generator to read the quasi-WQE from the first memory location and, responsive thereto, to read the indicated data from the system memory and to generate the outgoing response packet containing the indicated data;

wherein generating the outgoing write request packet and generating the outgoing read response packet comprise generating the packets using a hardware gather engine in the outgoing packet generator, which is coupled to gather both the write data and the read data from the system memory for inclusion in the respective outgoing request and response packets via a commonly shared data flow path from the system memory to the network; and

wherein generating the outgoing write request packet comprises generating a request descriptor indicative of the write data to a second memory location, and wherein generating the packets using the hardware gather engine comprises reading information from the quasi-WQE and from the request descriptor using the hardware gather engine and gathering the read data and the write data responsive thereto.

16. A method according to claim 15, wherein the outgoing packet generator comprises a plurality of schedule queues, and wherein generating the packets comprises generating the outgoing request packet and the outgoing response packet responsive to respective entries placed in the schedule queues of the plurality of schedule queues.

17. A method according to claim 16, wherein the outgoing request packet and the outgoing response packet are associated with respective transport service instances among a plurality of transport service instances in use on the network, and wherein generating the outgoing request packet and the outgoing response packet comprises assigning the transport service instances of the plurality of transport service instances to the schedule queues of the plurality of schedule queues based on respective service parameters of the transport service instances of the plurality of transport service instances, and placing the entries in the schedule queues, of the plurality of schedule queues, corresponding to the transport service instances, of the plurality of transport service instances, with which the packets are associated.

18. A method according to claim 17, wherein generating the outgoing request packet and the outgoing response packet comprises allocating resources to process the schedule queues, of the plurality of schedule queues, responsive to the respective service parameters.

19. A method according to claim 17, wherein the transport service instances of the plurality of transport service instances comprise queue pairs.

20. A method according to claim 16, wherein receiving the incoming request packet further comprises receiving an incoming write request packet on a reliable transport service, and wherein generating the outgoing response packet comprises adding an entry to the entries in the schedule queues of the plurality of schedule queues, causing the outgoing packet generator, responsive to the entry, to generate an acknowledgment packet.

21. A method according to claim 16, wherein generating the outgoing write request packet and generating the outgoing read response packet both comprise writing to doorbell registers of the outgoing packet generator in order to place the entries in the schedule queues of the plurality of schedule queues.

22. A method according to claim 15, wherein receiving the incoming request packet further comprises receiving an incoming write request packet containing write data to be written to a system memory associated with the host processor after receiving the incoming RDMA read request packet, and comprising conveying the write data to the system memory using the incoming packet processor without waiting for execution of the quasi-WQE associated with the outgoing read response packet.

23. A method according to claim 15, wherein receiving the incoming request packet further comprises receiving an incoming write request packet containing write data to be written to a system memory associated with the host processor before receiving the incoming RDMA read request packet, and comprising conveying the write data to the system memory using the incoming packet processor while preventing execution of the quasi-WQE associated with the outgoing read response packet until the write data have been written to the system memory.

24. A method according to claim 15, wherein transmitting the outgoing request packet comprises passing a notification from the output packet generator to the incoming packet processor to await the incoming response packet to be received in response to the outgoing request packet, and comprising writing a completion message to the host processor when the incoming packet processor receives the awaited packet.

25. A method according to claim 15, wherein receiving the incoming RDMA read request packet comprises receiving a plurality of incoming RDMA read request packets, and wherein writing the quasi-WQE comprises writing a list of said quasi-WQEs to the first memory location, causing the outgoing packet generator to generate a sequence of corresponding outgoing response packets containing respective indicated data.

26. A method according to claim 25, wherein receiving the plurality of incoming RDMA read request packets comprises receiving the plurality of incoming RDMA read request packets over a plurality of transport service instances on the network, and wherein writing the list of the descriptors comprises writing a respective list for each transport service instance of the plurality of the transport service instances to a response database held for the plurality of the transport service instances in common, causing the outgoing packet generator to generate the packets for transmission over the plurality of the transport service instances.

27. A method according to claim 26, wherein the transport service instances of the plurality of transport service instances comprise queue pairs.

28. A method according to claim 15, wherein the request comprises a write request, which is submitted by the host processor by generating a request descriptor in a second memory location indicating further data to be read from the system memory for inclusion in the outgoing request packet, and wherein generating the outgoing request packet comprises reading the request descriptor from the second memory location and, responsive thereto, generating a write request packet containing the indicated further data.