Reliable message transmission with packet-level resend

• US 7,013,419 B2
• Filed: 04/11/2002
• Issued: 03/14/2006
• Est. Priority Date: 04/11/2001
• Status: Active Grant

• Current Assignee: Mellanox Technologies Limited (NVIDIA Corporation)
• Original Assignee: Mellanox Technologies Limited (NVIDIA Corporation)
• Inventors: Crupnicoff, Diego, Kagan, Michael, Bloch, Gil, Shachar, Ariel, Levenvirth, Dafna
• Primary Examiner(s): Eng, David Y.
• Application Number:
  US10/119,808
• Publication Number:
  US 20020152315A1
• Time in Patent Office: 1,433 Days
• Field of Search: 714/749
• US Class Current: 714/749
• CPC Class Codes: G06F 13/1605 based on arbitration arbitration in handling access to a common bus or bus system G06F13/36
• CPC Class Codes: H04L 49/90 Buffering arrangements

Abstract

A method for communication over a network includes receiving from a host processor a descriptor defining a message including message data to be sent over the network, and responsive to the descriptor, generating a sequence of packets each containing a respective portion of the message data. An indication is entered in a selected packet among the packets in the sequence, other than the final packet, requesting that a recipient of the packets acknowledge the selected packet. Following an interruption in the sequence of the packets subsequent to the selected packet, sending of the packets in the sequence resumes beginning after the selected packet.

Claims

1. A method for communication over a network, comprising:

receiving from a host processor a descriptor defining a message comprising message data to be sent over the network;

responsive to the descriptor, generating a sequence of packets each containing a respective portion of the message data, the sequence ending in a final packet containing a final portion of the message data;

entering in a selected packet among the packets in the sequence, other than the final packet, an indication requesting that a recipient of the packets acknowledge the selected packet;

sending the packets in the sequence to the recipient; and

following an interruption in the sequence of the packets subsequent to the selected packet, resuming sending the packets in the sequence beginning after the selected packet.

2. A method according to claim 1, wherein receiving the descriptor comprises receiving a work request (WR) from the host processor, and wherein generating the sequence of the packets comprises executing a work queue item (WQE) responsive to the work request.

3. A method according to claim 2, wherein executing the WQE comprises placing the WQE in a send queue of a queue pair (QP) associated with a reliable transport service provided by the network.

4. A method according to claim 3, wherein entering the indication in the selected packet comprises noting a packet sequence number (PSN) of the selected packet in context data associated with the QP, and comprising receiving an acknowledgment of the selected packet from the recipient, responsive to the indication, and updating the context data responsive to the acknowledgment.

5. A method according to claim 1, wherein entering the indication in the selected packet comprises setting an acknowledgment request flag in a header of the selected packet.

6. A method according to claim 1, wherein receiving the descriptor comprises receiving a gather list, indicating addresses and lengths of the message data in a memory, and wherein generating the sequence of the packets comprises reading the message data from the memory responsive to the gather list, and wherein entering the indication in the selected packet comprises noting a point in the gather list corresponding to the message data contained in the selected packet.

7. A method according to claim 6, wherein resuming sending the packets comprises generating the packets in the sequence beginning from the noted point in the gather list.

8. A method according to claim 1, wherein entering the indication in the selected packet comprises selecting the packet in which to enter the indication responsive to a predetermined quantity of the data having been sent since a preceding one of the packets in the sequence that contained the indication was sent to the recipient.

9. A method according to claim 8, wherein selecting the packet comprises setting a parameter indicating a threshold number of the packets, and wherein selecting the packet comprises setting the indication in at least one of the packets in the sequence every time the threshold number of the packets is sent.

10. A method according to claim 1, wherein entering the indication comprises receiving an instruction to discontinue generating the packets in the sequence before generating the final packet, and entering the indication in the packet at which generating the packets is discontinued.

11. A method according to claim 10, wherein receiving the descriptor comprises receiving a first work request to be executed on a first transport service instance having a first priority, and wherein receiving the instruction to discontinue generating the packets comprises preempting generation of the packets for the first transport service instance in order to execute a second work request on a second transport service instance having a second priority, higher than the first priority.

12. A method according to claim 10, wherein receiving the descriptor comprises receiving a first work request to be executed on a first transport service instance, and wherein receiving the instruction to discontinue generating the packets comprises determining that a quantity of the data sent on the first transport instance has reached a predetermined threshold, whereupon generation of the packets for the first transport service instance should be terminated in order to execute a second work request on a second transport service instance.

13. A method according to claim 1, and comprising receiving an acknowledgment of the selected packet from the recipient, responsive to the indication, wherein resuming sending the packets in the sequence comprises choosing a point in the sequence after the selected packet with which to resuming sending the packets, responsive to having received the acknowledgment.

14. A method according to claim 13, wherein entering the indication in the selected packet comprises noting an identification of the selected packet in contexts data used in generating the packets, and wherein receiving the acknowledgment comprises updating the context data responsive to the acknowledgment, so as to indicate the point in the sequence at which to resume sending the packets following the interruption.

15. A method according to claim 13, wherein resuming sending the packets comprises resending the packets following the selected packet responsive to failing to receive the acknowledgment of a further packet in the sequence subsequent to the selected packet.

16. A method according to claim 15, wherein failing to receive the acknowledgment comprises failing to receive the acknowledgment within a predetermined timeout period.

17. A method according to claim 15, wherein failing to receive the acknowledgment comprises receiving a response from the recipient reporting that it did not properly receive one of the packets in the sequence subsequent to the selected packet.

18. A method according to claim 15, wherein resuming sending the packets comprises, responsive to failing to receive the acknowledgment of the further packet, interrupting generating the sequence of packets, so as to resend the packets following the selected packet.

19. An interface device for communication over a network, comprising:

a host interface, adapted to receive from a host processor a descriptor defining a message comprising message data to be sent over the network;

an execution unit, coupled to the host interface and adapted, responsive to the descriptor, to generate a sequence of packets each containing a respective portion of the message data, the sequence ending in a final packet containing a final portion of the message data, such that an indication requesting that a recipient of the packets acknowledge the selected packet is entered in a selected packet among the packets in the sequence, other than the final packet, the execution unit being further adapted, following an interruption in the sequence of the packets subsequent to the selected packet, to resume sending the packets in the sequence beginning after the selected packet; and

a network interface, coupled to the execution unit so as to send the packets in the sequence to the recipient over the network and to receive an acknowledgment of the selected packet sent by the recipient in response to the indication in the selected packet.

20. A device according to claim 19, wherein the descriptor is submitted by the host processor to the interface device as a work request (WR), causing the execution unit to execute a work queue item (WQE) responsive to the work request.

21. A device according to claim 20, wherein the execution unit is adapted to place the WQE in a send queue of a queue pair (QP) maintained by the execution unit, wherein the QP is associated with a reliable transport service provided by the network.

22. A device according to claim 21, wherein the execution unit is adapted to record a packet sequence number (PSN) of the selected packet in context data associated with the QP, and comprising a transport check unit, which is coupled to receive the acknowledgment from the network interface and to update the context data responsive to the acknowledgment.

23. A device according to claim 19, wherein the indication comprises an acknowledgment request flag, which is set in a header of the selected packet to request that the recipient acknowledge the selected packet.

24. A device according to claim 19, wherein the descriptor comprises a gather list, indicating addresses and lengths of the message data in a memory, and wherein the execution unit is adapted to read the message data from the memory responsive to the gather list, and to note a point in the gather list corresponding to the message data contained in the selected packet.

25. A device according to claim 24, wherein to resume sending the packets following the interruption, the execution unit is adapted to generate the packets in the sequence beginning from the noted point in the gather list.

26. A device according to claim 19, wherein the execution unit is adapted to select the packet in which to enter the indication responsive to a predetermined quantity of the data having been sent since a preceding one of the packets in the sequence that contained the indication was sent to the recipient.

27. A device according to claim 26, wherein the predetermined quantity of the data is determined by a parameter indicating a threshold number of the packets, such that the indication in at least one of the packets in the sequence every time the threshold number of the packets is sent.

28. A device according to claim 19, wherein the execution unit is adapted to receive an instruction to discontinue generating the packets in the sequence before generating the final packet, and to enter the indication in the packet at which it discontinues generating the packets is discontinued.

29. A device according to claim 28, wherein the descriptor corresponds to a first work request to be executed on a first transport service instance having a first priority, and wherein generating the packets for the first transport service instance is discontinued in order to enable the execution unit to execute a second work request on a second transport service instance having a second priority, higher than the first priority.

30. A device according to claim 28, wherein the descriptor corresponds to a first work request to be executed on a first transport service instance, and wherein generating the packets for the first transport service instance is discontinued upon determining that a quantity of the data sent on the first transport instance has reached a predetermined threshold, in order to enable the execution unit to execute a second work request on a second transport service instance.

31. A device according to claim 19, wherein the execution unit is adapted to choose a point in the sequence after the selected packet with which to resume sending the packets responsive to the acknowledgment having been received from the recipient of the selected packet in response to the indication.

32. A device according to claim 31, wherein the execution unit is adapted to note an identification of the selected packet in context data used in generating the packets, and comprising a transport check unit, which is coupled to receive the acknowledgment from the network interface and to update the context data responsive to the acknowledgment, so as to indicate the point in the sequence at which to resume sending the packets following the interruption.

33. A device according to claim 31, wherein the execution unit is adapted to resend the packets following the selected packet responsive to a failure to receive at the network interface the acknowledgment of a further packet in the sequence subsequent to the selected packet.

34. A device according to claim 33, wherein the failure to receive the acknowledgment of the further packet is considered to have occurred when the acknowledgment is not received within a predetermined timeout period.

35. A device according to claim 33, wherein the failure to receive the acknowledgment of the further packet is considered to have occurred when a response is received from the recipient reporting that the recipient did not properly receive one of the packets in the sequence subsequent to the selected packet.

36. A device according to claim 33, and comprising a transport check unit, which is coupled to detect the failure to receive the acknowledge, and responsive thereto, to interrupt generation of the sequence of packets by the execution unit, so as to cause the execution unit to resend the packets following the selected packet.