EFFICIENT LINK LAYER RETRY PROTOCOL UTILIZING IMPLICIT ACKNOWLEDGEMENTS

• US 20170026150A1
• Filed: 06/16/2016
• Published: 01/26/2017
• Est. Priority Date: 12/06/2013
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Birrittella, Mark S.
• Granted Patent: US 9,819,452 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: H04L 1/08 by repeating transmission, e.g. Verdan system H04L1/1858 and H04L1/189 take precedence
• CPC Class Codes: H04L 1/1657 Implicit acknowledgement of correct or incorrect reception, e.g. with a moving window
• CPC Class Codes: H04L 1/1867 Arrangements specially adapted for the transmitter end
• CPC Class Codes: H04L 1/1874 Buffer management
• CPC Class Codes: H04L 67/104 Peer-to-peer [P2P] networks

Abstract

Methods, apparatus, and systems for implementing a link layer retry protocol utilizing implicit ACKnowledgements (ACKs). Peer link interfaces are configured to facilitate confirmed error-free delivery of link-layer packets through use of implicit ACKs, while also providing retransmission of packets for which errors are detected and guaranteeing the link control data is either successfully received or data transfer over the link is prevented. In conjunction with transmitting packets, reliable packets are copied into sequential slots in a replay buffer. Each link interface tracks the slot at which each reliable packet is buffered, and in response to detection of an error, a retry request is sent to the transmit-side to retransmit the errant packet. The previously buffered copy of the errant packet is retrieved from the replay buffer and retransmitted. Through use of a link roundtrip detection mechanism, absence of a retry request by the time a replay buffer has returned to the slot of a reliable packet (plus a predetermined number of additional transfer cycles, if applicable) provides an implicit ACK that the packet was received without error.

Claims

1. An apparatus, comprising:

a link interface configured to support bi-directional communication with a link interface of a link peer when the apparatus is coupled to the link interface of the link peer via a link and operating, wherein the link interface for each of the apparatus and link peer includes,

a receive port; and

a transmit port comprising a replay buffer and circuitry and logic to,

temporarily store a copy of each of a plurality of reliable packets to be transmitted over the link to a receive port of the link peer in the replay buffer;

transmit packets including the reliable packets over the link to the receive port of the link peer;

determine that the reliable packets have been received without error at the receive port of the link peer without use of an explicit Acknowledgement (ACK) from the link peer; and

overwrite copies of previously transmitted reliable packets for which it has been determined were received at the receive port of the link peer without error.

2. The apparatus of claim 1, wherein the transmit port is further configured to detect an implicit ACK that a reliable packet has been received by the receive port of the link peer without error prior to overwriting the copy of the reliable packet in the replay buffer.

3. The apparatus of claim 2, wherein the transmit port is configured to detecting an implicit ACK by detecting that a retry request for a retransmission of a given reliable packet has not be received from the link peer within transmission of a predetermined number of packets following transmission of the given reliable packet.

4. The apparatus of claim 2, wherein the buffer comprises a circular buffer having a depth, and in which copies of reliable packets are sequential written into respective slots in a sequence that is restarted when the end of the circular buffer is reached, and wherein the transmit port circuitry and logic is configured to detect an implicit ACK by performing operations comprising:

tracking, for each transmitted reliable packet, a slot at which a copy of the reliable packet was written to in the circular buffer;

detecting that a retry request for a retransmission of a reliable packet copied to a given slot has not been received by the time the sequence for writing copies of reliable packets into the circular buffer has returned to the given slot.

5. The apparatus of claim 4, wherein the transmit port circuitry and logic is configure to:

determine that a number of slots in the circular buffer that would be written to during a roundtrip transmission between interface ports of the apparatus and the link peer is less than or equal to a depth of the circular buffer.

6. The apparatus of claim 2, wherein the buffer comprises a circular buffer having a depth, and in which copies of reliable packets are sequential written into respective slots in a sequence that is restarted when the end of the circular buffer is reached, and wherein the transmit port circuitry and logic is configured to detect an implicit ACK by performing operations comprising:

tracking, for each transmitted reliable packet, a slot at which a copy of the reliable packet was written to in the circular buffer;

detecting that a retry request for a retransmission of a reliable packet copied to a given slot has not been received by the time the sequence for writing copies of reliable packets into the circular buffer has returned to the given slot plus n additional packet transfer cycles, wherein n is an integer >

0.

7. The apparatus of claim 6, wherein the transmit port includes circuitry and logic to determine the n additional packet transfer cycles by performing operations comprising:

sending a first packet to the link peer, the first packet including indicia to solicit return of a roundtrip marker from the link peer in response to receipt of the first packet;

beginning at a starting slot, transmitting packets retrieved from slots in the replay buffer to the link peer, wherein each packet transfer cycle advances the slot in the replay buffer by one until a last slot in the replay buffer is reached, at which point the replay buffer slot wraps to a first slot;

while transmitting the packets, detecting receipt of the roundtrip marker from the link peer; and

if the buffer slot has returned to the starting slot prior to receiving the roundtrip marker, starting a counter and transmitting additional packets to the link peer until the roundtrip marker is received, wherein the counter is incremented by one for each additional packet that is transmitter and wherein n is the value of the counter when the roundtrip marker is received.

8. The apparatus of claim 6, wherein the transmit port includes circuitry and logic to determine the n additional packet transfer cycles by performing operations comprising:

sending a first packet to the link peer, the first packet including indicia to solicit return of a roundtrip marker from the link peer in response to receipt of the first packet;

beginning at a starting slot, transmitting packets retrieved from slots in the replay buffer to the link peer, wherein each packet transfer cycle advances the slot in the replay buffer by one until a last slot in the replay buffer is reached, at which point the replay buffer slot wraps to a first slot;

while transmitting the packets, detecting receipt of the roundtrip marker from the link peer; and

if the roundtrip marker is received prior to the buffer slot returning to the starting slot, starting a counter at a value m and transmitting additional packets to the link peer until the buffer slot has returned to the starting slot, wherein the counter is decremented by one for each additional packet that is transmitted and wherein n is the value of the counter when the roundtrip marker is received.

9. The apparatus of claim 1, wherein the receive port includes circuitry and logic to:

receive and process packets transmitted from the transmit port of the link peer when operating in a normal active link transfer state;

detect sequential received packets are bad packets; and

in response to detecting sequential bad packets, performing at least one of taking the link offline, initiating retraining of the link, and initiating reinitialization of the link;

and wherein the transmit port includes further circuitry and logic to send link transfer control packets in sequential pairs such that at least one of the sequential pair of link transfer control packets is either received without error or the link is taken offline, retraining of the link is initiated, or reinitialization of the link is initiated.

10. The apparatus of claim 1, wherein the receive port includes circuitry and logic to:

receive packets transmitted from the transmit port of the link peer;

detect that a bad packet containing errant data has been received; and

send a retry request to the link peer requesting the link peer resend the bad packet.

11. The apparatus of claim 10, wherein the receive port further includes circuitry and logic to:

determine a buffer location in a replay buffer in the transmit port of the link peer at which a non-errant copy of the bad packet is temporarily stored; and

embed indicia in the retry request identifying the buffer location.

12. The apparatus of claim 10, wherein the receive port further includes circuitry and logic to:

in response to detecting the bad packet,

switch a state of the receive port to a tossing mode under which received packets are discarded; and

return the state of the receive port to a normal receive mode to receive a retransmission of the bad packet from the link peer.

13. The apparatus of claim 12, wherein the receive port further includes circuitry and logic to:

receive a retry marker sent from the link peer while the receive port is in tossing mode; and

prepare to return the state of the receive port to a normal receive mode after receiving the retry marker, wherein the retry marker indicates how many packets following transmission of the retry marker will be transmitted from the link peer prior to retransmission of the non-errant copy of the bad packet.

14. The apparatus of claim 13, further comprising circuitry and logic in the receive port and transmit port to:

in response to detecting receipt of a retry marker, transmitting a roundtrip marker from the transmit port to the link peer.

15. The apparatus of claim 10, further comprising circuitry and logic in the receive port and transmit port to:

receive a retry request sent from the link peer, the retry request identifying a slot in the replay buffer of the transmit port storing a non-errant copy of the packet that was received at the receive port of the link peer as a bad packet; and

sequentially retransmit copies of packets in the replay buffer for the transmit port, beginning with the copy of the packet stored in the slot identified by the retry request.

16. The apparatus of claim 15, further comprising circuitry and logic in the transmit port to transmit a retry marker identifying a number of packets that are to be transmitted between the retry marker and retransmission of the non-errant copy of the bad packet.

17. A method, comprising:

temporarily storing, in a buffer, a copy of each of a plurality of reliable packets to be transmitted over a link to a receiver;

transmitting packets including the reliable packets over the link to the receiver;

determining that the reliable packets have been received without error at the receiver without use of explicit Acknowledgements (ACKs) from the receiver; and

overwriting copies of previously transmitted reliable packets for which it has been determined were received at the receiver without error.

18. The method of claim 17, further comprising:

detecting an implicit ACK that a reliable packet has been received without error prior to overwriting the copy of the reliable packet in the buffer.

19. The method of claim 18, wherein the buffer comprises a circular buffer having a depth, and in which copies of reliable packets are sequential written into respective slots in a sequence that is restarted when the end of the circular buffer is reached, and detecting an implicit ACK comprises:

tracking, for each transmitted reliable packet, a slot at which a copy of the reliable packet was written to in the circular buffer;

determining that a number of slots in the circular buffer that would be written to during a link roundtrip transmission is less than or equal to a depth of the circular buffer.

detecting that a retry request for a retransmission of a reliable packet copied to a given slot has not been received by the time the sequence for writing copies of reliable packets into the circular buffer has returned to the given slot.

20. The method of claim 19, wherein the buffer comprises a circular buffer having a depth, and in which copies of reliable packets are sequential written into respective slots in a sequence that is restarted when the end of the circular buffer is reached, the method further comprising:

determining a number of non-reliable packets to transmit after cycling through the circular buffer;

tracking, for each transmitted reliable packet, a slot at which a copy of the reliable packet was written to in the circular buffer; and

detecting the implicit ACK by detecting that a retry request for a retransmission of a reliable packet copied to a given slot has not been received by the time the sequence for writing copies of reliable packets into the circular buffer has returned to the given slot and the determined number of non-reliable packets have been retransmitted.

21. The method of claim 17, wherein the link comprises a bi-directional link coupled between a first port including a first transmitter and first receiver and a second port including a second transmitter and second receiver, wherein the transmitter comprises the first transmitter and the receiver comprises the second receiver, the method further comprising:

receiving packets transmitted from the second transmitter at the first receiver;

detecting that a bad packet containing errant data has been received; and

sending a retry request to the second transmitter requesting the second transmitter resend the bad packet.

22. The method of claim 21, further comprising:

determining, at the first receiver, a buffer location in a buffer in the second transmitter at which a copy of errant packet is temporarily stored; and

including indicia in the retry request indicating the buffer location.

23. A method comprising:

reliably transferring packets over a link coupled between link interface peers using a replay mechanism under which reliable packets for which errors are detected are retransmitted, and successful transmission of data over the link is confirmed using implicit ACKnowledgements (ACKs); and

ensuring the link is operating reliably by guaranteeing that link transfer control data is either successfully received or a state of the link is changed to a link state under which transfer of reliable packets over the link is unavailable.

24. The method of claim 23, further wherein the method supports reliable transmission of reliable packets and ensures the link is operating reliably for a link having substantially any length.

25. The method of claim 23, wherein ensuring the link is operating reliably comprises detecting errors in sequential packets and transmitting link transfer control packets containing link transfer control data in sequential pairs such that at least one of the sequential pair of link transfer control packets is either received without error or the link is taken offline, retraining of the link is initiated, or reinitialization of the link is initiated.

26. A set of components, comprising:

first and second chips including respective first and second link interfaces, each of the first and second link interfaces including a transmit port and a receive port and configured to perform bi-directional communication with a link peer including a link interface with a similar configuration when installed in a system and operating, the system including a fabric comprising a plurality of point-to-point fabric links;

wherein each of first and second chips are configured to be implemented in respective fabric endpoints coupled to a fabric link, comprise respective fabric endpoints coupled to a fabric link, or be implemented in a fabric switch to which a plurality of fabric links are coupled;

and wherein the transmit ports and receive ports of the first and second link interfaces include circuitry and logic to perform operations when the first and second chips are implemented in the system including,

transmitting packet data from the first link interface over a fabric link to the second link interface; and

detecting at the first link interface the packet data has been received without error at the second link interface without using explicit ACKnowledgements (ACKs).

27. The set of components of claim 26,

wherein the transmit port of the first link interface is configured to:

generate a plurality of a first type of packets;

divide each first type of packet into a plurality of data units;

bundle the data units into groups to form a plurality of a second type of packets; and

transmit the plurality of second type of packets onto a fabric link coupled to the first link interface;

and wherein the receive port of the second link interface is configured to,

receive second type of packets transmitted over a fabric link coupled to the second link interface;

for each second type of packet that is received, detect whether it has any errors; and

extract data units from second type of packets for which an error is not detected and reassemble the data units to regenerated the plurality of first type of packets.

28. The set of components of claim 27, further comprising a fabric switch having third and fourth link interfaces, each including a transmit port and a receive port, wherein the first link interface that is connected to the third link interface on the fabric switch via a first fabric link and the fourth link interface on the fabric switch is connected to the second link interface via a second fabric link, and wherein:

the transmitter of the first link interface is configured to bundle data units from the first type of packets into a first plurality of second type of packets and transmit the first plurality of second type of packets over the first fabric link to a receive port on the third link interface;

the fabric switch is configured to extract data units from the first plurality of second type of packets received at the receive port of the third link interface, and rebundle the data units to generate a second plurality of second type of packets and transmit the second plurality of second type of packets from a transmit port of the fourth link interface over the second fabric link to a receive port of the second link interface; and

the receive port of the second link interface is configured to extract error-free data units from the second plurality of second type of packets and reassemble the error free data units to regenerate the first type of packets,

and wherein the transmit ports for each of the first and fourth link interfaces are configured to determine each second type of packet transmitted from that transmit port has been received without error by a receive port of a peer link interface without use of explicit ACKs.

29. The set of components of claim 27, wherein each transmit port is configured to transmit reliable and non-reliable second type of packets and includes a replay buffer in which copies of each reliable second type of packet transmitted from that transmit port is temporarily buffered, wherein each link interface is configured to send a retry request back to a link interface peer in response to detection of a second type of packet that has been received with an error;

and wherein each transmit port is configured to retransmit copies of reliable second type of packets from its replay buffer in response to receiving a retry request at the receive port of the link interface comprising that transmit port.

30. The set of components of claim 26, wherein the first and second chips comprise host fabric interface chips including an integrated link interface.

31. The set of components of claim 26, wherein at least one of the first and second chips comprises a System on a Chip (SoC) including a processor operatively coupled to a host fabric interface including an integrated link interface.

32. A method implemented in a system including a fabric having a plurality of point-to-point fabric links to which first and second fabric endpoints are coupled, the method comprising:

transmitting packet data from the first fabric endpoint to the second fabric endpoint via at least one fabric link; and

determining, for each fabric link, that packet data has been received without error using implicit ACKnowledgements (ACKs).

33. The method of claim 32, further comprising:

generating a plurality of fabric packets at the first fabric endpoint;

dividing each fabric packet into a plurality of flits;

bundling the data units into groups to form a plurality of a link transfer packets (LTPs);

transmitting the plurality of LTPs onto a fabric link coupled to the first fabric endpoint;

receiving a second plurality of LTPs from a fabric link coupled to the second fabric endpoint;

for each LTP that is received, detecting whether it has any errors; and

extracting flits from LTPs for which an error is not detected and reassembling the flits to regenerate the plurality of first fabric packets.

34. The method of claim 33, wherein the first and second fabric endpoints include respective first and second link interfaces and the fabric includes at least one fabric switch having a plurality of link interfaces, the method further comprising:

transmitting, from a transmit port of the first link interface, the first plurality of LTPs onto a first fabric link coupled between the first link interface and a third link interface on the fabric switch;

receiving the first plurality of LTPs at the third link interface;

checking errors for each LTP, and extracting flits from the LTPs for which an error is not detected; and

rebundling flits to form the second plurality of LTPs and transmitting the second plurality of LTPs from a transmit port of a fourth link interface on the fabric switch over a second fabric link coupled between the fourth link interface and the second link interface of the second fabric endpoint,

wherein the transmit ports for each of the first and fourth link interfaces determine each LTP transmitted from that transmit port has been received without error by a receive port of a peer link interface using implicit ACKs.

35. The method of claim 33, wherein each transmit port transmits reliable and non-reliable LTPs and temporarily buffers each reliable LTP it transmits in a replay buffer, wherein each link interface sends a retry request back to a link interface peer in response to detection of an LTP has been received with an error;

and wherein each transmit port retransmits copies of reliable LTPs from its replay buffer in response to receiving a retry request at the receive port of the link interface comprising that transmit port.