High performance memory based communications interface

• US 20070223483A1
• Filed: 09/26/2006
• Published: 09/27/2007
• Est. Priority Date: 11/12/2005
• Status: Active Grant

• Current Assignee: III Holdings 1, LLC
• Original Assignee: Liquid Computing Corporation
• Inventors: Huang, Kaiyuan, Kemp, Michael
• Granted Patent: US 7,773,630 B2
• Time in Patent Office: Days
• US Class Current: 370/394
• CPC Class Codes: H04L 47/10 Flow control; Congestion control
• CPC Class Codes: H04L 47/245 using preemption
• CPC Class Codes: H04L 47/35 by embedding flow control information in regular packets, e.g. piggybacking
• CPC Class Codes: H04L 49/90 Buffering arrangements
• CPC Class Codes: H04L 49/9021 Plurality of buffers per packet
• CPC Class Codes: H04L 49/9042 Separate storage for different parts of the packet, e.g. header and payload
• CPC Class Codes: H04L 49/9094 Arrangements for simultaneous transmit and receive, e.g. simultaneous reading/writing from/to the storage element

Abstract

Embodiments of the present invention include enhanced functionalities and components within a Communication Endpoint Processor (CEP) that act as an interface between computational and communications domains. The embodiments disclosed herein deliver a complete memory mapped high performance interface that has the ability to support the simultaneous transmission of multiple frames of multiple sizes, and that has the ability to interrupt the transmission of lower priority frames in order to send higher priority frames.

Claims

1. A method for transmitting a data packet to a communications medium, the data packet including a header and a payload, the method comprising the steps of:

segmenting the data packet into a plurality of fragments stored in non-contiguous portions of a computer memory;

generating a fragment descriptor for each of the plurality of fragments such that one of the generated fragment descriptors is associated with the header and at least one other of the generated fragment descriptors is associated with the payload;

storing the generated fragment descriptors in contiguous portions of the computer memory;

sending the stored plurality of fragment descriptors across an interface;

retrieving the sent fragment descriptors from the interface;

reassembling the segmented data packet by retrieving the header and the payload associated with the retrieved fragment descriptors from the computer memory, and

transmitting the reassembled data packet to the communications medium.

2. The method of claim 1, wherein the segmenting, generating, storing and sending steps are carried out by a communications endpoint processor (CEP) kernel.

3. The method of claim 1, wherein the segmenting, storing and sending steps are carried out in protected kernel space.

4. The method of claim 1, wherein the segmenting, storing and sending steps are carried out in user space.

5. The method of claim 2, wherein at least some of the functionality of the CEP kernel is carried out by software.

6. The method of claim 1, wherein the retrieving, reassembling and transmitting steps are carried out by a CEP engine.

7. The method of claim 6, wherein at least some of the functionality of the CEP engine is carried out by hardware.

8. The method of claim 1, wherein the segmenting step is carried out such that each fragment descriptor includes a data structure that defines a physical address in the computer memory of its corresponding fragment.

9. The method of claim 1, wherein the segmenting step is carried out such that each fragment descriptor includes a pointer to a physical address in the computer memory of its corresponding fragment, a length field indicating a length in bytes of its corresponding fragment, a flag field indicating whether its corresponding fragment is a last fragment of the payload, and an instance ID field.

10. The method of claim 9, wherein the segmenting step is carried out such that each fragment descriptor of each fragment into which the data packet is segmented stores a same instance ID in the instance ID field.

11. The method of claim 1, further comprising generating a segment descriptor that is associated with all of the fragment descriptors of the fragments into which the data packet is segmented.

12. The method of claim 1, wherein the segmenting step is carried out with the data packet being categorized into a medium data packet whose size is less than or equal to a predetermined maximum size or into a large data packet whose size exceeds the predetermined maximum size.

13. The method of claim 12, wherein the segmenting step segments the large data packet into at least two segments whose size is equal to or less than the predetermined maximum size.

14. The method of claim 13, wherein each of the at least two segments is associated with a respective segment descriptor and wherein each of the at least two segments includes a same header and a plurality of fragments, the header and the plurality of fragments each being associated with a respective fragment descriptor.

15. The method of claim 14, wherein the segmenting step includes a chaining step to chain the respective segment descriptors associated with the at least two segments.

16. The method of claim 15, wherein chaining step includes generating a first segment chain descriptor that includes a start address field configured to store a start address of a first fragment descriptor of a first fragment of a first one of the at least two segments and at least one second segment chain descriptor between other ones of the at least two segments.

17. The method of claim 14, wherein each respective segment descriptor includes an instance ID field and wherein the segmenting step includes storing unique and sequentially numbered instance IDs in the respective instance ID fields.

18. A computer communications interface for transmitting a data packet to a communications medium, the data packet including a header and a payload, comprising:

a communications endpoint processor (CEP) interface;

a CEP kernel coupled to the CEP interface, the CEP kernel being configured to segment the data packet into a plurality of fragments, to store the plurality of fragments in non-contiguous portions of a computer memory, to generate a fragment descriptor for each of the plurality of fragments such that one of the generated fragment descriptors is associated with the header and at least one other of the generated fragment descriptors is associated with the payload, to store the generated fragment descriptors in contiguous portions of the computer memory and to send the stored plurality of fragment descriptors across the CEP interface, and

a CEP engine coupled to the CEP interface, the CEP engine being configured to retrieve the fragment descriptors from the interface, to reassemble the segmented data packet by retrieving the header and the payload associated with the retrieved fragment descriptors from the computer memory, and to transmit the reassembled data packet to the communications medium.

19. The computer communications interface of claim 18, wherein at least some of the functionality of the CEP kernel is implemented in software.

20. The computer communications interface of claim 18, wherein at least some of the functionality of the CEP engine is implemented in hardware.

21. The computer communications interface of claim 18, wherein the CEP kernel is configured to segment the data packet into a plurality of fragments such that each fragment descriptor includes a data structure that defines a physical address in the computer memory of its corresponding fragment.

22. The computer communications interface of claim 18, wherein the CEP kernel is further configured to segment the data packet into a plurality of fragments such that each fragment descriptor includes a pointer to a physical address in the computer memory of its corresponding fragment, a length field indicating a length in bytes of its corresponding fragment, a flag field indicating whether its corresponding fragment is a last fragment of the payload and an instance ID field.

23. The computer communications interface of claim 22, wherein CEP kernel is further configured to segment the data packet into a plurality of fragments such that each fragment descriptor of each fragment into which the data packet is segmented stores a same instance ID in the instance ID field.

24. The computer communications interface of claim 18, wherein the CEP kernel is further configured to generate a segment descriptor that is associated with all of the fragment descriptors of the fragments into which the data packet is segmented.

25. The computer communications interface of claim 18, wherein the data packet is categorized into a medium data packet whose size is less than a predetermined maximum size or into a large data packet whose size exceeds the predetermined maximum size.

26. The computer communications interface of claim 25, wherein the CEP kernel is further configured to segment the data packet such that the large data packet is segmented into at least two segments whose size is equal to or less than the predetermined maximum size.

27. The computer communications interface of claim 26, wherein each of the at least two segments is associated with a respective segment descriptor and wherein each of the at least two segments includes a same header and a plurality of fragments, the header and the plurality of fragments each being associated with a respective fragment descriptor.

28. The computer communications interface of claim 27, wherein the CEP kernel is further configured to chain the respective segment descriptors associated with the at least two segments.

29. The computer communications interface of claim 28, wherein the CEP kernel is further configured to generate a first segment chain descriptor that includes a start address field configured to store a start address of a first fragment descriptor of a first fragment of a first one of the at least two segments and at least one second segment chain descriptor between other ones of the at least two segments.

30. The computer communications interface of claim 27, wherein each respective segment descriptor includes an instance ID field and wherein the segmenting step includes storing unique and sequentially numbered instance IDs in the respective instance ID fields.

31. The computer communications interface of claim 18, wherein the CEP kernel runs in protected kernel space.

32. The computer communications interface of claim 18, wherein the CEP kernel runs in user space.

34. The method of claim 31, wherein the constructing step is carried out with the memory operation command being one of a remote read command;

a remote posted write command;

a remote non posted write, a remote read-modify-write, a remote DMA read, and a remote DMA write.

33. A method for a local computational host to communicate with a remote computational host, the local and remote computational hosts each comprising an operating system, a workload;

a communication endpoint processor (CEP) kernel coupled to the workload over an interface and a CEP engine coupled to the CEP kernel over a CEP engine interface, the method comprising the steps of;

the local workload invoking a one sided remote memory operation across the interface to the local CEP kernel;

the local CEP kernel constructing a memory operation command;

the local CEP kernel sending the constructed memory operation command across a local CEP kernel to remote CEP engine protocol;

the remote CEP engine consuming and executing the sent memory operation command without invoking services of the remote CEP kernel, the remote workload or of the remote operating system, and

the local CEP engine receiving one of a result of the executed memory operation command and a confirmation that the sent memory operation command has been carried out.

35. A method for optimizing a data packet in a high performance computer system, comprising the steps of:

preparing a payload of the data packet;

passing the prepared payload to a communications endpoint processor (CEP) kernel;

appending, by the CEP kernel, a header to the prepared payload to generate a CEP kernel protocol unit that is optimized for a processor environment;

transmitting, by the CEP kernel, the CEP kernel protocol unit to a CEP driver that is optimized for high throughput and low latency, and

transforming, by the CEP driver, the transmitted CEP kernel protocol unit into one of a plurality of CEP kernel protocol unit formats depending upon a size range of the payload, each of the plurality having a different format for efficient handling by a CEP engine that is configured to automate communication tasks and to interface to a system memory controller for direct memory operations.

36. The method of claim 35, wherein the size range of the payload corresponding to each of the plurality of CEP kernel protocol units is programmable.

37. The method of claim 35, wherein the transforming step transforms the CEP kernel protocol unit into a medium size CEP kernel format, a large CEP kernel format or a small CEP kernel format.

38. The method of claim 35, wherein the transforming step is carried out independently of the CEP kernel.

39. The method of claim 37, wherein the transforming step includes a step of representing the medium size CEP kernel format of the CEP kernel protocol unit in a segment descriptor that includes a definition of the physical address of all memory locations at which the data packet is stored.

40. The method of claim 39, wherein the segment descriptor includes an ordered plurality of fragment descriptors, each of the plurality of fragment descriptors corresponding to a fragment that contains a portion of the payload and defining a physical memory location of its corresponding fragment.

41. The method of claim 40, wherein the plurality of fragment descriptors are stored in order in consecutive physical memory space.

42. The method of claim 35, wherein the transforming step includes transmitting to the CEP engine a first data structure containing a virtual address in memory of the header, and at least one second data structure containing a virtual address in memory of the payload.

43. The method of claim 39, further including steps of:

transmitting, by the CEP driver, the first and the at least one second data structure to the CEP engine, and

retrieving the header and payload from the memory using the transmitted first and at least one second data structure.

44. The method of claim 37, wherein the transforming step transforms the CEP kernel protocol unit into the large CEP kernel format when the payload is larger than a largest packet size that is optimal for transmission through a communication system of the high performance computer system.

45. The method of claim 44, wherein the transforming step transforms the CEP kernel protocol unit into the large CEP kernel format by segmenting the payload into a plurality of segments, all but a last one of the plurality of segments being of a size that is equal to or less than the largest packet size.

46. The method of claim 37, wherein the transforming step transforms the CEP kernel protocol unit into the small size CEP kernel format and wherein the method further includes a step of copying the payload into a sequential physical memory segment of a main memory of the high performance computer system.

47. The method of claim 46, wherein the transforming step generates a single fragment descriptor that includes a physical memory location of the payload and wherein the method further includes a step of transmitting the single fragment descriptor to the CEP engine.

48. The method of claim 37, wherein the transforming step transforms the CEP kernel protocol unit into the small size CEP kernel format and copies the payload directly into a memory coupled to the CEP engine to which the CEP engine has direct access.

49. The method of claim 48, wherein the memory coupled to the CEP engine includes a memory mapped packet output queue.

50. The method of claim 35, further including the steps of providing the CEP engine with a descriptor output queue that is configured as a memory mapped FIFO and the CEP driver transmitting the transformed CEP kernel protocol unit to the descriptor output queue.

51. The method of claim 50, further including the steps of:

providing the CEP engine with a memory mapped descriptor output queue ready register, and

the CEP driver writing to the descriptor output queue ready register when descriptors for at least a portion of the transformed CEP kernel protocol unit have been written to the descriptor output queue.

52. The method of claim 51 wherein after the writing step, the method further includes a step of the CEP engine performing a descriptor output queue input operation to accept the descriptors written to the descriptor output queue.

53. The method of claim 50, further comprising a step of the CEP driver including, in one of the descriptors to be written to the descriptor output queue, an indication that descriptors for a CEP kernel protocol unit previously written to the descriptor output queue are ready for the CEP engine to remove from the descriptor output queue.