System and method for explict communication of messages between processes running on different nodes in a clustered multiprocessor system

• US 20040083317A1
• Filed: 10/23/2002
• Published: 04/29/2004
• Est. Priority Date: 10/23/2002
• Status: Active Grant

• Current Assignee: FG SRC LLC (Freeman Capital Partners LP)
• Original Assignee: SRC Computers, Inc.
• Inventors: OʼConnor, James, Poznanovic, Daniel, Caliga, David, Dickson, Christopher
• Granted Patent: US 7,124,211 B2
• Time in Patent Office: Days
• US Class Current: 710/22
• CPC Class Codes: G06F 13/1663 Access to shared memory
• CPC Class Codes: G06F 13/28 using burst mode transfer, e.g. direct memory access DMA, cycle steal G06F13/32 takes precedence
• CPC Class Codes: G06F 9/544 Buffers; Shared memory; Pipes

Abstract

Embodiments of the invention include a mechanism for explicit communication in a clustered multiprocessor system that supports low-latency, protected, user-mode, communication across the machine boundaries of a clustered multiprocessor. Data transport may be accomplished over persistent, unidirectional, point-to point connections, each of which may be embodied in a small amount of state at each end, along with a statically allocated per-connection memory buffer, which may be directly accessible to the transport mechanism at both ends of each notional link. System Memory protection may be afforded by operating system (“OS”) facilitated allocation of both restricted control of the network interface, and responsibility for data transport, to an application thread that may execute in the context of the processor-managed virtual address space. Connection buffer protection may be afforded by restricting access to connection state to those entries associated with the currently controlling thread.

Claims

1. An OS-bypass message transport mechanism to communicate between processor nodes that execute a process on a clustered multiprocessor system, said transport mechanism comprising:

a connection that includes a send side coupled to a source processor node and a receive side coupled to a destination processor node; and

a per-connection memory buffer, and coupled to both the send side and the receive side of the connection,

wherein the send side includes an output data buffer coupled between the source processor node and the per-connection memory buffer and the receive side includes an input data buffer coupled between the per-connection memory buffer and the destination processor node, and

wherein the per-connection memory buffer is dedicated to the process.

2. The OS-bypass message transport mechanism of claim 1, wherein the per-connection memory buffer comprises a portion of a common memory of the clustered multiprocessor system.

3. The OS-bypass message transport mechanism of claim 1, wherein said send side includes outgoing direct memory access (ODMA) engine that controls the transfer of data from the output data register to the per-connection memory buffer.

4. The OS-bypass message transport mechanism of claim 1, comprising an IPI generator coupled to the send side and the receive side, wherein the IPI generator generates control data shared between the send side and receive side.

5. The OS-bypass message transport mechanism of claim 4, comprising an IPI handler coupled to the send side and the receive side for processing the control data generated by the IPI generator.

6. The OS-bypass message transport mechanism of claim 1, wherein said output data buffer and input data buffer comprise first-in-first-out (FIFO) buffers.

7. The OS-bypass message transport mechanism of claim 1, wherein said connection comprises a switch/network adaptor port (SNAP).

8. The OS-bypass message transport mechanism of claim 1, wherein the send side includes an outgoing local connection table (OLCT) and the receive side includes an incoming local connection table (ILCT), and

wherein a connection is established between the send side and the receive side by entries in the OLCT and ILCT.

9. The OS-bypass message transport mechanism of claim 8, wherein the connection is a persistent, unidirectional connection.

10. The OS-bypass message transport mechanism of claim 1, wherein said processor nodes are coupled to a multi-adaptive processor (MAP) element.

13. The OS-bypass message transport mechanism of claim 1, wherein the per-connection memory buffer is statically allocated.

11. The OS-bypass message transport mechanism of claim 11, wherein the processor nodes coupled to the MAP element are configured for a dual in-line memory module (DIMM) socket.

14. A clustered multiprocessor system comprising:

an OS-bypass message transport mechanism to communicate between processor nodes that execute a process on the system, wherein said mechanism includes

a connection that includes a send side coupled to a source processor node and a receive side coupled to a destination processor node; and

a per-connection memory buffer coupled to both the send side and the receive side of the connection,

wherein the send side includes an output data buffer coupled between the source processor node and the per-connection memory buffer and the receive side includes an input data buffer coupled between the per-connection memory buffer and the destination processor node, and

wherein the per-connection memory buffer is dedicated to the process.

15. A method of transferring process data from a source processor node to a destination processor node that bypasses an operating system in a clustered multiprocessor system, said method comprising the steps of:

allocating a per-connection memory buffer accessible by both the source processor node and the destination processor node;

transferring the process data from the source processor node to an output data buffer on the send side;

transferring the process data from the output data buffer to the per-connection memory buffer;

transferring the process data from the per-connection memory buffer to an input data buffer on the receive side; and

processing the process data from the input data buffer using the destination processor node.

16. The method of claim 15, wherein the per-connection memory buffer is statically allocated.

17. The method of claim 15, further comprising the step of restricting the per-connection memory buffer to the process data while transferring the process data from the source processor node to the destination processor node.

18. The method of claim 15, wherein the source processor node and the destination processor node are coupled in a duel in-line memory module (DIMM) format.

19. The method of claim 15, wherein the connection includes a switch/network adaptor port.