System and method for preserving message order while employing both programmed I/O and DMA operations

• US 5,887,134 A
• Filed: 06/30/1997
• Issued: 03/23/1999
• Est. Priority Date: 06/30/1997
• Status: Expired due to Term

• Current Assignee: Oracle America, Inc. (Oracle Corporation)
• Original Assignee: Sun Microsystems (Oracle Corporation)
• Inventors: Ebrahim, Zahir
• Primary Examiner(s): Coleman, Eric
• Assistant Examiner(s): Monestime, Mackly
• Application Number:
  US08/885,151
• Time in Patent Office: 631 Days
• Field of Search: 395/200.3, 395/200.53, 395/200.5
• US Class Current: 709/200
• CPC Class Codes: G06F 15/17331 Distributed shared memory [DSM], e.g. remote direct memory access [RDMA]

Abstract

In a cluster of computer nodes, each node has network interface and at least one processor. Transmission of a multipart message from a first node to a second node is initiated by sending to a network interface of the first node a sequence of PIO store and DMA store commands, each PIO store and DMA store command specifying a respective component of the multipart message to be stored in a respective specified memory mapped location in the second node, the sequence of the PIO store and DMA store commands corresponding to a predefined message component order. The first node'"'"'s network interface packetizes the sequence of PIO and DMA commands to generate an ordered stream of data transfer packets whose order corresponds to the predefined message component order, and transmits the ordered stream of data transfer packets to the second node so as to store the respective components of the multipart message in their respective specified memory locations in the second node in the predefined message component order.

Claims

1. A method of transmitting messages between nodes of a distributed computer system using a combination of memory mapped programmed I/O (PIO) commands and memory mapped DMA commands to transmit each of a subset of the messages, comprising steps of:

initiating the transmission of a multipart message from a first node to a second node of the distributed computer system by sending to a network interface of the first node a sequence of PIO store and DMA store commands, each PIO store and DMA store command specifying a respective component of the multipart message to be stored in a respective specified memory mapped location in the second node, the sequence of the PIO store and DMA store commands corresponding to a predefined message component order;

at the first node'"'"'s network interface;

receiving the sequence of PIO and DMA commands corresponding to the multipart message;

storing each PIO and DMA command received by the first node'"'"'s network interface in a queue, wherein the PIO and DMA commands are stored in the queue in a same order as they are received by the first node'"'"'s network interface, which corresponds to the predefined message component order; and

packetizing the PIO and DMA commands stored in the queue to generate an ordered stream of data transfer packets whose order corresponds to the predefined message component order, and transmitting the ordered stream of data transfer packets to the second node so as to store the respective components of the multipart message in their respective specified memory locations in the second node in a same order as the predefined message component order, the transmitting step preserving packet order within the ordered stream of data transfer packets.

2. The method of claim 1, wherein

the sequence of PIO store and DMA store commands includes a sender notification command;

the first node'"'"'s network interface stores the sender notification command in the queue PIO after storing in the queue any earlier PIO store and DMA store commands in the sequence; and

the first node'"'"'s network interface executes the sender notification command by sending a notification message to a specified memory location in the first node, only after processing of all PIO store and DMA store commands stored earlier in the queue has been completed by the first node'"'"'s network interface.

3. The method of claim 2, wherein

the step of sending each DMA store command to the network interface is performed by executing a single processor store instruction to a location that is memory mapped to the network interface.

4. The method of claim 2, wherein

the step of sending each DMA store command to the network interface is performed by executing zero or more processor load instructions and a single processor store instruction to a location that is memory mapped to the network interface.

5. The method of claim 1, wherein

the sequence of PIO store and DMA store commands includes a PIO store commit command that specifies a respective component of the multipart message to be stored in a respective specified memory mapped location in the second node, wherein the PIO store commit command requires an end-to-end acknowledgment from the second node after the respective message component has been stored in the respective specified memory mapped location in the second node;

the sequence of PIO store and DMA store commands includes at least one PIO or DMA store command that does not require an end-to-end acknowledgment from the second node after the respective message component has been stored in the respective specified memory mapped location in the second node;

the first node'"'"'s network interface executes the PIO store commit command by;

(A) generating a data transfer packet containing the respective message component and an end-to-end acknowledgment indicia, (B) transmitting the data transfer packet to the second node, and (C) waiting until the end-to-end acknowledgment from the second node is received before completing execution of the PIO store commit command.

6. A method of transmitting messages between nodes of a distributed computer system using a combination of memory mapped programmed I/O (PIO) commands and memory mapped DMA commands to transmit each of a subset of the messages, comprising steps of:

establishing a plurality of queues in the first node'"'"'s network interface;

initiating the transmission of a multipart message from a first node to a second node of the distributed computer system by sending to a network interface of the first node a sequence of PIO store and DMA store commands, each PIO store and DMA store command specifying a respective component of the multipart message to be stored in a respective specified memory mapped location in the second node, the sequence of the PIO store and DMA store commands corresponding to a predefined message component order;

assigning each of the PIO store and DMA store commands that specify a respective component of the multipart message to a same one of the queues;

at the first node'"'"'s network interface;

receiving the sequence of PIO and DMA commands corresponding to the multipart message;

storing each PIO and DMA command received by the first node'"'"'s network interface in the assigned one of the queues, wherein the PIO and DMA commands stored in each queue are stored in a same order as they are received by the first node'"'"'s network interface, which corresponds to the predefined message component order;

for each queue, packetizing the PIO and DMA commands therein so as to generate an ordered stream of data transfer packets whose order corresponds to the order in which the PIO and DMA commands are stored in the queue, and transmitting each of the ordered stream of data transfer packets to a destination node specified therein, the transmitting step preserving packet order within the ordered stream of data transfer packets, at least with respect to packets being transmitted to a same destination node, so as to store the respective components of the multipart message in their respective specified memory locations in the destination node in the predefined message component order.

7. The method of claim 6, including:

each DMA store command specifying a respective source address that is a local virtual address and a respective destination address that is a local physical address;

each PIO store command specifying a respective destination address that is a local physical address;

at the first node;

establishing an outgoing memory management unit (OMMU) and an incoming memory management unit (IMMU);

storing in the OMMU entries, each OMMU entry mapping a specified range of local physical addresses to a corresponding range of global address, each OMMU entry also specifying one of the queues in the first node'"'"'s network interface in which to store PIO and DMA store commands whose destination address corresponds the range of local physical addressed mapped thereby;

storing in the IMMU entries, a first subset of the IMMU entries each mapping a specified range of global addresses to a corresponding range of local physical address, and a second subset of the IMMU entries each mapping a specified range of local virtual addresses to a corresponding range of local physical address;

at the first node'"'"'s network interface;

for each received PIO store and DMA store command, locating an OMMU entry in the OMMU corresponding to the respective destination address specified therein, and converting the respective destination address in each received PIO store and DMA store command into a corresponding global address in accordance with the located OMMU entry;

for each received DMA store command, locating an IMMU entry in the IMMU corresponding to the respective source address specified therein, and converting the respective specified source address into a corresponding local physical address in accordance with the located IMMU entry;

wherein the packetizing step is performed after said converting steps so that each data transfer packet specifies a global destination address;

said packetizing step including, for each DMA store command, retrieving data from the local physical address produced by the converting step for the DMA store command.

8. A network interface for a first node of a distributed computer system, comprising:

receive logic for receiving a sequence of PIO store and DMA store commands, each PIO store and DMA store command specifying a respective component of a multipart message to be stored in a respective specified memory mapped location in a second node of the distributed computer system, the sequence of the PIO store and DMA store commands corresponding to a predefined message component order;

the receive logic including a queue for storing each of the received PIO and DMA commands, wherein the PIO and DMA commands are stored in the queue in a same order as they are received, which corresponds to the predefined message component order;

means, coupled to the receive logic, for packetizing the PIO and DMA commands stored in the queue to generate an ordered stream of data transfer packets whose order corresponds to the predefined message component order; and

means, coupled to the packetizing means, for transmitting the ordered stream of data transfer packets to the second node so as to store the respective components of the multipart message in their respective specified memory locations in the second node in a same order as the predefined message component order;

the transmitting means preserving packet order within the ordered stream of data transfer packets.

9. The network interface of claim 8, wherein

the sequence of PIO store and DMA store commands includes a sender notification command;

the first node'"'"'s network interface stores the sender notification command in the queue after storing in the queue any earlier PIO store and DMA store commands in the sequence; and

the network interface includes state machine logic that executes the sender notification command by sending a notification message to a specified memory location in the first node, only after processing of all earlier PIO store and DMA store commands stored earlier in the queue has been completed.

10. The network interface of claim 8, wherein

the sequence of PIO store and DMA store commands includes a PIO store commit command that specifies a respective component of the multipart message to be stored in a respective specified memory mapped location in the second node, wherein the PIO store commit command requires an end-to-end acknowledgment from the second node after the respective message component has been stored in the respective specified memory mapped location in the second node;

the sequence of PIO store and DMA store commands includes at least one PIO or DMA store command that does not require an end-to-end acknowledgment from the second node;

the packetizing means generates a data transfer packet for the PIO store commit command containing the respective message component and an end-to-end acknowledgment indicia; and

the transmitting means transmits the data transfer packet for the PIO store commit command to the second node and waits until the end-to-end acknowledgment from the second node is received before completing execution of the PIO store commit command.

11. A network interface for a first node of a distributed computer system, comprising:

receive logic for receiving a sequence of PIO store and DMA store commands, each PIO store and DMA store command specifying a respective component of a multipart message to be stored in a respective specified memory mapped location in a second node of the distributed computer system, the sequence of the PIO store and DMA store commands corresponding to a predefined message component order;

the packetizing means including;

a plurality of queues;

classification logic for assigning each of the PIO store and DMA store commands that specify a respective component of the multipart message to a same one of the queues and for storing each PIO and DMA command received by the first node'"'"'s network interface in the assigned one of the queues, wherein the PIO and DMA commands stored in each queue are stored in a same order as they are received by the first node'"'"'s network interface, which corresponds to the predefined message component order; and

packetizing logic, for each queue, for packetizing the PIO and DMA commands therein so as to generate an ordered stream of data transfer packets whose order corresponds to the order in which the PIO and DMA commands are stored in the queue;

the transmitting means including means for transmitting each of the ordered stream of data transfer packets to a destination node specified therein while preserving packet order within the ordered stream of data transfer packets at least with respect to packets being transmitted to a same destination node.

12. The network interface of claim 11, wherein:

each DMA store command specifies a respective source address that is a local virtual address and a respective destination address that is a local physical address;

each PIO store command specifies a respective destination address that is a local physical address;

the network interface includes an outgoing memory management unit (OMMU) and an incoming memory management unit (IMMU), the OMMU storing a plurality of OMMU entries, each OMMU entry mapping a specified range of local physical addresses to a corresponding range of global address, each OMMU entry also specifying one of the queues in the first node'"'"'s network interface in which to store PIO and DMA store commands whose destination address corresponds the range of local physical addressed mapped thereby;

the IMMU storing a plurality of IMMU entries, a first subset of the IMMU entries each mapping a specified range of global addresses to a corresponding range of local physical address, and a second subset of the IMMU entries each mapping a specified range of local virtual addresses to a corresponding range of local physical address;

the first node'"'"'s network interface includes;

OMMU lookup logic for locating, for each received PIO store and DMA store command, an OMMU entry in the OMMU corresponding to the respective destination address specified therein, and converting the respective destination address in each received PIO store and DMA store command into a corresponding global address in accordance with the located OMMU entry;

IMMU lookup logic for locating, for each received DMA store command, an IMMU entry in the IMMU corresponding to the respective source address specified therein, and converting the respective specified source address into a corresponding local physical address in accordance with the located IMMU entry;

the packetizing logic stores in each generated data transfer packet a global destination address corresponding to a global address obtained by the OMMU lookup logic; and

the packetizing logic, when packetizing each DMA store command, retrieves data from the local physical address obtained by the IMMU lookup logic for the source address specified by the DMA store command.

13. A network interface for a first node of a distributed computer system, comprising:

receive logic for receiving a sequence of PIO store and DMA store commands, each PIO store and DMA store command specifying a respective component of a multipart message to be stored in a respective specified memory mapped location in a second node of the distributed computer system, the sequence of the PIO store and DMA store commands corresponding to a predefined message component order;

the receive logic including a queue for storing each of the received PIO and DMA commands, wherein the PIO and DMA commands are stored in the queue in a same order as they are received, which corresponds to the predefined message component order;

command processing circuitry, coupled to the receive logic, for packetizing the sequence of PIO and DMA commands stored in the queue to generate an ordered stream of data transfer packets whose order corresponds to the predefined message component order; and

packet handling circuitry, coupled to the command processing circuit, for transmitting the ordered stream of data transfer packets to the second node so as to store the respective components of the multipart message in their respective specified memory locations in the second node in the predefined message component order;

the packet handling circuitry preserving packet order within the transmitted ordered stream of data transfer packets.

14. A network interface for a first node of a distributed computer system, comprising:

receive logic for receiving a sequence of PIO store and DMA store commands, each PIO store and DMA store command specifying a respective component of a multipart message to be stored in a respective specified memory mapped location in a second node of the distributed computer system, the sequence of the PIO store and DMA store commands corresponding to a predefined message component order;

command processing circuitry, coupled to the receive logic, including;

a plurality of queues;

classification logic for assigning each of the PIO store and DMA store commands that specify a respective component of the multipart message to a same one of the queues and for storing each PIO and DMA command received by the first node'"'"'s network interface in the assigned one of the queues, wherein the PIO and DMA commands stored in each queue are stored in a same order as they are received by the first node'"'"'s network interface, which corresponds to the predefined message component order; and

a packetizing engine, for each queue, for packetizing the PIO and DMA commands therein so as to generate an ordered stream of data transfer packets whose order corresponds to the order in which the PIO and DMA commands are stored in the queue; and

packet handling circuitry, coupled to the each of the queues in the command processing circuitry, including a scheduler for scheduling transmission of the ordered stream of data transfer packets in each queue to the destination nodes specified therein while preserving packet order within each queue'"'"'s ordered stream of data transfer packets, at least with respect to packets being transmitted to a same destination node, so as to store the respective components of the multipart message in their respective specified memory locations in the destination node in the predefined message component order.

15. The network interface of claim 14, wherein:

each DMA store command specifies a respective source address that is a local virtual address and a respective destination address that is a local physical address;

each PIO store command specifies a respective destination address that is a local physical address;

the network interface includes an outgoing memory management unit (OMMU) and an incoming memory management unit (IMMU), the OMMU storing a plurality of OMMU entries, each OMMU entry mapping a specified range of local physical addresses to a corresponding range of global address, each OMMU entry also specifying one of the queues in the first node'"'"'s network interface in which to store PIO and DMA store commands whose destination address corresponds the range of local physical addressed mapped thereby;

the IMMU storing a plurality of IMMU entries, a first subset of the IMMU entries each mapping a specified range of global addresses to a corresponding range of local physical address, and a second subset of the IMMU entries each mapping a specified range of local virtual addresses to a corresponding range of local physical address;

the first node'"'"'s network interface includes;

OMMU lookup logic for locating, for each received PIO store and DMA store command, an OMMU entry in the OMMU corresponding to the respective destination address specified therein, and converting the respective destination address in each received PIO store and DMA store command into a corresponding global address in accordance with the located OMMU entry;

IMMU lookup logic for locating, for each received DMA store command, an IMMU entry in the IMMU corresponding to the respective source address specified therein, and converting the respective specified source address into a corresponding local physical address in accordance with the located IMMU entry;

the packetizing engine for each queue stores in each generated data transfer packet a global destination address corresponding to a global address obtained by the OMMU lookup logic; and

the packetizing engine for each queue, when packetizing each DMA store command, retrieves data from the local physical address obtained by the IMMU lookup logic for the source address specified by the DMA store command.