Adaptive routing mechanism for torus interconnection network

• US 5,701,416 A
• Filed: 04/13/1995
• Issued: 12/23/1997
• Est. Priority Date: 04/13/1995
• Status: Expired due to Term

• Current Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Original Assignee: CRAY Incorporated (Hewlett-Packard Enterprise Company)
• Inventors: Scott, Steven L., Thorson, Gregory M.
• Primary Examiner(s): Harvey, Jack B.
• Assistant Examiner(s): PANCHOLI, JIGAR K
• Application Number:
  US08/421,566
• Time in Patent Office: 985 Days
• Field of Search: 395/200.15, 395/200.01, 340/825.02, 340/825.03, 340/826, 340/827
• US Class Current: 712/11
• CPC Class Codes: G06F 15/17331 Distributed shared memory [DSM], e.g. remote direct memory access [RDMA]
• CPC Class Codes: G06F 15/17381 Two dimensional, e.g. mesh, torus

Abstract

A routing mechanism includes two acyclic non-adaptive virtual channels having two types of virtual channel buffers to store packets along deterministic virtual paths between nodes in an n-dimensional networked system, and an adaptive virtual channel having a third type of virtual channel buffer to store the packets along non-deterministic virtual paths between the nodes. The packets are routed between the nodes along either selected portions of the deterministic virtual paths or selected portions of the non-deterministic virtual paths based on routing information such that a packet is never routed on a selected portion of one of the non-deterministic virtual paths unless the third type virtual channel buffer associated with the selected portion of the one non-deterministic virtual path has sufficient space available to store the entire packet.

Claims

1. A method of routing a packet between a source node and a destination node in a networked system comprising a plurality of nodes and physical communication links interconnecting the nodes in an n-dimensional topology, the method comprising:

generating a header including routing information;

attaching the header to information to be transferred in order to form a packet;

defining two acyclic non-adaptive virtual channels utilizing associated virtual channel buffers assigned to the physical communication links to store the packet along a deterministic virtual path from the source node to the destination node based on the routing information;

defining an adaptive virtual channel utilizing associated virtual channel buffers assigned to the physical communication links to store the packet along a plurality of non-deterministic virtual paths from the source node to the destination node based on the routing information;

selecting either a portion of the deterministic virtual path from the source node to an adjacent node along the deterministic virtual path or a portion of one of the non-deterministic virtual paths from the source node to an adjacent node along the one non-deterministic virtual path based on the routing information, wherein the portion of the one non-deterministic virtual path is not selected unless the virtual channel buffer associated with the portion of the one non-deterministic virtual path has sufficient space available to store the entire packet;

routing the packet from the source node to the adjacent node along the portion of the selected virtual path; and

continuing to select virtual paths at each node from the source to the destination node and to route the packet on the selected virtual paths until the packet reaches the destination node.

2. The method of claim 1 further comprising the step of defining a direction order for routing packets, wherein the direction order defines a priority for routing the packets in each of 2n directions along the deterministic virtual path.

3. The method of claim 2 wherein the one non-deterministic virtual path is pre-selected for routing in the lowest priority direction of the 2n directions that is not blocked or disabled.

4. The method of claim 1 wherein the step of selecting a path for routing comprises the steps of:

pre-selecting the one non-deterministic virtual path;

requesting both the portion of the pre-selected non-deterministic virtual path and the portion of the deterministic virtual path for routing of the packet;

granting or denying access to the physical communication links in response to said requests and other competing requests for the physical communication links, wherein if an adaptive virtual channel and a non-adaptive virtual channel are both requesting access of a physical communication link the non-adaptive virtual channel is granted access to the physical communication link;

selecting the portion of a granted virtual path if at least one of said requests is granted, wherein the portion of the one non-deterministic virtual path is selected if both of said requests are granted; and

retrying the requesting step if neither of said requests are granted.

5. The method of claim 1 wherein the method routes multiple packets between the plurality of nodes, and wherein the method further comprising the steps of:

determining order dependent groups of packets which are required to arrive at the destination node in a pre-determined order; and

selecting only the deterministic virtual paths for routing packets which are in an order dependent group of packets.

6. The method of claim 1 wherein the information to be transferred includes request and response information, and wherein the step of defining acyclic non-adaptive virtual channels includes defining two acyclic non-adaptive virtual channels for handling the request information and two acyclic non-adaptive virtual channels for handling the response information.

7. The method of claim 1 further comprising sending the packet in a free hop from the source node to an adjacent node prior to the selecting step.

8. The method of claim 7 wherein the n-dimensional topology comprises physical communication lines connecting nodes in 2n directions including first and second directions in each dimension, and wherein the step of sending the packet in a free hop from the source node to an adjacent node comprises sending the packet in any one of the first directions.

9. The method of claim 1 wherein the step of transferring the packet further comprises sending the packet on a final hop from a node adjacent to the destination node to complete the route to the destination node.

10. The method of claim 9 wherein the final hop is taken based on the routing information.

11. The method of claim 9 wherein the final hop is limited to a single direction in one of the n dimensions.

12. A method of routing a packet between a source node and a destination node in a networked system comprising a plurality of nodes and physical communication links interconnecting the nodes in an n-dimensional topology, the method comprising:

generating a header including routing information;

attaching the header to information to be transferred in order to form a packet;

defining two acyclic non-adaptive virtual channels utilizing associated virtual channel buffers assigned to the physical communication links to store the packet along a deterministic virtual path from the source node to the destination node based on the routing information;

defining an adaptive virtual channel utilizing the third type of virtual channel buffers to store the packet along a plurality of non-deterministic virtual paths from the source node to the destination node based on the routing information;

selecting either a portion of the deterministic virtual path from the source node to an adjacent node along the deterministic virtual path or a portion of one of the non-deterministic virtual paths from the source node to an adjacent node along the one non-deterministic virtual path based on the routing information, wherein the portion of the one non-deterministic virtual path is not selected unless the virtual channel buffer associated with the portion of the one non-deterministic virtual path is empty;

routing the packet from the source node to the adjacent node along the portion of the selected virtual path;

if a packet is routed to the portion of the one non-deterministic virtual path, recording a direction and a deterministic virtual path the packet is on as the packet enters the portion of the one non-deterministic virtual path, and when the packet reenters the deterministic virtual path continue routing the packet in the recorded direction on the recorded deterministic virtual path if all transfers in the recorded direction are not completed; and

continuing to select virtual paths at each node from the source to the destination node and to route the packet on the selected virtual paths until the packet reaches the destination node.

13. The method of claim 12 further comprising the step of defining a direction order for routing packets, wherein the direction order defines a priority for routing the packets in each of 2n directions along the deterministic virtual path.

14. The method of claim 12 wherein the one non-deterministic virtual path is pre-selected for routing in the lowest priority direction of the 2n directions that is not blocked or disabled.

15. The method of claim 12 wherein the step of selecting a path for routing comprises the steps of:

pre-selecting the one non-deterministic virtual path;

requesting both the portion of the pre-selected non-deterministic virtual path and the portion of the deterministic virtual path for routing of the packet;

granting or denying access to the physical communication links in response to said requests and other competing requests for the physical communication links, wherein if an adaptive virtual channel and a non-adaptive virtual channel are both requesting access of a physical communication link the non-adaptive virtual channel is granted access to the physical communication link;

selecting the portion of a granted virtual path if at least one of said requests is granted, wherein the portion of the one non-deterministic virtual path is selected if both of said requests are granted; and

retrying the requesting step if neither of said requests are granted.

16. The method of claim 12 wherein the method routes multiple packets between the plurality of nodes, and wherein the method further comprising the steps of:

determining order dependent groups of packets which are required to arrive at the destination node in a pre-determined order; and

selecting only the deterministic virtual paths for routing packets which are in an order dependent group of packets.

17. The method of claim 12 wherein the information to be transferred includes request and response information, and wherein the step of defining acyclic non-adaptive virtual channels includes defining two acyclic non-adaptive virtual channels for handling the request information and two acyclic non-adaptive virtual channels for handling the response information.

18. The method of claim 12 further comprising sending the packet in a free hop from the source node to an adjacent node prior to the selecting step.

19. The method of claim 12 wherein the step of transferring the packet further comprises sending the packet on a final hop from a node adjacent to the destination node to complete the route to the destination node.

20. A multiprocessor computer system comprising:

a plurality of processing element nodes;

physical communication links interconnecting the processing element nodes in an n-dimensional topology;

means for generating headers having routing information;

means for attaching the headers to information to be transferred between the processing element nodes in order to form packets;

two acyclic non-adaptive virtual channels having virtual channel buffers assigned to each physical communication link to store the packets along deterministic virtual paths between the plurality of processing element nodes based on the routing information;

an adaptive virtual channel having virtual channel buffers assigned to each physical communication link to store the packets along non-deterministic virtual paths between the plurality of processing element nodes based on the routing information;

means for selecting either a portion of a deterministic virtual path between two processing element nodes along said deterministic virtual path or a portion of a non-deterministic virtual path between two processing element nodes along said non-deterministic virtual path based on the routing information, wherein the portion of said non-deterministic virtual path is not selected unless the virtual channel buffer associated with the portion of said non-deterministic virtual path has sufficient space available to store an entire packet; and

means for routing the packets between the processing element nodes along the selected portions of said virtual paths.

21. A multiprocessor computer system comprising:

a plurality of processing element nodes;

physical communication links interconnecting the processing element nodes in an n-dimensional topology;

means for generating headers having routing information;

means for attaching the headers to information to be transferred between the processing element nodes in order to form packets;

two acyclic non-adaptive virtual channels having virtual channel buffers assigned to each physical communication link to store the packets along deterministic virtual paths between the plurality of processing element nodes based on the routing information;

an adaptive virtual channel having virtual channel buffers assigned to each physical communication link to store the packets along non-deterministic virtual paths between the plurality of processing element nodes based on the routing information;

means for selecting either a portion of a deterministic virtual path between two processing element nodes along said deterministic virtual path or a portion of a non-deterministic virtual path between two processing element nodes along said non-deterministic virtual path based on the routing information, wherein the portion of said non-deterministic virtual path is not selected unless the virtual channel buffer associated with the portion of said non-deterministic virtual path is empty;

means for routing the packets between the processing element nodes along the selected portions of said virtual paths including means for recording a direction and a deterministic virtual path which a packet is on when the packet enters the portion of said non-deterministic virtual path, and means for routing the packet in the recorded direction on the recorded deterministic virtual path if all transfers in the recorded direction are not completed when the packet reenters the deterministic virtual path from the portion of said non-deterministic virtual path.

22. A routing mechanism for routing packets coming information to be transferred between nodes in an n-dimensional networked system, the routing mechanism comprising:

two acyclic non-adaptive virtual channels having virtual channel buffers to store the packets along deterministic virtual paths between the nodes;

an adaptive virtual channel having virtual channel buffers to store the packets along non-deterministic virtual paths between the nodes;

means for routing the packets between the nodes along either selected portions of the deterministic virtual paths or selected portions of the non-deterministic virtual paths, wherein a packet is never routed on a selected portion of one of the non-deterministic virtual paths unless the virtual channel buffer associated with the selected portion of the one non-deterministic virtual path has sufficient space available to store the entire packet.