Combining separate infiniband subnets into virtual subnets

• US 20040024903A1
• Filed: 07/30/2002
• Published: 02/05/2004
• Est. Priority Date: 07/30/2002
• Status: Active Grant

• Current Assignee: Avago Technologies International Sales Pte Limited (Broadcom, Inc.)
• Original Assignee: Brocade Communications Systems, Inc. (Broadcom, Inc.)
• Inventors: Costatino, Lino, Green, Howard, Forbes, Brian
• Granted Patent: US 7,401,157 B2
• Time in Patent Office: Days
• US Class Current: 709/238
• CPC Class Codes: H04L 45/00 Routing or path finding of packets in data switching networks routing or path finding in wireless networks H04W40/00

Abstract

A system and method are disclosed for combining physically separate subnets of an Infiniband (IB) network into virtual subnets. In the virtual subnets, all of the end node in one subnet may be allowed to access all of the end nodes in another subnet, or the access may be limited to only a selected subset of another subnet. In addition to processing of globally-routed IB packets, the router also communicates (alone or in cooperation with another router) locally-routed IB packets traveling between a designated end nodes in different subnets. The routers may encapsulate the packets into raw datagrams for transport across intervening subnets. The described system and method may advantageously allow remote end nodes to act as a part of a given subnet, and may advantageously allow routers to provide both switch and routing functionality.

Claims

1. A network that comprises:

a first set of switch-connected end nodes;

a second set of switch-connected end nodes; and

a router configured to process globally-routed Infiniband packets traveling between the first and second sets to provide local route headers suitable for a receiving set of switch-connected end nodes,

wherein the router is further configured to communicate locally-routed Infiniband packets traveling between a first end node in the first set and a second end node in the second set.

2. The network of claim 1, wherein the router is configured to emulate one or more switches that connect the second end node to the first set of switch-connected end nodes.

3. The network of claim 1, wherein the router is configured to emulate one or more switches that connect a selected subset of end nodes from the second set of switch-connected end nodes to the first set of switch-connected end nodes.

4. The network of claim 1, wherein the router is configured to emulate a switch for each router port, and wherein the router is further configured to emulate a switch that connects the emulated router port switches corresponding to the ports that connect to the first and second sets of switch-connected end nodes.

5. The network of claim 1, wherein the router includes an internal subnet configured to transport Infiniband packets between the router ports based on contents of the local route headers (LRH) in the packets.

6. The network of claim 5, wherein a first router port encapsulates locally-routed Infiniband packets directed from the first set of switch connected end nodes to the second end node, wherein the encapsulated packets are raw datagram packets with local route headers, and wherein the internal subnet transports the raw datagram packets to a second router port that de-encapsulates the locally-routed Infiniband packets and provides the de-encapsulated locally-routed Infiniband packets to the second set of switch-connected end nodes.

7. The network of claim 1, wherein the router is configured to couple physically separate subnets into a single logical subnet that is configured by a single master subnet manager.

8. The network of claim 1, further comprising:

a second router attached to the second set of switch-connected end nodes,

wherein the first router is attached to the first set of switch-connected end nodes, and

wherein the second router is configured to cooperate with the first router to communicated locally-routed Infiniband packets traveling between the first and second end-nodes.

9. The network of claim 8, wherein the routers are configured to emulate a switch for each router port, and wherein the routers are further configured to emulate a switch that connects the emulated router port switches corresponding to the router ports that connect to the first and second sets of switch-connected end nodes.

10. The network of claim 8, wherein the routers are configured to emulate one or more switches that connect a selected subset of end nodes from the second set of switch-connected end nodes to the first set of switch-connected end nodes.

11. The network of claim 8, wherein the first router encapsulates locally-routed Infiniband packets directed from the first set of switch connected end nodes to the second end node, wherein the encapsulated packets are raw datagram packets with global route headers, and wherein the first router directs the raw datagram packets to the second router, which de-encapsulates the locally-routed Infiniband packets and provides the de-encapsulated locally-routed Infiniband packets to the second set of switch-connected end nodes.

12. The network of claim 8, wherein the routers are configured to couple physically separate subnets into a single logical subnet.

13. A method of connecting a first Infiniband subnet to a second Infiniband subnet, wherein the method comprises configuring a router for:

receiving globally-routed packets from the first Infiniband subnet;

processing the globally-routed packets to provide local route headers suitable for the second Infiniband subnet;

receiving locally-routed packets from the first Infiniband subnet that are directed to an end node in the second Infiniband subnet; and

delivering the locally-routed packets to the second Infiniband subnet.

14. The method of claim 13, further comprising configuring the router for:

emulating one or more switches that connect the end node in the second Infiniband subnet to the first Infiniband subnet.

15. The method of claim 13, further comprising configuring the router for:

emulating one or more switches that connect a selected subset of end nodes in the second Infiniband subnet to the first Infiniband subnet.

16. The method of claim 13, further comprising configuring the router for:

encapsulating locally-routed packets into raw datagrams for transport over any Infiniband subnets between the first and second Infiniband subnets; and

de-encapsulating the locally-routed packets before providing the locally-routed packets to the second Infiniband subnet.

17. The method of claim 13, further comprising configuring the router for:

combining the first and second Infiniband subnets into a single logical subnet.

18. The method of claim 13, further comprising configuring the router for:

receiving globally-routed packets from the second Infiniband subnet;

processing the globally-routed packets to provide local route headers suitable for the first Infiniband subnet;

receiving locally-routed packets from the second Infiniband subnet that are directed to an end node in the first Infiniband subnet; and

delivering the locally-routed packets to the first Infiniband subnet.

19. A method of connecting a first Infiniband subnet to a second Infiniband subnet, wherein the method comprises:

receiving locally-routed packets from the first Infiniband subnet that are directed to an end node in the second Infiniband subnet;

encapsulating the locally-routed packets in raw datagrams;

communicating the raw datagrams across a third Infiniband subnet;

de-encapsulating the locally-routed packets; and

providing the locally-routed packets to the second Infiniband subnet.

20. The method of claim 19, wherein the encapsulating includes:

prepending a new local route header, a raw header, and an extended raw header to the locally-routed packet; and

recalculating the variable cyclic redundancy code value with the prepended headers.

21. The method of claim 19, wherein the encapsulating includes:

prepending a new local route header, a global route header, and a global raw header to the locally-routed packet; and

recalculating the variable cyclic redundancy code value with the prepended headers.

22. A router that comprises:

multiple port interface circuits coupled to corresponding Infiniband subnets; and

an internal Infiniband subnet that couples the multiple port interface circuits together,

wherein a first port interface circuit is configured to receive locally-routed packets from a first Infiniband subnet that are directed to an end node in a second Infiniband subnet,

wherein the first port interface circuit is configured to encapsulate the locally-routed packets in raw datagrams that the internal Infiniband subnet transports to a second port interface circuit, and

wherein the second port interface circuit is configured to de-encapsulate the locally-routed packets and provide them to the second Infiniband subnet.

23. The method of claim 22, wherein the first port interface circuit is configured to encapsulate by:

prepending a new local route header, a raw header, and an extended raw header to the locally-routed packet; and

recalculating the variable cyclic redundancy code value with the prepended headers.

24. The method of claim 22, wherein the first port interface circuit is configured to encapsulate by:

prepending a new local route header, a global route header, and a global raw header to the locally-routed packet; and

recalculating the variable cyclic redundancy code value with the prepended headers.