Torroidal backbone connections for network deployment

• US 8,489,718 B1
• Filed: 05/19/2010
• Issued: 07/16/2013
• Est. Priority Date: 05/19/2010
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Brar, Jagwinder Singh, Marr, Michael David, Lamoreaux, Tyson J., Kelly, Mark N.
• Primary Examiner(s): Nguyen, Dustin
• Application Number:
  US12/783,257
• Time in Patent Office: 1,154 Days
• Field of Search: 370/256, 370/351, 370/389, 709/238, 709/219, 709/239, 712/29
• US Class Current: 709/222
• CPC Class Codes: H04L 45/586 of virtual routers
• CPC Class Codes: H04L 49/15 Interconnection of switching modules
• CPC Class Codes: H04L 49/356 for storage area networks
• CPC Class Codes: H04L 49/40 Constructional details, e.g. power supply, mechanical construction or backplane
• CPC Class Codes: H04L 49/45 Arrangements for providing or supporting expansion

Abstract

The efficiency of scaling of a network of computing devices can be improved by grouping highly-connected portions into deployment units wherein devices only have to be aware of other devices in the respective deployment unit. The various deployment units can be connected by a backbone mechanism. In some embodiments, the backbone utilizes a torroidal connection scheme to connect the backbone switches and pass routing protocol information. Host traffic can still utilize connections between tiers, such that standard routing and networking protocols can be utilized. In other embodiments, logical interfaces can be initialized in each backbone switch, such that each switch itself can function as a point backbone. The devices of the various deployment units then can communicate with each other via the point backbones using standard networking and routing protocols.

Claims

1. A computer-implemented method of connecting computing devices, comprising:

under control of one or more computer systems configured with executable instructions,

providing a plurality of deployment units each comprising a first tier of edge switches each connected to each of a second tier of spine switches, each edge switch being further connected to a respective set of computing devices, the edge switches and spine switches being connected using only connections between tiers, wherein each switch in a deployment unit maintains information about other switches located only within the same deployment unit, and wherein only a portion of the edge switches in the deployment unit are configured as egress switches having external connections to one or more devices outside of the deployment unit;

connecting the plurality of deployment units to an external network using a backbone tier comprising a plurality of backbone switches, connections for network traffic between the computing devices and the external network being only between-tier connections; and

connecting the plurality of backbone switches using a plurality of connections within the backbone tier, each connection between backbone switches used to transmit routing information for conventional routing protocols, the plurality of backbone switches being connected using a torroidal connection design wherein each of the backbone switches is connected to at most four of the plurality of backbone switches,

wherein the backbone switches are enabled to pass routing information upon unavailability of up to three of the backbone switches in the backbone tier.

2. The computer-implemented method of claim 1, wherein the backbone switches are connected using lower bandwidth ports than are used to route network traffic between the computing devices and the external network.

3. The computer-implemented method of claim 1, wherein each deployment unit comprises a Clos network, fat tree network, high radix network, butterfly network, or dragonfly network.

4. A computer-implemented method of connecting computing devices, comprising:

under control of one or more computer systems configured with executable instructions,

grouping switches into a plurality of network areas each comprising a first tier of edge switches each connected to each of a second tier of spine switches, each edge switch comprising ports for connection to a respective set of computing devices, wherein each switch in a network area maintains information about other switches located only within the same network area;

connecting the plurality of network areas to an external network using a backbone tier comprising a plurality of backbone switches; and

connecting the plurality of backbone switches using a plurality of intra-tier connections within the backbone tier, each of the intra-tier connections between backbone switches used to transmit routing information for the plurality of network areas, the plurality of backbone switches being connected using a torroidal connection design wherein each of the backbone switches is connected to only a portion of the plurality of backbone switches.

5. The computer-implemented method of claim 4, wherein each of the plurality of edge switches and the plurality of spine switches is a commodity network switch or application-specific integrated circuit (ASIC).

6. The computer-implemented method of claim 4, wherein the edge switches and spine switches of each network area are connected using only connections between tiers.

7. The computer-implemented method of claim 4, wherein connections for network traffic between the computing devices and the external network are only between-tier connections.

8. The computer-implemented method of claim 4, wherein the backbone switches are connected using a torroidal connection design.

9. The computer-implemented method of claim 4, wherein each edge switch connected to a set of host devices is a top of rack (TOR) switch connected to a plurality of host devices in a network rack.

10. The computer-implemented method of claim 4, wherein a number of spine and edge switches for a first network area is different than a number of spine and edge switches for a second network area.

11. The computer-implemented method of claim 4, wherein the backbone tier is connected to the external network using a high radix interconnection network.

12. The computer-implemented method of claim 4, wherein at least one of the network areas is a Clos network, fat tree network, high radix network, butterfly network, or dragonfly network.

13. A data center, comprising:

a plurality of computing devices each operable to receive a request and perform a computing operation;

a plurality of deployment units each comprising a first tier of edge switches each connected to each of a second tier of spine switches, each edge switch being further connected to a respective set of the plurality of computing devices, the edge switches and spine switches being connected using only connections between tiers, wherein each switch in a deployment unit maintains information about other switches located only within the same deployment unit; and

a backbone tier comprising a plurality of backbone switches, the backbone switches connecting the plurality of deployment units to an external network such that network traffic passes between the computing devices and the external network using only between-tier connections, the plurality of backbone switches being connected to each other using a plurality of intra-tier connections within the backbone tier, each of the intra-tier connections being operable to transmit routing information for the plurality of deployment units between the backbone switches, the plurality of backbone switches being connected to each other using a torroidal connection design wherein each of the backbone switches is connected to a portion of the plurality of backbone switches.

14. The data center of claim 13, wherein each of the backbone switches has a first plurality of higher bandwidth ports and a second plurality of lower bandwidth ports, the network traffic being routed using the higher bandwidth ports and the intra-tier connections being made using the lower bandwidth ports.

15. The data center of claim 13, wherein at least one of the deployment units is a Clos network, fat tree network, high radix network, butterfly network, or dragonfly network.

16. The data center of claim 13, further comprising a highly-connected aggregation fabric connecting the backbone tier to the external network.

17. A system for determining connections for computing devices, comprising:

a processor; and

a memory device including instructions that, when executed by the processor, cause the processor to;

group switches into a plurality of logical areas each comprising a first tier of edge switches each connected to each of a second tier of spine switches, each edge switch comprising ports for connection to a respective set of computing devices wherein each switch in a logical area maintains information about other switches located only within the same logical area;

determine first connections to connect the plurality of logical areas to an external network using a backbone tier comprising a plurality of backbone switches; and

determine second connections to connect the plurality of backbone switches using a plurality of intra-tier connections within the backbone tier, each of the intra-tier connections between backbone switches being used to transmit routing information for the plurality of logical areas, the plurality of backbone switches being connected using a torroidal connection design wherein each of the backbone switches is connected to only a portion of the plurality of backbone switches.

18. The system of claim 17, wherein each of the plurality of edge switches and the plurality of spine switches is a commodity network switch or application-specific integrated circuit (ASIC).

19. The system of claim 17, wherein the backbone switches are connected using a torroidal connection design.

20. The system of claim 17, wherein at least one of the logical areas is a Clos network, fat tree network, high radix network, butterfly network, or dragonfly network.

21. A computer-readable storage medium storing instructions for determining connections for computing devices, the instructions when executed by a processor causing the processor to:

group switches into a plurality of logical areas each comprising a first tier of edge switches each connected to each of a second tier of spine switches, each edge switch comprising ports for connection to a respective set of computing devices, wherein each switch in a logical area maintains information about other switches located only within the same logical area;

determine first connections to connect the plurality of logical areas to an external network using a backbone tier comprising a plurality of backbone switches; and

determine second connections to connect the plurality of backbone switches using a plurality of intra-tier connections within the backbone tier, each of the intra-tier connections between backbone switches being used to transmit routing information for the plurality of logical areas, the plurality of backbone switches being connected using a torroidal connection design wherein each of the backbone switches is connected to only a portion of the plurality of backbone switches.

22. The computer-readable storage medium of claim 21, wherein each of the plurality of edge switches and the plurality of spine switches is a commodity network switch or application-specific integrated circuit (ASIC).

23. The computer-readable storage medium of claim 21, wherein the backbone switches are connected using a torroidal connection design.

24. The computer-readable storage medium of claim 21, wherein at least one of the logical areas is a Clos network, fat tree network, high radix network, butterfly network, or dragonfly network.