Scalable cluster router

• US 8,904,028 B2
• Filed: 07/19/2010
• Issued: 12/02/2014
• Est. Priority Date: 07/17/2009
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Iannaccone, Gianluca, Ratnasamy, Sylvia, Manesh, Maziar, Argyraki, Katerina, Chun, Byung-Gon, Fall, Kevin, Knies, Allan, Egi, Norbert, Dobrescu, Mihai, Baset, Salman
• Primary Examiner(s): ZHANG, SHIRLEY X
• Application Number:
  US12/838,785
• Publication Number:
  US 20110016223A1
• Time in Patent Office: 1,597 Days
• Field of Search: 709230-235, 370229-238, 370/351, 370/389, 370/400, 370412-418
• US Class Current: 709/232
• CPC Class Codes: H04L 45/00 Routing or path finding of packets in data switching networks routing or path finding in wireless networks H04W40/00
• CPC Class Codes: H04L 45/125 based on throughput or bandwidth
• CPC Class Codes: H04L 45/583 Stackable routers
• CPC Class Codes: H04L 45/60 Router architectures
• CPC Class Codes: H04L 47/125 by balancing the load, e.g. traffic engineering
• CPC Class Codes: H04L 49/45 Arrangements for providing or supporting expansion

Abstract

Generally, this disclosure describes a scalable cluster router that includes a plurality of server-class computers interconnected together to form a router. Each server may be configured to independently schedule switching of packets to reduce the switch speed requirements on a per server basis. Each server may include a scheduler that independently load balances packet flows across servers of the cluster. Router capacity may be incrementally scaled by adding more servers, and router capacity may be increased by load balancing techniques within individual servers.

Claims

1. A server, comprising:

a set of internal ports;

at least one external port; and

a scheduler;

wherein;

said set comprises a plurality of internal ports;

each of said plurality of internal ports is configured to be coupled to and to communicate with at least one other server and included in a cluster of N interconnected servers to form a router having M router ports corresponding to the total number of external ports of all the servers in the cluster;

said at least one external port is configured to couple to and to communicate with a network or a network device to receive one of a plurality of packet flows;

said scheduler is configured to switch packets from said at least one external port to at least one of the plurality of internal ports of said set, or from one of the plurality of internal ports of said set to another one of the plurality internal ports of said set, by load balancing a packet flow comprising a plurality of packets;

said load balancing comprises randomly selecting one of said plurality of internal ports from said set to transmit each packet of said plurality of packets to a receiving interconnected server coupled to the selected internal port; and

the scheduler is further configured to switch at a processing rate that is independent of M.

2. The server of claim 1, wherein a line rate of each of said external ports is R bits per second (bps), and the scheduler is configured to switch at processing rate of c*s*R, where s represents the number of external ports of said server and c is between 2 and 3.

3. The server of claim 1, further comprising a network adapter configured to process a header of each packet of said plurality of packets to identify an output node for each of the packets and to modify a MAC address of each of the packets with queue information to enable the receiving server to identify the output node without further processing of the packet header.

4. The server of claim 1, further comprising:

a processor having a plurality of cores; and

a plurality of receive queues for receiving data from said external ports;

wherein each said receive queue is assigned to a specific packet flow of said plurality of packet flows, and wherein packet processing of said specific packet flow received by said at least one external port is distributed across said cores.

5. The server of claim 1, wherein said selecting one of said plurality of said internal ports from said set comprises:

determining a source of said one of said plurality of packet flows for said at least one external port;

determining if a packet within the identified packet flow has been received within a predetermined time interval.

6. The server of claim 1, wherein said selecting one of said plurality of said internal ports from said set comprises:

determining if an internal port is overloaded or has exceeded a predetermined threshold.

7. The server of claim 1, wherein said selecting one of said plurality of said internal ports from said set further comprises:

determining if a selected internal port has a throughput that is less than a soft limit;

determining if a selected internal port is a preferred port; and

determining if a selected internal port has a throughput less than a hard limit.

8. The server of claim 1, wherein said server is configured to interconnect with said cluster of N interconnected servers in a full mesh topology.

9. The server of claim 1, wherein said server is further configured to interconnect with a plurality of intermediate servers in a k-ary n-fly butterfly multihop topology.

10. A method, comprising:

receiving a packet flow at an external port of an input node server, wherein said input node server is one of a plurality of N node servers coupled in an interconnect topology in a cluster router, the cluster router comprising a plurality of M router ports corresponding to the total number of external ports of all the servers in the cluster,

processing said packet flow to determine a destination of each of the packets of the packet flow, wherein said destination comprises an output node server of said plurality of node servers;

switching packets of said packet flow from an external port to an internal port of a set of internal ports, or from one internal port of said set to another internal port of said set, by load balancing a packet flow;

wherein;

said load balancing comprises randomly selecting, for each packet of said packet flow, one internal port of said set of internal ports to transmit each packet of said packet flow to a receiving interconnected server coupled to the selected internal port; and

said switching occurs at rate that is independent of M.

11. The method of claim 10, wherein a line rate of each of said external ports is R bits per second (bps), and the switching is configured to occur at processing rate of c*s*R, where s represents the number of external ports of said server and c is between 2 and 3.

12. The method of claim 10, further comprising:

processing a packet header of each packet of said packet flow to identify an output node for each of the packets; and

modifying a MAC address of each packet of said packet flow with queue information to enable the receiving server to identify an the output node without further processing of the packet header.

13. The method of claim 10, wherein:

said input node server further comprises a processor having a plurality of cores, and a plurality of receive queues for receiving data from said external port;

said method further comprises;

assigning a packet flow to a receive queue; and

distributing packet processing of said packet flow across said cores.

14. The method of claim 10, wherein said selecting one of said plurality of said internal ports of said set comprises:

determining a source of an identified packet flow for a given one of said external ports;

determining if a packet within the identified packet flow has been received within a predetermined time interval.

15. The method of claim 10, wherein said selecting one of said plurality of said internal ports of said set comprises:

determining if the internal port is overloaded or has exceeded a predetermined threshold.

16. The method of claim 10, wherein said selecting one of said plurality of said internal ports of said set comprises:

determining if a selected internal port has a throughput that is less than a soft limit;

determining if a selected internal port is a preferred port; and

determining if a selected internal port has a throughput less than a hard limit.

17. The method of claim 10, wherein said server is configured to interconnect with said cluster of N interconnected servers in a full mesh topology.

18. The method of claim 10, wherein said server is further configured to interconnect with a plurality of intermediate servers in a k-ary n-fly butterfly multihop topology.

19. An apparatus comprising a computer-readable storage medium having stored thereon computer readable instructions that when executed by one or more processors result in the following operations, comprising:

processing a packet flow received at an external port of an input node server to determine a destination of the packets of the packet flow, wherein said destination comprises an output node server;

switching packets from said external port to one internal port of a set of internal ports of the input node server, or from one internal port of said set to another internal port of said set, by load balancing a packet flow;

wherein;

said load balancing comprises randomly selecting, for each packet of a plurality of packets within the packet flow, one of said set of internal ports to transmit each packet of said plurality of packets to another server coupled to the selected internal port;

said input node server is one of a plurality of N node servers coupled in an interconnect topology in a cluster router, the cluster router comprising a plurality of M router ports corresponding to the total number of external ports of all the servers in the cluster; and

said switching occurs at rate that is independent of M.

20. The apparatus of claim 19, wherein a line rate of each of said external ports is R bits per second (bps), and the switching is configured to occur at processing rate of c*s*R, where s represents the number of external ports of said server and c is between 2 and 3.

21. The apparatus of claim 19, wherein the instructions that when executed by one or more of the processors result in the following additional operations comprising:

processing a packet header of each packet of said plurality of packets to identify an output node for each of the packets; and

modifying a MAC address of each of the packets with queue information to enable the receiving server to identify the output node without further processing of the packet header.

22. The apparatus of claim 19, wherein said input node server further comprises:

a processor having a plurality of cores; and

a plurality of receive queues for receiving data from said external ports;

wherein each said receive queue is assigned to a specific packet flow, and wherein packet processing of said specific packet flow is distributed across said cores.

23. The apparatus of claim 19, wherein, for the selecting operation, the instructions that when executed by one or more of the processors result in the following additional operations comprising:

determining a source of an identified packet flow for a given external port;

determining if a packet within the identified packet flow has been received within a predetermined time interval.

24. The apparatus of claim 19, wherein, for the selecting operation, the instructions that when executed by one or more of the processors result in the following additional operation comprising:

determining if the internal port is overloaded or has exceeded a predetermined threshold.

25. The apparatus of claim 19, wherein, for the selecting operation, the instructions that when executed by one or more of the processors result in the following additional operations comprising:

determining if a selected internal port has a throughput that is less than a soft limit;

determining if a selected internal port is a preferred port; and

determining if a selected internal port has a throughput less than a hard limit.

26. The apparatus of claim 19, wherein said server is configured to interconnect with said cluster of N interconnected servers in a full mesh topology.

27. The apparatus of claim 19, wherein said server is further configured to interconnect with a plurality of intermediate servers in a k-ary n-fly butterfly multihop topology.