Dis-aggregated and distributed data-center architecture using a direct interconnect fabric

• US 8,140,719 B2
• Filed: 05/05/2009
• Issued: 03/20/2012
• Est. Priority Date: 06/21/2007
• Status: Active Grant

• Current Assignee: Seamicro Incorporated (Advanced Micro Devices, Inc.)
• Original Assignee: Seamicro Incorporated (Advanced Micro Devices, Inc.)
• Inventors: Lauterbach, Gary, Rao, Anil R.
• Primary Examiner(s): Sorrell, Eron J
• Assistant Examiner(s): LEWIS-TAYLOR, DAYTON A.
• Application Number:
  US12/435,618
• Publication Number:
  US 20090216920A1
• Time in Patent Office: 1,050 Days
• Field of Search: None
• US Class Current: 710/36
• CPC Class Codes: H04L 49/351 for local area network [LAN], e.g. Ethernet switches
• CPC Class Codes: H04L 49/356 for storage area networks
• CPC Class Codes: H04L 49/40 Constructional details, e.g. power supply, mechanical construction or backplane
• CPC Class Codes: H04L 49/557 Error correction, e.g. fault recovery or fault tolerance

Abstract

A data center has several dis-aggregated data clusters that connect to the Internet through a firewall and load-balancer. Each dis-aggregated data cluster has several dis-aggregated compute/switch/disk chassis that are connected together by a mesh of Ethernet links. Each dis-aggregated compute/switch/disk chassis has many processing nodes, disk nodes, and I/O nodes on node cards that are inserted into the chassis. These node cards are connected together by a direct interconnect fabric. Using the direct interconnect fabric, remote I/O and disk nodes appear to the operating system to be located on the local processor'"'"'s own peripheral bus. A virtual Ethernet controller and a virtual generic peripheral act as virtual endpoints for the local processor'"'"'s peripheral bus. I/O and disk node peripherals are virtualized by hardware without software drivers. Rack and aggregation Ethernet switches are eliminated using the direct interconnect fabric, which provides a flatter, dis-aggregated hierarchy.

Claims

1. A dis-aggregated compute/switch/storage chassis comprising:

a plurality of node cards;

a plurality of slots having sockets for receiving the plurality of node cards;

a direct interconnect fabric having wiring traces that connect to the plurality of node cards through the plurality of slots;

processing nodes on a first subset of the plurality of node cards, the processing nodes each having multiple processors for executing programs and for generating requests to access peripherals;

storage nodes on a second subset of the plurality of node cards, the storage nodes each for connecting to storage media;

Input-Output I/O nodes on a third subset of the plurality of node cards, the I/O nodes each having a network interface controller (NIC) for connecting to a local-area network (LAN) link;

wherein the first subset of the plurality of node cards is non-overlapping with the second subset and with the third subset of the plurality of node cards; and

a direct interconnect switch on each of the plurality of node cards, for connecting through the direct interconnect fabric to other direct interconnect switches on other node cards.

2. The dis-aggregated compute/switch/storage chassis of claim 1 further comprising:

a dis-aggregated data cluster which comprises a plurality of the dis-aggregated compute/switch/storage chassis that are connected together by LAN links from the I/O nodes;

wherein the third subset comprises at least three I/O nodes for connecting to at least three LAN links;

wherein each dis-aggregated compute/switch/storage chassis in the dis-aggregated data cluster connects to at least two other dis-aggregated compute/switch/storage chassis in the plurality of dis-aggregated compute/switch/storage chassis;

wherein the plurality of dis-aggregated compute/switch/storage chassis in the dis-aggregated data cluster are connected together by a mesh of the LAN links.

3. The dis-aggregated compute/switch/storage chassis of claim 2 further comprising:

a first data-center LAN link in the dis-aggregated data cluster, wherein the first data-center LAN link is one of the LAN links that does not connect to LAN links for other dis-aggregated compute/switch/storage chassis;

a plurality of the dis-aggregated data clusters;

a first core switch; and

a first gateway for connecting to an external network;

wherein the first core switch is coupled to the first data-center LAN link for each dis-aggregated data cluster in the plurality of the dis-aggregated data clusters;

wherein the first core switch is also coupled to the first gateway.

4. The dis-aggregated compute/switch/storage chassis of claim 3 wherein the first gateway further comprises:

a firewall for blocking unauthorized incoming traffic from the external network; and

a load-balancer for distributing incoming requests from the external network among the processing nodes in the plurality of dis-aggregated compute/switch/storage chassis.

5. The dis-aggregated compute/switch/storage chassis of claim 4 further comprising:

a load-balancer for distributing incoming requests from the external network among the processing nodes in the plurality of dis-aggregated compute/switch/storage chassis;

a first data-center LAN link in the modified dis-aggregated compute/switch/storage chassis, wherein the first data-center LAN link is one of the LAN links that does not connect to LAN links for other dis-aggregated compute/switch/storage chassis;

wherein the third subset comprises at least three I/O nodes for connecting to at least three LAN links;

wherein each dis-aggregated compute/switch/storage chassis connects to at least three other dis-aggregated compute/switch/storage chassis in the plurality of dis-aggregated compute/switch/storage chassis;

wherein the modified dis-aggregated compute/switch/storage chassis connects to at least two other dis-aggregated compute/switch/storage chassis in the plurality of dis-aggregated compute/switch/storage chassis and to the external network;

wherein the plurality of dis-aggregated compute/switch/storage chassis and the modified dis-aggregated compute/switch/storage chassis are connected together by a mesh of the LAN links.

6. The dis-aggregated compute/switch/storage chassis of claim 3 wherein the storage media comprises a plurality of disk drives or flash memory and wherein the LAN links are Ethernet links.

7. The dis-aggregated compute/switch/storage chassis of claim 3 wherein each processing node comprises at least four Central Processing Units (CPUs);

wherein the first subset comprises at least eight processing nodes;

wherein the second subset comprises at least three storage nodes;

wherein the third subset comprises at least three I/O nodes;

wherein the dis-aggregated data cluster comprises at least six dis-aggregated compute/switch/storage chassis.

8. The dis-aggregated compute/switch/storage chassis of claim 1 further comprising:

a plurality of the dis-aggregated compute/switch/storage chassis that are connected together by the LAN links from the I/O nodes;

a modified dis-aggregated compute/switch/storage chassis which comprises;

a dis-aggregated compute/switch/storage chassis;

a firewall for blocking unauthorized incoming traffic from an external network; and

whereby the remote peripherals are accessed remotely through the direct interconnect fabric and virtual endpoints appear to be on the local peripheral bus from the point of view of software executing on the processor.

9. The dis-aggregated compute/switch/storage chassis of claim 8 wherein each of the processing nodes further comprises:

a main memory for storing instructions executed and data used by a processor of the multiple processors;

a local peripheral bus that carries the requests to access peripherals;

an interconnect interface device, coupled to the processor, for acting as a virtual endpoint for the requests to access peripherals from the processor, wherein the virtual endpoints appear to be on the local peripheral bus from a point of view of software executing on the processor;

wherein the storage nodes and the I/O nodes comprise remote peripherals,

a second data-center LAN link in the dis-aggregated data cluster, wherein the second data-center LAN link is one of the LAN links that does not connect to LAN links for other dis-aggregated compute/switch/storage chassis;

a second core switch; and

a second gateway for connecting to an external network;

wherein the second core switch is coupled to the second data-center LAN link for each dis-aggregated data cluster in the plurality of the dis-aggregated data clusters;

wherein the second core switch is also coupled to the second gateway.

10. The dis-aggregated compute/switch/storage chassis of claim 9 wherein the direct interconnect fabric carries interconnect packets containing the requests to access peripherals generated by the multiple processors on the processing nodes;

wherein the I/O nodes and the storage nodes each comprise;

a remote peripheral bus that connects to a remote peripheral, the remote peripheral being a network interface controller (NIC) or a storage media controller;

a device manager, coupled to the direct interconnect fabric, for sending the requests to access peripherals contained in the interconnect packets over the remote peripheral bus to the remote peripheral, and for receiving reply data from the remote peripheral and sending the reply data in interconnect packets over the direct interconnect fabric to the interconnect interface device on the processing node to respond to the requests to access peripherals.

11. The dis-aggregated compute/switch/storage chassis of claim 10 further comprising:

a distributed routing table coupled to the interconnect interface device, the distributed routing table storing mapping entries, the mapping entries containing mappings of local addresses on the local peripheral bus to interconnect addresses that identify the device manager coupled to the remote peripheral buses attached to the remote peripheral that is virtually addressed by the request to access peripherals.

12. The dis-aggregated compute/switch/storage chassis of claim 11 further comprising:

a device distributed routing table coupled to the device manager, the device distributed routing table storing return mapping entries, the return mapping entries each identifying a requesting interconnect interface device that is coupled to the processor generating a request received by the device manager.

13. A dis-aggregated data cluster comprising:

a plurality of dis-aggregated chassis, each dis-aggregated chassis comprising;

a direct interconnect fabric having interconnect that connects to node cards;

processing nodes each having a plurality of processors for executing instructions;

storage nodes for controlling storage media that stores files;

Input/Output I/O nodes each having a network interface controller (NIC) for connecting to a network link;

wherein the I/O nodes comprise a first I/O node for connecting to a first local network link, a second I/O node for connecting to a second local network link, and a third I/O node for connecting to a third local network link;

wherein the processing nodes are formed on a first form factor of the node cards;

wherein the storage nodes are formed on a second form factor of the node cards;

wherein the I/O nodes are formed on a third form factor of the node cards;

a direct interconnect switch on the processing nodes and connected to the direct interconnect fabric, for virtualizing access of the storage nodes and the I/O nodes by the plurality of processors;

a first modified dis-aggregated chassis comprising;

a direct interconnect fabric having interconnect that connects node cards;

processing nodes each having a plurality of processors for executing instructions;

storage nodes for controlling storage media that stores files;

Input/Output I/O nodes each having a network interface controller (NIC) for connecting to a network link;

wherein the I/O nodes comprise a first I/O node for connecting to a first local network link, a second I/O node for connecting to a second local network link, and a third I/O node for connecting to an external network link;

a gateway for connecting to an external network through the external network link;

wherein the processing nodes are formed on a first form factor of the node cards;

wherein the storage nodes are formed on a second form factor of the node cards;

wherein the I/O nodes are formed on a third form factor of the node cards;

a direct interconnect switch on the processing nodes and connected to the direct interconnect fabric, for virtualizing access of the storage nodes and the I/O nodes by the plurality of processors;

wherein each dis-aggregated chassis connects to at least two other dis-aggregated chassis using the first local network link and the second local network link and the third local network link;

wherein the first modified dis-aggregated chassis connects to at least two other dis-aggregated chassis using the first local network link and the second local network link;

wherein the plurality of dis-aggregated chassis and the first modified dis-aggregated chassis are connected together by a mesh of local network links.

14. The dis-aggregated data cluster of claim 13 wherein the gateway comprises:

a firewall for blocking unauthorized incoming traffic from the external network; and

a load-balancer for distributing incoming requests from the external network among the processing nodes in the dis-aggregated data cluster.

15. A dis-aggregated data center comprising:

a plurality of dis-aggregated chassis, each dis-aggregated chassis comprising;

direct interconnect fabric means for interconnecting node cards;

processing nodes each having a plurality of processor means for executing instructions;

storage nodes for controlling storage media that stores files;

Input/Output (I/O) nodes each having a network interface controller (NIC) means for connecting to a network link;

wherein the I/O nodes comprise a first I/O node for connecting to a first local network link, a second I/O node for connecting to a second local network link, and a third I/O node for connecting to a third local network link;

wherein the processing nodes are formed on first node cards;

wherein the storage nodes are formed on second node cards;

wherein the I/O nodes are formed on third node cards;

direct interconnect switch means, on the processing nodes and connected to the direct interconnect fabric means, for virtualizing access of the storage nodes and the I/O nodes by the plurality of processors means;

a dis-aggregated data cluster which comprises a plurality of the dis-aggregated chassis wherein each dis-aggregated chassis in the dis-aggregated data cluster connects to at least two other dis-aggregated chassis in the dis-aggregated data cluster using the first local network link and the second local network link and the third local network link;

a first data-center LAN link in the dis-aggregated data cluster, wherein the first data-center LAN link is one of the third local network links that does not connect to LAN links for other dis-aggregated chassis in the dis-aggregated data cluster;

a second data-center LAN link in the dis-aggregated data cluster, wherein the second data-center LAN link is one of the third local network links that does not connect to LAN links for other dis-aggregated chassis in the dis-aggregated data cluster;

a first gateway to an external network, the first gateway connected to the first data-center LAN link; and

a second gateway to the external network, the second gateway connected to the second data-center LAN link,

whereby processing, storage, and I/O nodes are dis-aggregated into each dis-aggregated chassis.

16. The dis-aggregated data center of claim 15 wherein the dis-aggregated data center further comprises a plurality of the dis-aggregated data clusters;

wherein the first gateway further comprises;

first firewall means for blocking unauthorized incoming traffic from the external network; and

first load-balancer means for distributing incoming requests from the external network among the processing nodes in the dis-aggregated data center;

wherein the second gateway further comprises;

second firewall means for blocking unauthorized incoming traffic from the external network; and

second load-balancer means for distributing incoming requests from the external network among the processing nodes in the dis-aggregated data center.

17. The dis-aggregated data center of claim 16 wherein the first gateway further comprises first core switch means for connecting to the first data-center LAN link of each dis-aggregated data cluster in the plurality of the dis-aggregated data clusters;

wherein the second gateway further comprises second core switch means for connecting to the second data-center LAN link of each dis-aggregated data cluster in the plurality of the dis-aggregated data cluster.

18. The dis-aggregated data center of claim 17 wherein each of the processing nodes further comprises:

a main memory for storing instructions executed by a Central Processing Unit (CPU) in the plurality of processor means;

a local peripheral bus;

a north bridge that couples the main memory to the CPU, and drives the local peripheral bus with the peripheral requests generated by the CPU;

a south bridge interconnect fabric chip that comprises;

an I/O bus interface for physically interfacing to the local peripheral bus;

a virtual network controller that acts as a virtual endpoint for network-protocol requests from the CPU to access a network;

a virtual generic peripheral controller that acts as a virtual endpoint for generic requests from the CPU to access a virtualized local peripheral;

wherein the virtualized local peripheral appears to be located on the local peripheral bus and is addressable on the local peripheral bus by unmodified software running on the CPU that is not modified for virtualizing accesses of peripheral devices;

an I/O mapper that maps the network-protocol requests from the CPU to a first interconnect address, and that maps the generic requests from the CPU to a second interconnect address;

a packet formatter that encapsulates the network-protocol requests into interconnect packets, and that encapsulates the generic requests into interconnect packets; and

a fabric interface that sends and receives interconnect packets over the direct interconnect fabric means.

19. The dis-aggregated data center of claim 18 wherein each of the I/O nodes further comprises:

an interconnect switch for sending and receiving interconnect packets from the direct interconnect fabric means;

a remote packet formatter for extracting the network-protocol requests and the generic requests from the interconnect packets, and for encapsulating reply data into interconnect packets sent back to the processing nodes through the direct interconnect fabric means;

a remote network interface card that receives the network-protocol requests from the remote packet formatter, and sends and receives network packets over an external network in response to the network-protocol requests, and generates reply data received from the external network.

20. The dis-aggregated data center of claim 19 wherein each of the storage nodes further comprises:

an interconnect switch for sending and receiving interconnect packets from the direct interconnect fabric means;

a remote packet formatter for extracting the network-protocol requests and the generic requests from the interconnect packets, and for encapsulating reply data into interconnect packets sent back to the processing nodes through the direct interconnect fabric means; and

a remote generic peripheral controller that receives the generic requests from the remote packet formatter, the remote generic peripheral controller accessing a hard disk to generate the reply data in response to generic requests to access the hard disk.