System and method for using a multi-protocol fabric module across a distributed server interconnect fabric

• US 9,680,770 B2
• Filed: 10/12/2013
• Issued: 06/13/2017
• Est. Priority Date: 10/30/2009
• Status: Active Grant

• Current Assignee: III Holdings 2, LLC
• Original Assignee: III Holdings 2, LLC
• Inventors: Davis, Mark Bradley
• Primary Examiner(s): Anwar, Mohammad
• Assistant Examiner(s): Detse, Kokou R
• Application Number:
  US14/052,723
• Publication Number:
  US 20150103826A1
• Time in Patent Office: 1,340 Days
• Field of Search: 370400, 370392
• CPC Class Codes: H04L 45/52 Multiprotocol routers
• CPC Class Codes: H04L 45/74 Address processing for routing
• CPC Class Codes: H04L 49/25 Routing or path finding in a switch fabric
• CPC Class Codes: H04L 49/602 Multilayer or multiprotocol switching, e.g. IP switching

Abstract

A multi-protocol personality module enabling load/store from remote memory, remote Direct Memory Access (DMA) transactions, and remote interrupts, which permits enhanced performance, power utilization and functionality. In one form, the module is used as a node in a network fabric and adds a routing header to packets entering the fabric, maintains the routing header for efficient node-to-node transport, and strips the header when the packet leaves the fabric. In particular, a remote bus personality component is described. Several use cases of the Remote Bus Fabric Personality Module are disclosed: 1) memory sharing across a fabric connected set of servers; 2) the ability to access physically remote Input Output (I/O) devices across this fabric of connected servers; and 3) the sharing of such physically remote I/O devices, such as storage peripherals, by multiple fabric connected servers.

Claims

1. A network system, comprising:

a plurality of switching nodes, wherein each switching node includes;

one or more processors;

a bus fabric coupled to the one or more processors;

a fabric switch coupled to a switched interconnect fabric that connects the plurality of switching nodes; and

a plurality of protocol-specific personality modules, each coupled to the bus fabric and to the fabric switch wherein at least one of the protocol-specific personality modules includes a routing header processor, and wherein the routing header processor is configured to;

inspect received data packets;

add a routing header to received data packets that are entering the switched interconnect fabric; and

remove the routing header from received data packets that are leaving the switched interconnect fabric, wherein addition and removal of the routing header is performed on a protocol-dependent basis.

2. The network system of claim 1, wherein at least two of the protocol-specific personality modules are configured in accordance with different protocols with respect to one other.

3. A switching node, comprising:

a plurality of protocol-specific personality modules, wherein at least two of the protocol-specific personality modules are configured in accordance with different protocols with respect to each other, and wherein at least one of the protocol-specific personality modules includes a routing header processor that is configured to;

inspect received data packets;

add a routing header to received data packets that are entering the switched interconnect fabric; and

remove the routing header from received data packets that are leaving the switched interconnect fabric, wherein addition and removal of the routing header is performed on a protocol-dependent basis.

4. A System on a Chip (SoC) fabric node for transporting data packets across a fabric, the SoC fabric node comprising:

one or more processors;

a bus fabric coupled to the one or more processors;

a fabric switch coupled to the fabric, wherein the fabric switch is configured to allow a packet to be routed to a destination within the fabric using only information in a fabric routing header of the packet without inspection of a data payload of the packet;

a local area network data communication portion coupled to the bus fabric and to the fabric switch, wherein the local area network data communication portion includes;

a local area network data communication controller connected to the bus fabric and configured to generate local data communication frames;

a local area network data communication bridge personality module connected to the local area network data communication controller, wherein the local area network data communication bridge personality module is configured to add a fabric routing header to a local area network data communication frame to create a routing frame that is to be routed to a destination within the fabric using only information in the fabric routing header of the routing frame, wherein the local area network data communication bridge personality module is coupled to the fabric switch, and wherein the fabric switch is configured to accept the routing frame and output the routing frame to another node of the fabric; and

a remote bus portion coupled to the bus fabric and the fabric switch in parallel with the local area network data communication portion, wherein the remote bus portion is configured to allow bus transactions to be transported between nodes of the fabric.

5. The SoC fabric node of claim 4, wherein the remote bus portion includes a remote interrupt manager connected to a remote bus personality processor and to an interrupt line, wherein the interrupt manager is configured to accept an interrupt and create a packet tagged as an interrupt packet to be sent to the remote bus personality processor.

6. The SoC fabric node of claim 4, wherein the remote bus portion includes:

a remote address translation module connected to the bus fabric and configured to convert a local address of a bus transaction to a remote node address;

a bus bridge connected to the remote address translation module and configured to packetize the bus transaction; and

a remote bus personality processor connected to the bus bridge and configured to add a fabric routing header to the packetized bus transaction to create a bus transaction routing packet to be routed to a destination within the fabric using only information in the fabric routing header of the bus transaction routing packet.

7. The SoC fabric node of claim 6, wherein the remote bus personality processor is coupled to the fabric switch, and wherein the fabric switch is configured to accept the bus transaction routing packet and output the bus transaction routing packet to another node of the fabric.

8. A remote bus personality portion of a System on a Chip (SoC) fabric node, the remote bus personality portion comprising:

a remote address translation module configured to accept a data packet that has a local address and to convert the local address to a remote address of a different SoC fabric node;

a bus bridge configured to transform a bus transaction to one or more packets that are compatible with a switched interconnect fabric through which the SoC fabric node is coupled to the different SoC fabric node; and

a remote bus processor module configured to add a routing header to the one or more packets such that the one or more packets can enter the switched interconnect fabric through a fabric switch that allows the one or more packets to be routed to an intended destination within the fabric using only information in the fabric routing header of the one or more packets without inspection of a data payload of the one or more packets, and wherein information in the routing header is representative of the intended destination.

9. The remote bus personality portion of claim 8, further comprising an interrupt line in communication with an interrupt manager, wherein the interrupt manager is connected to the remote bus processor module and configured to generate an interrupt.

10. A method of sharing memory across nodes in a fabric of interconnected nodes, the method comprising:

issuing a transaction instruction to a remote bus personality portion of a local System on a Chip (SoC) fabric node to initiate a bus transaction;

packetizing at least a portion of the bus transaction by creating one or more packets compatible with the fabric;

creating a routing header for a destination SoC fabric node and adding the routing header to the one or more packets to create a routing frame for routing the one or more packets to the destination SoC fabric node using only information in the fabric routing header of the one or more packets and without inspection of a data payload of the one or more packets; and

transmitting the routing frame into the fabric and delivering the routing frame to the destination SoC fabric node using only information within the routing header.

11. A method of accessing an Input/Output (I/O) device, the method comprising:

issuing a bus transaction to a remote bus personality portion of a server, wherein the remote bus personality portion is coupled to a bus fabric and to a fabric switch of the server for enabling the bus transaction to be transported between nodes of a fabric of interconnected nodes, wherein the remote bus personality portion includes;

a remote address translation module connected to the bus fabric for converting a local address of the bus transaction to a remote node address;

a bus bridge connected to the remote address translation module to packetize the bus transaction; and

a remote bus personality processor connected to the bus bridge for adding a fabric routing header to the packetized bus transaction to create a routing packet configured to be routed to a destination within the fabric using only information in the fabric routing header;

sending the bus transaction from the remote bus personality portion to the I/O device via the fabric;

creating a routing header at the I/O device for the server and adding the routing header to a payload to create a routing frame configured to be routed to the destination using only information in the routing header; and

transmitting the routing frame from the I/O device into the fabric and delivering the routing frame to the server using only information within the routing header and without inspection of the payload.

12. The method of claim 11, wherein the I/O device is a remote Peripheral Component Interconnect Express (PCIe) controller.

13. The method of claim 11, wherein the I/O device is a remote Serial Advanced Technology Attachment (SATA) controller.

14. The method of claim 11, wherein the I/O device includes a direct memory access (DMA) engine.

15. The method of claim 11, further comprising routing direct memory access (DMA) traffic bi-directionally via the fabric.

16. The method of claim 11, further comprising generating an interrupt by the I/O device and transmitting the interrupt to the server.

17. A method of sharing storage among a plurality of servers and a plurality of storage peripherals, the method comprising:

sending a message from a server to a peripheral, wherein the message describes information to be accessed at the peripheral, and wherein the message includes a physical address of the information on the server;

issuing a transaction instruction to a remote bus personality portion on the peripheral for the information;

packetizing the information into one or more packets;

creating a routing header;

adding the routing header to the one or more packets to create one or more routing frames, wherein each of the one or more routing frames is configured to be routed to the server using only information in the routing header and without inspection of a data payload of the one or more packets; and

transmitting the one or more routing frames into a fabric of interconnected nodes and delivering the one or more routing frames to the server.

18. The method of claim 17, wherein peripheral is a Serial Advanced Technology Attachment (SATA) controller.

19. The method of claim 17, wherein the message contains a description of a read address and of the physical address in the server.

20. The method of claim 17, wherein said transmitting comprises using a direct memory access (DMA) engine to initiate a DMA transfer over the fabric from the peripheral to the server.