UNIFIED SYSTEM AREA NETWORK AND SWITCH

• US 20110202701A1
• Filed: 04/21/2011
• Published: 08/18/2011
• Est. Priority Date: 11/05/2009
• Status: Active Grant

• Current Assignee: GigaIO Networks, Inc.
• Original Assignee: RJ Intellectual Property LLC
• Inventors: Maitra, Jayanta Kumar
• Granted Patent: US 8,463,934 B2
• Time in Patent Office: Days
• US Class Current: 710/308
• CPC Class Codes: G06F 13/4022 using switching circuits, e.g. switching matrix, connection or expansion network G06F13/4009 takes precedence
• CPC Class Codes: G06F 13/4027 using bus bridges G06F13/4022 takes precedence
• CPC Class Codes: G06F 13/4221 being an input/output bus, e.g. ISA bus, EISA bus, PCI bus, SCSI bus
• CPC Class Codes: H04L 41/0806 for initial configuration or provisioning, e.g. plug-and-play
• CPC Class Codes: H04L 49/35 Switches specially adapted for specific applications

Abstract

A network switch, based on the PCI Express protocol, is disclosed. The switch includes a processor, local memory and a plurality of non-transparent bridges. By configuring the non-transparent bridges appropriately, the network switch can facilitate a number of different communication mechanisms, including TCP/IP communication between servers, server clusters, and virtualized I/O device utilization. For example, the network switch may configure the non-transparent bridges so as to have access to the physical memory of every server attached to it. It can then move data from the memory of any server to the memory of any other server. In another embodiment, the network switch is connected to an I/O device, and multiple servers are given access to that I/O device via virtualized connections.

Claims

1. A method of transmitting messages between a plurality of servers, each server having a local memory and processor, comprising:

connecting each of said plurality of servers to a switch using PCI Express, said switch comprising a plurality of non-transparent bridges, one of said non-transparent bridges used to connect to a respective one of said plurality of servers,

configuring each of said non-transparent bridges to allow said network switch to access at least a portion of said local memory of said server connected to said non-transparent bridge;

building a look up table in said network switch associating a destination address with one of said plurality of non-transparent bridges;

creating a message in said local memory in a first server, said message having a destination address;

using said network switch to read said destination address in said message by accessing said local memory of said first server using a first non-transparent bridge associated with said first server;

determining the destination server by looking up said destination address in said look up table;

using said network switch to read said message by accessing said local memory of said first server using said first non-transparent bridge; and

copying said message to local memory of said destination server by accessing said local memory of said destination server by using a second non-transparent bridge associated with said destination server.

2. The method of claim 1, wherein said messages comprise IP messages and said destination address comprises an IP address.

3. The method of claim 1, wherein a direct memory access (DMA) engine located in said network switch is used to read said message from said first server and copy said message to said destination server.

4. The method of claim 1, wherein said first non-transparent bridge comprises a messaging register to allow communication between said processor of said first server and a processor in said network switch, and said processor of said first server notifies said network switch of the existence of said message using said messaging register.

5. The method of claim 1, wherein said second non-transparent bridge comprises a messaging register to allow communication between said processor of said destination server and a processor in said network switch, and said network switch notifies said destination server of the existence of said message using said messaging register.

6. The method of claim 1, wherein said first server, said destination server and said network switch are all attached via a standard network connection, and said processor of said first server notifies said network switch of the existence of said message using said network connection.

7. The method of claim 1, wherein said first server, said destination server and said network switch are all attached via a standard network connection, and said network switch notifies said destination server of the existence of said message using said network connection.

8. The method of claim 1, wherein said first server creates a transmit list comprising one or more entries, wherein each of said entries comprises a pointer to the physical address of a message in said local memory.

9. The method of claim 8, wherein said transmit list comprises a head, indicating next message to be transmitted, and a tail, indicating the last message to be sent.

10. The method of claim 9, wherein said first server updates said tail and said network switch updates said head.

11. The method of claim 8, wherein said first server creates a plurality of said transmit lists, wherein said list is associated with a different destination address.

12. The method of claim 1, wherein said destination server creates a receive list comprising one or more entries, wherein each of said entries comprises a pointer to the physical address in said local memory where a received message is copied.

13. The method of claim 12, wherein said receive list comprises a head, indicating the location of the next message to be processed by said destination server, and a tail, indicating the last location used to store a received message by said network switch.

14. The method of claim 13, wherein said destination server updates said head and said network switch updates said tail.

15. The method of claim 13, wherein said receive list further comprises a free pointer, indicating the location in said local memory of said destination server where said network switch should copy the next message.

16. A system for transmitting messages between a plurality of servers, comprising:

a first server having a first local memory, a first processor and a first PCI Express port,

A second server having a second local memory, a second processor and a second PCI Express port;

a third server having a third local memory, a third processor and a third PCI Express port;

a network switch comprising a fourth processor, a fourth local memory, a first non-transparent bridge connected to said first PCI Express port, a second non-transparent bridge connected to said second PCI express port, and a third non-transparent bridge connected to said third PCI express port wherein said fourth local memory further comprises computer executable instructions adapted to;

configure each of said first, second and third non-transparent bridges to allow said network switch to access at least a portion of said local memory of said first, second and third servers;

build a look up table in said fourth local memory associating a destination address with one of said non-transparent bridges;

read a destination address in a message in said first local memory by accessing said first local memory by using said first non-transparent bridge;

determine the destination server by looking up said destination address in said look up table, wherein said destination server is selected from said second and said third server;

read said message by accessing said first local memory of said first server using said first non-transparent bridge; and

copy said message to local memory of said destination server by accessing said local memory of said destination server by using said non-transparent bridge associated with said destination server.

17. The system of claim 16, wherein said messages comprise IP messages and said destination address comprises an IP address.

18. The system of claim 16, wherein said first non-transparent bridge further comprises at least one messaging register, and said fourth memory further comprises instructions to communicate with said first processor using said messaging register.

19. The system of claim 16, further comprising a network connection attached to said first server and said network switch, and said fourth memory further comprises instructions to communicate with said first processor using said network connection.

20. The system of claim 16, wherein said network switch further comprises a PCI Express port;

and further comprising an extension switch, said extension switch comprising a bridge connected to said PCI Express port; and

at least one non-transparent bridge for connection to a fourth server.

21. A method of clustering two servers, each server having a local memory and processor, comprising:

connecting each of said two servers to a switch using PCI Express, said switch comprising a processor, a local memory and a plurality of non-transparent bridges, a first of said non-transparent bridges used to connect to a first server and a second of said non-transparent bridges used to connect to a second server;

configuring said first non-transparent bridge to allow said switch to access a first portion of said local memory of said first server;

configuring said second non-transparent bridge to allow said switch to access a second portion of local memory of said second server;

using said first server to write data destined for said second server to said first portion;

using said network switch to copy data from said first portion using said first non-transparent bridge to said second portion using said second non-transparent bridge; and

using said second server to read data from said second portion.

22. The method of claim 21, wherein a direct memory access (DMA) engine located in said network switch is used to read said data from said first portion and copy said data to said second portion.

23. The method of claim 21, wherein said first non-transparent bridge comprises a messaging register to allow communication between said processor of said first server and a processor in said network switch, and said processor of said first server notifies said network switch of the existence of said data in said first portion using said messaging register.

24. The method of claim 21, wherein said second non-transparent bridge comprises a messaging register to allow communication between said processor of said second server and a processor in said network switch, and said network switch notifies said second server of the existence of said data in said second portion using said messaging register.

25. The method of claim 21, wherein said first server, said second server and said network switch are attached via a network connection and said processor of said first server notifies said network switch of the existence of said data in said first portion using said network connection.

26. The method of claim 21, wherein said first server, said second server and said network switch are attached via a network connection and said processor of said network switch notifies said second server of the existence of said data in said second portion using said network connection.

27. A system for clustering two servers, comprising:

a first server having a first local memory, a first processor and a first PCI Express port,

A second server having a second local memory, a second processor and a second PCI Express port;

network switch comprising a third processor, a third local memory, a first non-transparent bridge connected to said first PCI Express port, and a second non-transparent bridge connected to said second PCI express port, wherein said third local memory further comprises computer executable instructions adapted to;

configure said first non-transparent bridge to allow said network switch to access a first portion of said first local memory;

configure said second non-transparent bridge to allow said network switch to access a second portion of said second local memory;

receive a notification from said first server of new data written in said first portion;

copy data from said first portion to said second portion using said first and second non-transparent bridges, in response to said notification; and

notify said second server that new data is available in said second portion after copying said data to said second portion.

28. The system of claim 27, further comprising a DMA engine in said network switch to copy said data from said first portion to said second portion.

29. The system of claim 27, wherein said non-transparent bridges each comprise a messaging register, and said notifications are transmitted using said messaging register.

30. The system of claim 27, wherein said first server, said second server and said network switch are attached via a network connection and said notifications are transmitted using said network connection.

31. A method of clustering two servers, each server having a local memory and processor, comprising:

connecting each of said two servers to a switch using PCI Express, said switch comprising a plurality of non-transparent bridges, a first of said non-transparent bridges used to connect to a first server and a second of said non-transparent bridges used to connect to a second server;

configuring said first non-transparent bridge such that accesses from said first server and accesses from said second server utilize the same logical address range, such that read and write operations from said first server via said first non-transparent bridge are claimed by said second non-transparent bridge and read and write operations from said second server via said second non-transparent bridge are claimed by said first non-transparent bridge;

using said first server to perform read and write operations, whereby said operations are performed on said local memory of said second server; and

using said second server to perform read and write operations, whereby said operations are performed on said local memory of said first server.

32. A system for clustering two servers, comprising:

a first server having a first local memory, a first processor and a first PCI Express port,

A second server having a second local memory, a second processor and a second PCI Express port;

network switch comprising a third processor, a third local memory, a first non-transparent bridge connected to said first PCI Express port, and a second non-transparent bridge connected to said second PCI express port, wherein said third local memory further comprises computer executable instructions configured to;

configure said first non-transparent bridge such that accesses from said first server and accesses from said second server utilize the same logical address range, such that read and write operations from said first server via said first non-transparent bridge are claimed by said second non-transparent bridge and read and write operations from said second server via said second non-transparent bridge are claimed by said first non-transparent bridge.

33. A method of sharing an I/O device among a plurality of computers, comprising:

connecting each of said plurality of computers to a network switch using PCI Express, said switch comprising a plurality of non-transparent bridges, one of said non-transparent bridges used to connect to a respective one of said plurality of servers, and at least one PCI Express port;

connecting said I/O device to said network switch via said PCI Express port;

using said network switch to enumerate said I/O device and assign an I/O space to said I/O device;

configuring each of said non-transparent bridges to utilize said assigned I/O space, such that each of said computers can access said I/O device directly.

34. The method of claim 33, wherein each of said non-transparent bridges comprises a messaging register and said computers request access to said I/O device via said messaging register.

35. The method of claim 33, wherein said plurality of computers and said network switch are attached to a network connection, and wherein said computers request access to said I/O device via said network connection.

36. A system for sharing a I/O device among two computers, comprising:

a first computer having a first local memory, a first processor and a first PCI Express port,

a second computer having a second local memory, a second processor and a second PCI Express port;

a network switch comprising a third processor, a third local memory, a first non-transparent bridge connected to said first PCI Express port, a second non-transparent bridge connected to said second PCI express port, and a third bridge connected to said I/O device, wherein said third local memory further comprises computer executable instructions configured to;

enumerate said I/O device and assign an I/O space to said I/O device; and

configure each of said non-transparent bridges to utilize said assigned I/O space, such that each of said computers can access said I/O device directly.

37. The system of claim 36, wherein each of said non-transparent bridges comprises a messaging register, and said third local memory comprises instruction configured to:

accept access requests to said I/O device from said computers via said messaging register; and

grant access to one of said computers.