Switching apparatus and method for providing shared I/O within a load-store fabric

• US 7,219,183 B2
• Filed: 04/19/2004
• Issued: 05/15/2007
• Est. Priority Date: 01/21/2003
• Status: Active Grant

• Current Assignee: Avago Technologies International Sales Pte Limited (Broadcom, Inc.)
• Original Assignee: NextIO Incorporated
• Inventors: Khan, Asif, Pagan, Annette, Utley, Robert Haskell, Pekkala, Richard E., Pettey, Christopher J.
• Primary Examiner(s): Auve; Glenn A.
• Assistant Examiner(s): Cleary; Thomas J.
• Application Number:
  US10/827,622
• Publication Number:
  US 20050025119A1
• Time in Patent Office: 1,121 Days
• Field of Search: 710/300, 710/305, 710/306, 710/316, 710/317, 710 2- 4, 710 72- 74, 710/30, 710/38, 370/351, 370/360, 370/362, 370/428, 709/236, 709/238
• US Class Current: 710/316
• CPC Class Codes: H04L 49/25 Routing or path finding in a switch fabric
• CPC Class Codes: H04L 49/253 using establishment or release of connections between ports
• CPC Class Codes: H04L 49/254 Centralised controller, i.e. arbitration or scheduling
• CPC Class Codes: H04L 49/602 Multilayer or multiprotocol switching, e.g. IP switching
• CPC Class Codes: H04L 49/604 Hybrid IP/Ethernet switches

Abstract

An apparatus and method for sharing I/O devices. The apparatus has a first plurality of I/O ports, a second I/O port, and core logic. The first plurality is coupled to a plurality of operating system domains through a load-store fabric. Each of the first plurality routes transactions between the operating system domains and the switching apparatus. The second I/O port is coupled to a first shared input/output endpoint. The first shared input/output endpoint requests/completes transactions for each of the plurality of operating system domains. The core logic is coupled to the first plurality of I/O ports and the second I/O port. The core logic routes the transactions between the first plurality of I/O ports and the second I/O port and associates each of the transactions with a corresponding one of the plurality of operating system domains (OSDs) by encapsulating an OS domain header within a transaction layer packet.

Claims

1. A switching apparatus for sharing input/output endpoints, the switching apparatus comprising:

a first plurality of I/O ports, coupled to a plurality of operating system domains through a load-store fabric, each configured to route transactions between said plurality of operating system domains and the switching apparatus;

a second I/O port, coupled to a first shared input/output endpoint, wherein said first shared input/output endpoint is configured to request/complete said transactions for each of said plurality of operating system domains; and

core logic, coupled to said first plurality of I/O ports and said second I/O port, configured to route said transactions between said first plurality of I/O ports and said second I/O port, and configured to associate each of said transactions with a corresponding one of said plurality of operating system domains (OSDs), said corresponding one of said plurality of OSDs corresponding to one or more root complexes, wherein said core logic designates said corresponding one of said plurality of OSDs according to a variant of a protocol that otherwise provides for routing of said transactions only for a single operating system domain, and wherein said variant comprises encapsulating an OS domain header within a transaction layer packet that otherwise comports with said protocol.

2. The switching apparatus as recited in claim 1, wherein the switching apparatus routes said transactions according to said, and wherein said first plurality of I/O ports communicates with said plurality of operating system domains according to said protocol.

3. The switching apparatus as recited in claim 2, wherein said protocol comprises PCI Express, and wherein said core logic isolates said transactions over a plurality of PCI Express bus hierarchies according to said each of said plurality of operating system domains.

4. The switching apparatus as recited in claim 1, wherein said load-store fabric comprises a point-to-point fabric.

5. The switching apparatus as recited in claim 1, wherein said core logic maps said each of said plurality of operating system domains to a corresponding root complex.

6. The switching apparatus as recited in claim 5, wherein said corresponding root complex is associated with one or more processing complexes, each comprising:

a processing core, configured to process data, wherein said processing core accesses a memory that is configured to store said data.

7. The switching apparatus as recited in claim 6, wherein said data is transferred from said corresponding root complex to a corresponding one of said first plurality of I/O ports according said protocol.

8. The switching apparatus as recited in claim 6, wherein said data is transferred to said root complex from a corresponding one of said first plurality of I/O ports according to said protocol.

9. The switching apparatus as recited in claim 1, wherein said load-store fabric comprises hardware, software, or a combination of hardware and software that is configured to route said transactions.

10. The switching apparatus as recited in claim 1, wherein said first shared input/output endpoint comprises a shared network interface controller (NIC).

11. The switching apparatus as recited in claim 10, wherein said shared NIC comprises a shared Ethernet controller.

12. The switching apparatus as recited in claim 1, wherein said first shared input/output endpoint comprises a Fiber Channel controller.

13. The switching apparatus as recited in claim 1, wherein said first shared input/output endpoint comprises a shared RAID controller.

14. The switching apparatus as recited in claim 1, wherein said first shared input/output endpoint is integrated into the switching apparatus.

15. The switching apparatus as recited in claim 1, wherein said first shared input/output endpoint is integrated into a root complex.

16. The switching apparatus as recited in claim 1, wherein said first shared input/output endpoint is integrated into a processing complex.

17. The switching apparatus as recited in claim 1, wherein said plurality of OSDs comprises 16 OSDs.

18. The switching apparatus as recited in claim 1, wherein said transaction layer packet is routed between said second I/O port and said first shared input/output endpoint.

19. The switching apparatus as recited in claim 18, wherein said first shared I/O endpoint is configured to detect said OS domain header and to perform a specified operation according to said protocol exclusively for said one of said plurality of OSDs.

20. The switching apparatus as recited in claim 1, wherein said first shared input/output endpoint designates one of said transactions for a particular one of said plurality of operating system domains by encapsulating an OS domain header within a transaction layer packet, and wherein said transaction layer packet is routed to said core logic via said second I/O port.

21. The switching apparatus as recited in claim 20, wherein said core logic is configured to detect and decapsulate said OS domain header from said transaction layer packet, and is configured to route said one of said transactions to said particular one of said plurality of operating system domains according said protocol.

22. The switching apparatus as recited in claim 1, wherein said core logic is integrated within a shared I/O switch.

23. The switching apparatus as recited in claim 1, wherein said core logic is integrated within a root complex.

24. The switching apparatus as recited in claim 1, wherein said core logic is integrated within a processing complex.

25. A shared input/output (I/O) switching mechanism, comprising:

core logic, configured to enable operating system domains to share one or more I/O endpoints, said core logic comprising;

global routing logic, configured to route first transactions to/from said operating system domains, and for routing second transactions to/from said one or more I/O endpoints, wherein each of said second transactions designates an associated one of said operating system domains for which an operation specified by each of said first transactions be performed, wherein said second transactions comport with a variant of a protocol that otherwise provides exclusively for a single operating system domain within said load-store fabric, and wherein said variant comprises encapsulating an OS domain header within a transaction layer packet of said each of said second transactions, wherein said each of said second transactions otherwise comports with said protocol.

26. The shared I/O switching mechanism as recited in claim 25, wherein said first transactions comport with said protocol.

27. The shared I/O switching mechanism as recited in claim 25, wherein said first transactions comport with PCI Express.

28. The shared I/O switching mechanism as recited in claim 27, wherein said core logic isolates said first and second transactions over a plurality of PCI Express bus hierarchies according to each of said operating system domains.

29. The shared I/O switching mechanism as recited in claim 28, wherein said first and second transactions are routed in accordance with addressing mechanisms for a particular one of said plurality of PCI Express bus hierarchies, said particular one of said plurality of PCI Express bus hierarchies corresponding to said associated one of said operating system domains.

30. The shared I/O switching mechanism as recited in claim 25, wherein said core logic associates each of said operating system domains with each of a corresponding root complex.

31. The shared I/O switching mechanism as recited in claim 25, wherein said transaction layer packet is routed between said the shared I/O switching mechanism and one of said one or more I/O endpoints.

32. The shared I/O switching mechanism as recited in claim 31, wherein said one of said one or more I/O endpoints is configured to detect said OS domain header and to perform said operation according to said protocol exclusively for said one of said operating system domains.

33. The shared I/O switching mechanism as recited in claim 25, wherein said the shared I/O switching mechanism comprises a shared I/O switch.

34. The shared I/O switching mechanism as recited in claim 25, wherein one of said one or more I/O endpoints designates one of said second transactions for a particular one of said operating system domains by encapsulating said OS domain header within a transaction layer packet corresponding to said one of said second transactions, and wherein said transaction layer packet is routed to the shared I/O switching mechanism.

35. The shared I/O switching mechanism as recited in claim 34, wherein VMAC logic within the shared I/O switching mechanism is configured to detect and decapsulate said OS domain header from said transaction layer packet corresponding to said one of said second transactions, and wherein said core logic routes a corresponding one of said first transactions to said particular one of said plurality of operating system domains according to said protocol.

36. A method for interconnecting independent operating system domains to a shared I/O endpoint comprising:

via first ports, first communicating with each of the independent operating system domains according to a protocol that provides exclusively for a single operating system domain within a the load-store fabric;

via a second port, second communicating with the shared I/O endpoint according to a variant of the protocol to enable the shared I/O endpoint to associate a prescribed operation with a corresponding one of the independent operating system domains, said second communicating comprising;

employing the variant of the protocol to associate a unique root complex with the corresponding one of the operating system domains by encapsulating an OS domain header within a transaction layer packet that otherwise comports with the protocol, wherein the value of the OS domain header designates the corresponding one of the operating system domains; and

via core logic within a switching apparatus, mapping the independent operating system domains to the shared I/O endpoint.

37. The method as recited in claim 36, further comprising:

over a point-to-point fabric, first interconnecting the independent operating system domains to the switching apparatus via the first ports; and

over the point-to-point fabric, second interconnecting the shared I/O endpoint to the switching apparatus via the second port.

38. The method as recited in claim 36, wherein said first communicating comprises:

associating each of the operating system domains with a corresponding root complex.

39. The method as recited in claim 38 wherein the corresponding root complex is associated with one or more processing complexes, each configured to process data, and wherein the data is stored in a memory.

40. The method as recited in claim 39, further comprising:

transferring the data to/from the corresponding root complex according to the protocol.

41. The method as recited in claim 39, further comprising:

transferring the data to the corresponding root complex according to the protocol.

42. The method as recited in claim 36, wherein said first communicating comprises:

employing PCI Express as the protocol.

43. The method as recited in claim 42, wherein said mapping comprises:

isolating transactions over a plurality of PCI Express bus hierarchies according to each of the operating system domains.

44. The method as recited in claim 43, wherein said mapping further comprises:

routing the transactions in accordance with addressing mechanisms for a particular one of the plurality of PCI Express bus hierarchies, the particular one of said plurality of PCI Express bus hierarchies corresponding to the corresponding one of the operating system domains.

45. The method as recited in claim 36, wherein said second communicating further comprises:

routing the transaction layer packet between the second port and the shared I/O endpoint.

46. The method as recited in claim 45, wherein said second communicating further comprises:

within the I/O endpoint, detecting the OS domain header; and

performing the prescribed operation according to the protocol, wherein said performing is accomplished exclusively for the corresponding one of the operating system domains.

47. The method as recited in claim 36, wherein said second communicating comprises:

within the shared I/O endpoint, designating a transaction for a particular one of the operating system domains by embedding an OS domain header within a transaction layer packet corresponding to the transaction; and

first transmitting the transaction layer packet to the second port.

48. The method as recited in claim 47, wherein said second communicating comprises:

within the switching apparatus, detecting and removing the OS domain header from the transaction layer packet corresponding to the transaction.

49. The method as recited in claim 48, wherein said first communicating comprises:

second transmitting the transaction layer packet corresponding to the transaction to the particular one of the operating system domains according to the protocol.