Method and apparatus for shared I/O in a load/store fabric

• US 7,457,906 B2
• Filed: 01/14/2004
• Issued: 11/25/2008
• Est. Priority Date: 01/21/2003
• Status: Active Grant

• Current Assignee: Mellanox Technologies Limited (NVIDIA Corporation)
• Original Assignee: NextIO Incorporated
• Inventors: Pagan, Annette, Pekkala, Richard E., Utley, Robert Haskell, Pettey, Christopher J., Khan, Asif
• Primary Examiner(s): Auve, Glenn A
• Assistant Examiner(s): VU, TRISHA U
• Application Number:
  US10/757,713
• Publication Number:
  US 20040172494A1
• Time in Patent Office: 1,777 Days
• Field of Search: 710/305, 710/306, 710/311, 710/30, 710/31, 710/38, 710/316, 710/317, 370351-357, 370/350, 370/362, 370386-388, 370/360, 370/389, 709/244
• US Class Current: 710/316
• CPC Class Codes: G06F 13/4022 using switching circuits, e.g. switching matrix, connection or expansion network G06F13/4009 takes precedence
• CPC Class Codes: G06F 2213/0026 PCI express
• CPC Class Codes: H04L 12/4633 Interconnection of networks using encapsulation techniques, e.g. tunneling

Abstract

An apparatus and method is provided for allowing I/O devices to be shared and/or partitioned among a plurality of processing complexes within the load/store fabric of each of the processing complexes without requiring modification to the operating system or driver software of the processing complexes. The apparatus and method includes a switch for selectively coupling each of the processing complexes to one or more shared I/O devices. The apparatus and method further includes placing information within packets transmitted between the switch and the I/O devices to identify which of the processing complexes the packets are associated with. The invention further includes an apparatus and method within the shared I/O devices to allow the shared I/O devices to service each of the processing complexes independently.

Claims

1. A shared input/output (I/O) fabric within a load/store domain, comprising:

a plurality of PCI Express root complexes, wherein at least two of said root complexes do not include a dedicated network interface controller;

a shared I/O switch coupled to said plurality of root complexes; and

a shared PCI Express endpoint coupled to said shared switch, the endpoint comprising an I/O controller configured to be shared by the at least two of said root complexes;

wherein said shared I/O switch

receives packets from each of said plurality of root complexes,

places root complex identification within said packets for use by said shared I/O controller, and

transmits said packets with said root complex identification to said shared I/O controller for processing.

2. The shared input/output (I/O) fabric as recited in claim 1 wherein the shared input/output (I/O) fabric utilizes a PCI Express Architecture.

3. The shared input/output (I/O) fabric as recited in claim 1 wherein said plurality of root complexes comprise a plurality of operating system domains (OSD'"'"'s).

4. The shared input/output (I/O) fabric as recited in claim 3 wherein each of said plurality of OSD'"'"'s has an operating system.

5. The shared input/output (I/O) fabric as recited in claim 4 wherein at least one of said plurality of OSD'"'"'s executes an operating system that is different from other ones of said plurality of OSD'"'"'s.

6. The shared input/output (I/O) fabric as recited in claim 3 wherein each of said plurality of OSD'"'"'s comprise:

a processing complex; and

memory, coupled to said processing complex for storing data to be utilized by said processing complex.

7. The shared input/output (I/O) fabric as recited in claim 6 wherein said processing complex comprises one or more processors.

8. The shared input/output (I/O) fabric as recited in claim 1 wherein each of said plurality of root complexes comprise at least one port for coupling to said shared I/O switch.

9. The shared input/output (I/O) fabric as recited in claim 8 wherein said port conforms to a serial load/store architecture.

10. The shared input/output (I/O) fabric as recited in claim 8 wherein said port conforms to PCI Express architecture.

11. The shared input/output (I/O) fabric as recited in claim 1 wherein said shared I/O switch comprises a plurality of ports.

12. The shared input/output (I/O) fabric as recited in claim 11 wherein each of said plurality of root complexes is coupled to at least one of said plurality of ports to allow communication there between.

13. The shared input/output (I/O) fabric as recited in claim 1 wherein when said shared I/O switch receives a packet from one of said plurality of root complexes, it identifies which of said one of said plurality of root complexes transmitted said packet based on a port within said shared I/O switch coupled to said one of said plurality of root complexes.

14. The shared input/output (I/O) fabric as recited in claim 13 wherein the port is associated with a bus hierarchy within the load/store domain

15. The shared input/output (I/O) fabric as recited in claim 13 wherein said shared I/O switch places said root complex identification in said packet based upon said port.

16. The shared input/output (I/O) fabric as recited in claim 13 wherein said shared I/O switch associates said one of said plurality of root complexes that transmitted said packet with said one of said plurality of root complexes based on a PCI bus hierarchy within said shared I/O switch.

17. The shared input/output (I/O) fabric as recited in claim 13 wherein the association by said shared I/O switch utilizes a lookup table.

18. The shared input/output (I/O) fabric as recited in claim 1 wherein said shared I/O controller comprises:

a plurality of OS Domain portions.

19. The shared input/output (I/O) fabric as recited in claim 18 wherein each of said plurality of OS Domain portions comprise control registers.

20. The shared input/output (I/O) fabric as recited in claim 19 wherein each of said plurality of OS Domain portions further comprise a descriptor.

21. The shared input/output (I/O) fabric as recited in claim 1 wherein said shared I/O controller further comprises a bus interface for determining which of said plurality of root complexes is identified within said packets.

22. The shared input/output (I/O) fabric as recited in claim 1 wherein said shared I/O controller further comprises a bus interface for determining which of a plurality of operating system domains (OSD'"'"'s) is identified within said packets.

23. The shared input/output (I/O) fabric as recited in claim 1 wherein a bus interface within said shared I/O controller places said root complex identification within packets destined for said plurality of root complexes to allow said shared I/O switch to transmit said packets to an appropriate one of said plurality of root complexes.

24. The shared input/output (I/O) fabric as recited in claim 1 wherein said root complex identification comprises a plurality of bits which are inserted within said packets between said shared I/O switch and said shared I/O controller.

25. The shared input/output (I/O) fabric as recited in claim 1, wherein said root complex identification identifies a particular PCI Express root complex which is a source of a received packet.

26. A serial communication architecture between a plurality of operating system domains (OSD'"'"'s) and a plurality of PCI Express endpoints to allow each of the plurality of OSD'"'"'s to share each of the plurality of endpoints, the architecture comprising:

a first link, between each of the plurality of OSD'"'"'s and a shared I/O switch;

a second link, between said shared I/O switch and each of the plurality of endpoints, said shared I/O switch associating packets from the plurality of OSD'"'"'s with the OSD'"'"'s by embedding a header within said packets before transmitting said packets to the plurality of endpoints;

wherein at least two of said plurality of operating system domains do not include a dedicated network interface controller and are configured to share said plurality of endpoints.

27. The serial communication architecture as recited in claim 26 wherein the plurality of endpoints comprise:

a first endpoint for network communication; and

a second endpoint for communication with data storage devices.

28. The serial communication architecture as recited in claim 26 wherein the plurality of endpoints comprise:

a keyboard controller; and

a mouse controller.

29. The serial communication architecture as recited in claim 28 wherein the plurality of endpoints further comprise:

a video controller.

30. The serial communication architecture as recited in claim 27 wherein each of the plurality of operating system domains communicates with a network through said first endpoint.

31. The serial communication architecture as recited in claim 26 wherein said first link comprises PCI Express.

32. The serial communication architecture as recited in claim 26 wherein said first link is replicated for each of the plurality of OSD'"'"'s and said shared I/O switch.

33. The serial communication architecture as recited in claim 26 wherein said shared I/O switch comprises a plurality of ports, at least one of said ports associated with each one of the plurality of OSD'"'"'s.

34. The serial communication architecture as recited in claim 26 wherein said second link comprises PCI Express enhanced to allow for identification of operating system domains and/or root complexes.

35. The serial communication architecture as recited in claim 26 wherein said second link is configured to support data with an embedded field for storing said header.

36. The serial communication architecture as recited in claim 35 wherein said embedded field associates a packet from one of the plurality of OSD'"'"'s with that one of the plurality of OSD'"'"'s.

37. The serial communication architecture as recited in claim 35 wherein said embedded field comprises a six bit field for associating said packets with at least 64 distinct ones of the plurality of OSD'"'"'s.

38. The serial communication architecture as recited in claim 35 wherein said embedded field associates said packets with ones of the plurality of OSD'"'"'s utilizing a plurality of bit fields.

39. The serial communication architecture as recited in claim 26 wherein said header is transmitted with said packets from said shared I/O switch to at least one of the plurality of endpoints.

40. The serial communication architecture as recited in claim 39 wherein said at least one of the plurality of endpoints utilizes said header to determine which of the plurality of OSD'"'"'s it is performing processing for.

41. The serial communication architecture as recited in claim 40 wherein each of the plurality of endpoints performs processing for at least two of the plurality of OSD'"'"'s.

42. The serial communication architecture as recited in claim 40 wherein said packets from at least two of the plurality of OSD'"'"'s reside within said at least one of the plurality of endpoints at the same time.

43. An apparatus for associating packets in a load/store serial communication fabric with PCI Express root complexes to allow the root complexes to share a PCI Express endpoint comprising an input/output (I/O) endpoint, the apparatus comprising:

a shared I/O switch, coupled to each of the root complexes, said shared I/O switch having routing control to associate the packets from each of the root complexes with the root complex they originate from by incorporating a field within the packets; and

a link between said shared I/O switch and the input/output (I/O) endpoint, wherein said link allows the packets to be transferred from said shared I/O switch to the input/output (I/O) endpoint with said field;

wherein the input/output (I/O) endpoint associates the packets with their associated root complexes by examining said field; and

wherein at least two of said plurality of root complexes do not include a dedicated network interface controller and are configured to share said endpoint.

44. The apparatus for associating packets as recited in claim 43 wherein the packets are PCI Express packets.

45. The apparatus for associating packets as recited in claim 43 wherein the root complexes comprise:

a component in a PCI Express hierarchy that connects to a host bus segment on an upstream side with one or more PCI Express links on a downstream side.

46. The apparatus for associating packets as recited in claim 43 wherein the root complexes comprise operating system domains.

47. The apparatus for associating packets as recited in claim 43 wherein the root complexes comprise processing complexes.

48. The apparatus for associating packets as recited in claim 43 wherein the input/output (I/O) endpoint comprises a shared network interface controller.

49. The apparatus for associating packets as recited in claim 43 wherein the input/output (I/O) endpoint comprises a shared network storage controller.

50. The apparatus for associating packets as recited in claim 43 wherein the load/store serial communication fabric utilizes PCI Express.

51. The apparatus for associating packets as recited in claim 43 wherein said shared I/O switch comprises:

a plurality of ports for connecting said shared I/O switch to the root complexes and to the input/output (I/O) endpoint, wherein at least one of the plurality of ports is associated with each of the root complexes, and at least one of the plurality of ports is associated with the input/output (I/O) endpoint; and

said routing control, coupled to said plurality of ports, wherein said routing control is aware of which of said plurality of ports is associated with which of the root complexes.

52. The apparatus for associating packets as recited in claim 51 wherein said routing control comprises a table lookup that associates each of the plurality of ports with the root complexes.

53. The apparatus for associating packets as recited in claim 52 wherein said field includes information from said table lookup to associate the packets with their root complexes.

54. The apparatus for associating packets as recited in claim 43 wherein said link comprises a PCI Express link enhanced to allow for identification of operating system domains and/or root complexes.

55. The apparatus for associating packets as recited in claim 43 wherein the input/output (I/O) endpoint contains packets from more than one of the root complexes at the same time.

56. The apparatus for associating packets as recited in claim 43, wherein said root complex identification identifies a particular PCI Express root complex which is a source of a received packet.

57. A method for associating packets, within a serial load/store fabric, from a plurality of PCI Express root complexes with their originating root complex, to allow the plurality of root complexes to share a PCI Express endpoint comprising an I/O endpoint, the method comprising:

providing a first link between the plurality of root complexes and a switch, the packets in the first link unaware that the root complexes are sharing the I/O endpoint;

within the switch, embedding a header in the packets to associate the packets with their originating root complex;

providing a second link between the switch and the I/O endpoint, the second link capable of communicating the packets with the embedded header between the switch and the I/O endpoint; and

at the I/O endpoint, examining the packets with the embedded header to allow the I/O endpoint to associate each of the packets with their originating root complex;

wherein at least two of said plurality of root complexes do not include a dedicated network interface controller and are configured to share the I/O endpoint.

58. The method for associating packets as recited in claim 57 wherein a first one of the root complexes is a network computer server with a Windows operating system.

59. The method for associating packets as recited in claim 58 wherein a second one of the root complexes is a network computer server with a Linux operating system.

60. The method for associating packets as recited in claim 57 wherein a first one of the root complexes comprises a processing complex.

61. The method for associating packets as recited in claim 60 wherein said processing complex comprises:

one or more processors; and

memory, coupled to said one or more processors for storing data utilized by said one or more processors.

62. The method for associating packets as recited in claim 57 wherein a first one of the root complexes comprises an operating system domain.

63. The method for associating packets as recited in claim 57 wherein the I/O endpoint is a 1 Gig Ethernet controller.

64. The method for associating packets as recited in claim 57 wherein the I/O endpoint is a 10 Gig Ethernet controller.

65. The method for associating packets as recited in claim 57 wherein the I/O endpoint is a Fiber Channel controller.

66. The method for associating packets as recited in claim 57 wherein the I/O endpoint is a serial ATA controller.

67. The method for associating packets as recited in claim 57 wherein the first link comprises a PCI Express link.

68. The method for associating packets as recited in claim 57 wherein the second link comprises a PCI Express link enhanced to allow for identification of operating system domains and/or root complexes.

69. The method for associating packets as recited in claim 57 wherein said step of embedding comprises:

determining which port within the switch a packet is received from;

associating the packet with that port; and

assigning a header number to the packet associating the packet with a bus hierarchy for the port which received the packet.

70. The method for associating packets as recited in claim 69 wherein said step of associating the packet comprises:

performing a table lookup from a table which correlates that port with its associated root complex.

71. The method for associating packets as recited in claim 57 wherein a plurality of packets with embedded headers from a plurality of root complexes reside in the I/O endpoint at the same time.

72. The method for associating packets as recited in claim 57, wherein said root complex identification identifies a particular PCI Express root complex which is a source of a received packet.

73. A method for partitioning PCI Express I/O endpoint devices among a plurality of processing complexes of a PCI Express root complex, the partitioning performed within the load/store domain of each of the processing complexes, the method comprising:

providing a switch between the I/O devices and the plurality of processing complexes, the switch utilizing a first load/store fabric between the plurality of processing complexes and the switch, and a second load/store fabric between the switch and the I/O devices;

mapping each of the plurality of processing complexes to one or more of the I/O devices; and

transferring packets between the plurality of processing complexes to the I/O devices based upon said mapping;

wherein at least one of the I/O devices is mapped to only one of the plurality of processing complexes; and

wherein at least one of the I/O devices is mapped to two or more of the plurality of processing complexes;

wherein at least two of said plurality of processing complexes do not include a dedicated network interface controller and are configured to share the I/O devices.

74. The method for partitioning I/O devices as recited in claim 73 wherein the switch is a shared I/O switch.

75. The method for partitioning I/O devices as recited in claim 73 wherein the first load/store fabric comprises PCI Express.

76. The method for partitioning I/O devices as recited in claim 73 wherein the second load/store fabric comprises PCI Express enhanced to allow for identification of operating system domains and/or root complexes.

77. method for partitioning I/O devices as recited in claim 73, wherein said root complex identification identifies a particular PCI Express root complex which is a source of a received packet.