Method and apparatus for shared I/O in a load/store fabric

• US 7,706,372 B2
• Filed: 04/19/2006
• Issued: 04/27/2010
• Est. Priority Date: 01/21/2003
• Status: Active Grant

• Current Assignee: Mellanox Technologies Limited (NVIDIA Corporation)
• Original Assignee: NextIO Incorporated
• Inventors: Pekkala, Richard E., Pettey, Christopher J., Pagan, Annette, Khan, Asif, Utley, Robert Haskell
• Primary Examiner(s): Nguyen, Brian D
• Application Number:
  US11/379,264
• Publication Number:
  US 20070025354A1
• Time in Patent Office: 1,469 Days
• Field of Search: 370/389, 370/392, 370/401, 370/471, 370/465, 370/466, 370/467, 370/419, 370/463, 370/428
• US Class Current: 370/392
• CPC Class Codes: G06F 13/12 using hardware independent of the central processor, e.g. channel or peripheral processor

Abstract

An apparatus and method is provided for allowing I/O devices to be shared and/or partitioned among a plurality of processing complexes within the load/store fabric of each of the processing complexes without requiring modification to the operating system or driver software of the processing complexes. The apparatus and method includes a switch for selectively coupling each of the processing complexes to one or more shared I/O devices. The apparatus and method further includes placing information within packets transmitted between the switch and the I/O devices to identify which of the processing complexes the packets are associated with. The invention further includes an apparatus and method within the shared I/O devices to allow the shared I/O devices to service each of the processing complexes independently.

Claims

1. An apparatus for transferring data in a load/store fabric between a plurality of root complexes and a shared input/output (I/O) endpoint, the apparatus comprising:

an interface configured to receive a packet from at least one of the plurality root complexes, the packet including a header field for identifying the shared I/O endpoint for which the packet is destined, said interface coupled to the plurality of root complexes, wherein in lieu of including a controller within each of the root complexes for transferring data to the shared I/O endpoint, the plurality of root complexes are coupled to said interface to share the endpoint; and

circuitry configured to;

add an operating system domain (OSD) header field to the packet for identifying which one of the plurality of root complexes is associated with the packet; and

transfer the packet with the added OSD header field to the shared endpoint.

2. The apparatus as recited in claim 1 wherein each of said plurality of root complexes comprises an operating system domain (OSD).

3. The apparatus as recited in claim 2 wherein said operating system domain comprises:

a processing complex; and

a memory, coupled to said processing complex, for storing data utilized by said processing complex.

4. The apparatus as recited in claim 1 wherein at least one of said plurality of root complexes comprises a plurality of operating system domains.

5. The apparatus as recited in claim 4 wherein said plurality of operating system domains comprise:

a plurality of processing complexes, wherein each of said plurality of processing complexes are coupled to a memory for storing data utilized by each of said processing complexes.

6. The apparatus of claim 1 wherein the plurality of root complexes are configured to convey data according to a PCI Express protocol, and wherein said interface is configured to modify the conveyed data to include an operating system domain (OSD) identifier for identifying which one of the plurality of root complexes is associated with the packet.

7. The apparatus of claim 1 wherein the data comprises information which is desired to be transferred from the shared I/O endpoint to one of said plurality of root complexes.

8. The apparatus of claim 1 wherein the load/store fabric is hardware, software, or a combination of hardware and software that moves the data from one of said plurality of root complexes to the shared I/O endpoint.

9. The apparatus of claim 1 wherein the shared input/output (I/O) endpoint comprises a network interface controller (NIC).

10. The apparatus of claim 9 wherein said network interface controller is an Ethernet controller.

11. The apparatus of claim 1 wherein the shared input/output (I/O) endpoint comprises a Fiber Channel controller.

12. The apparatus of claim 1 wherein the shared input/output (I/O) endpoint comprises one of the following:

a Fibre Channel controller, a SCSI controller, or a shared RAID controller.

13. The apparatus of claim 1 wherein the load/store fabric utilizes PCI Express.

14. The apparatus of claim 13 wherein said header field comprises:

a transaction layer packet (TLP) header; and

an optional data payload.

15. The apparatus of claim 14 wherein said TLP header comprises a plurality of fields, including a field identifying the shared I/O endpoint for which the packet is destined.

16. The apparatus of claim 1 wherein the circuitry is further configured to add an operating system domain (OSD) header field comprising an OSD Number field for identifying which one of the plurality of root complexes is associated with the packet.

17. The apparatus of claim 16 wherein said OSD Number field is global throughout the load/store fabric.

18. The apparatus of claim 16 wherein said OSD Number field is local to each link in the load/store fabric.

19. The apparatus of claim 16 wherein said OSD Number field specifies of the plurality of root complexes from which the packet originated.

20. The apparatus of claim 16 wherein said OSD Number field specifies one of a plurality of OS Domains from which the packet originated.

21. The apparatus of claim 20 wherein said OSD Number field is a six (6) bit field for designating up to 64 distinct OS Domains within a link in the load/store fabric.

22. The apparatus of claim 16 wherein said OSD Number field is a six (6) bit field for designating up to 64 distinct root complexes.

23. The apparatus of claim 1 wherein said load/store fabric is a serial load/store fabric.

24. An apparatus for transferring data in a load/store fabric between a plurality of root complexes and a shared input/output (I/O) endpoint, the apparatus comprising:

an interface configured to;

receive a PCI Express packet; and

be coupled to the root complexes, wherein in lieu of including a controller within each of the root complexes that is configured to communicate using a first protocol, each of the root complexes is coupled to share the endpoint via said interface in order to communicate using the first protocol; and

circuitry configured to;

add a plurality of bits to said packet, wherein said plurality of bits identify an operating system domain from which the PCI Express packet originated; and

transfer said packet with the added plurality of bits to the shared I/O endpoint.

25. The apparatus as recited in claim 24 wherein said operating system domain comprises a root complex.

26. The apparatus as recited in claim 24 wherein said operating system domain comprises:

a processing complex; and

a memory, coupled to said processing complex for storing data utilized by said processing complex.

27. The apparatus as recited in claim 24 wherein said operating system domain comprises a port within a shared I/O switch to which a root complex is coupled.

28. The apparatus as recited in claim 24 wherein said shared I/O endpoint comprises at least one of the following:

a network interface controller, a Fibre Channel controller, a SCSI controller, or a RAID controller.

29. The apparatus as recited in claim 24 wherein said interface is configured to receive a PCI Express packet and add a plurality of bits for identifying which one of the plurality of root complexes is associated with the packet.

30. The apparatus as recited in claim 24 wherein said endpoint comprises a plurality of control register sets;

and a plurality of direct memory access (DMA) engines.

31. A method for identifying a root complex of a plurality of root complexes for a packet within a load/store fabric to allow sharing of an input/output (I/O) endpoint by the plurality of root complexes, the method comprising:

providing an architecture for the packet;

providing an interface for use in processing packets transferred between the root complexes and said shared I/O endpoint;

coupling the interface to the root complexes, wherein in lieu of including a controller within each of the root complexes for communicating to the I/O endpoint, each of the root complexes is coupled to the interface to share the I/O endpoint;

providing, by a circuitry, a field for inclusion in the packet to identify the root complex for the packet; and

;

wherein the shared input/output (I/O) endpoint utilizes the field provided in said step of providing a field to identify the root complex for the packet.

32. The method for identifying the root complex as recited in claim 31 wherein the root complex comprises a network computer server.

33. The method for identifying the root complex as recited in claim 32 wherein the network computer server is a blade server.

34. The method for identifying the root complex as recited in claim 31 wherein the root complex comprises an operating system domain.

35. The method for identifying the root complex as recited in claim 31 wherein the architecture in said step of providing an architecture conforms to a PCI Express System Architecture, and wherein the interface is configured to receive a PCI Express packet and add an operating system domain (OSD) identifier for identifying which one of the plurality of root complexes is associated with the packet.

36. The method for identifying the root complex as recited in claim 31 wherein the input/output (I/O) endpoint comprises a network interface controller.

37. The method for identifying the root complex as recited in claim 36 wherein the network interface controller is an Ethernet controller.

38. The method for identifying the root complex as recited in claim 31 wherein the input/output (I/O) endpoints comprises one of the following:

a Fibre Channel controller, a SCSI controller, or a RAID controller.

39. An apparatus for transferring data in a load/store fabric between a plurality of root complexes and a shared input/output (I/O) endpoint, the apparatus comprising:

an interface configured to receive a PCI Express packet; and

circuitry configured to;

add a plurality of bits to said packet, wherein said plurality of bits identify an operating system domain from which the PCI Express packet originated; and

transfer said packet with the added plurality of bits to the shared I/O endpoint;

wherein an OS Domain header is attached to a PCI Express Packet to form a PCI Express+Packet, wherein said PCI Express+Packet is sent to an endpoint device, wherein said endpoint device is a shared I/O network interface controller;

wherein said shared I/O network interface controller, upon receipt of said PCI Express+Packet;

examines the OS Domain header to determine which operating system domain the PCI Express Packet originated from;

after determining which operating system domain the PCI Express Packet originated from, applies controller resources that are associated with that operating system domain;

wherein said controller resources comprise;

a plurality of controller register sets; and

a plurality of direct memory access (DMA) engines.