Fibre channel controller shareable by a plurality of operating system domains within a load-store architecture

• US 7,617,333 B2
• Filed: 01/27/2005
• Issued: 11/10/2009
• Est. Priority Date: 01/21/2003
• Status: Active Grant

• Current Assignee: Mellanox Technologies Limited (NVIDIA Corporation)
• Original Assignee: NextIO Incorporated
• Inventors: Pettey, Christopher J.
• Primary Examiner(s): Sorrell, Eron J
• Application Number:
  US11/046,564
• Publication Number:
  US 20050172047A1
• Time in Patent Office: 1,748 Days
• Field of Search: 710/5
• US Class Current: 710/5
• CPC Class Codes: G06F 13/12 using hardware independent of the central processor, e.g. channel or peripheral processor

Abstract

A Fibre Channel (FC) controller shareable by a plurality of operating system domains (OSDs) within a load-store architecture is disclosed. The controller has a FC port that obtains a plurality of FC port identifiers for association with respective ones of the OSDs. A load-store bus interface is the target of a load-store transaction on a load-store bus from each OSD. The load-store transaction includes a command to perform an I/O operation with a remote FC device. Association logic populates an S_ID field of a FC frame with the FC port identifier associated with the respective OSD that initiated the command. The FC port transmits the FC frame on the FC port to the remote FC device. In one embodiment, the controller interfaces to an Advanced Switching fabric to receive packets encapsulating load-store transactions from the OSDs. Each packet includes an identifier identifying the OSD initiating the transaction.

Claims

1. A Fibre Channel (FC) controller shareable by a plurality of operating system domains (OSDs) within a load-store architecture, comprising:

a FC port, configured to obtain a plurality of FC port identifiers for association with respective ones of the plurality of OSDs;

a load-store bus interface, for coupling to a load-store bus, for being a target of at least one load-store transaction on said load-store bus from each of the plurality of OSDs, wherein each of the plurality of OSDs comprises a system load-store memory map that is associated with a respective processing complex, wherein the shared FC controller is mapped by each respective processing complex into a location within its associated system load-store memory map, wherein each respective processing complex executes at least one load-store instruction that generates said at least one load-store transaction targeted at the location of the shared FC controller within the associated system load-store memory map, wherein said at least one load-store transaction includes a command to perform an I/O operation with a remote FC device via said FC port;

association logic, coupled to said load-store bus interface and said FC port, configured to populate a Source_Identifier (S_ID) field of at least one FC frame with one of said plurality of FC port identifiers associated with said respective one of the plurality of OSDs that initiated said command, wherein said FC port is configured to transmit said at least one FC frame on said FC port to said remote FC device; and

a plurality of control/status register (CSR) banks, coupled to said load-store bus interface, wherein said load-store bus interface associates a different one of said plurality of CSR banks with each of the plurality of OSDs for being said target of said at least one load-store transaction including said command to perform said I/O operation.

2. The shareable FC controller as recited in claim 1, wherein said association logic comprises a mapping table for mapping each of the plurality of OSDs to said respective one of said plurality of FC port identifiers.

3. The shareable FC controller as recited in claim 1, further comprising:

a plurality of data structures, coupled to said association logic, each for storing entries specifying outstanding operations by said FC port for corresponding ones of the plurality of OSDs.

4. The shareable FC controller as recited in claim 1, wherein said at least one load-store transaction includes an OSD ID identifying said one of the plurality of OSDs that initiated said transaction, wherein said association logic associates said OSD ID with said respective one of said plurality of FC port identifiers.

5. The shareable FC controller as recited in claim 1, wherein each of said at least one load-store transactions further includes a memory address for specifying a control/status register in said respective one of said plurality of CSR banks with which said load-store bus interface associates said transaction.

6. The shareable FC controller as recited in claim 5, wherein said memory address is a source or destination memory address of a load or store instruction, respectively, executed by a processor of said one of the plurality of OSDs.

7. The shareable FC controller as recited in claim 5, wherein the shareable FC controller is selected as a destination of said load and store transactions based on said memory address.

8. The shareable FC controller as recited in claim 7, wherein the shareable FC controller is selected as a destination of said load and store transactions based on said memory address rather than based on a device identifier in said load and store transactions specifying the shareable FC controller as said destination.

9. The shareable FC controller as recited in claim 1, wherein each of said load and store transactions further includes an I/O address for specifying a control/status register in said respective one of said plurality of CSR banks to which said load-store bus interface directs said transaction.

10. The shareable FC controller as recited in claim 9, wherein said I/O address is a source or destination I/O address of a load or store instruction, respectively, executed by a processor of said one of the plurality of OSDs.

11. The shareable FC controller as recited in claim 9, wherein the shareable FC controller is selected as a destination of said load and store transactions based on said I/O address.

12. The shareable FC controller as recited in claim 11, wherein the shareable FC controller is selected as a destination of said load and store transactions based on said I/O address rather than based on a device identifier in said load and store transactions specifying the shareable FC controller as said destination.

13. The shareable FC controller as recited in claim 1, wherein each of the plurality of OSDs comprises a system load-store memory map, wherein each of said system load-store memory maps comprises a location of said respective one of said plurality of CSR banks.

14. The shareable FC controller as recited in claim 13, wherein said location of said respective one of said plurality of CSR banks is configurable by each of said plurality of OSDs within said system load-store memory map.

15. The shareable FC controller as recited in claim 13, wherein said location of said respective one of said plurality of CSR banks may be different for each of said plurality of OSD system load-store memory maps.

16. The shareable FC controller as recited in claim 13, wherein said location of said respective one of said plurality of CSR banks in said system load-store memory map does not survive a reset of the respective one of the plurality of OSDs.

17. The shareable FC controller as recited in claim 1, wherein said load-store bus interface associates said different one of said plurality of CSR banks with each of the plurality of OSDs for being said target of said at least one load-store transaction based on an OSD ID identifying said one of the plurality of OSDs that initiated said transaction.

18. The shareable FC controller as recited in claim 1, wherein said load-store bus interface is further configured to generate interrupts on said load-store bus to the plurality of OSDs.

19. The shareable FC controller as recited in claim 18, wherein the sharable FC controller generates said interrupts in response to completion of said command to perform said I/O operation with said remote FC devices.

20. The shareable FC controller as recited in claim 18, wherein each of said interrupts include an OSD identifier for identifying the one of the plurality of OSDs that commanded the shared FC controller to perform said I/O operation.

21. The shareable FC controller as recited in claim 18, wherein said interrupts comprise a message transmitted on said load-store bus, wherein said message includes an OSD identifier.

22. The shareable FC controller as recited in claim 21, wherein said message comprises a PCI message signaled interrupt (MSI) modified to include said OSD identifier.

23. The shareable FC controller as recited in claim 22, wherein said message comprises a PCI Express message signaled interrupt (MSI) packet modified to include said OSD identifier.

24. The shareable FC controller as recited in claim 1, wherein said a plurality of control/status register (CSR) banks are configured to concurrently receive said load-store transactions from the plurality of OSDs.

25. The shareable FC controller as recited in claim 1, wherein said at least one load-store transaction comprises a PCI Express request packet modified to include an OSD identifier.

26. The shareable FC controller as recited in claim 1, wherein said load-store bus interface is configured for coupling to a serial load-store bus.

27. The shareable FC controller as recited in claim 1, wherein the plurality of OSDs comprises a plurality of processors coupled to a processor bus, wherein said load-store bus interface is configured for coupling to said processor bus.

28. The shareable FC controller as recited in claim 27, wherein each of the plurality of processors provides a processor ID on said processor bus with each of said load-store transactions, wherein an OSD ID comprises said processor ID.

29. The shareable FC controller as recited in claim 1, wherein said FC port appears as a plurality of FC end nodes to a FC fabric coupled thereto, by performing a plurality of logins to said FC fabric, wherein said plurality of FC end nodes correspond to the plurality of OSDs.

30. The shareable FC controller as recited in claim 1, wherein said FC port comprises a FC Node Port (N_Port) linked to a FC Fabric_Port (F_Port) supporting a FC Multiple N_Port_ID Assignment capability, wherein said FC N_Port obtains from said F_Port a respective FC N_Port identifier for each of the plurality of OSDs.

31. The shareable FC controller as recited in claim 1, wherein the controller is configured to determine whether a FC F_Port to which said FC port is linked supports a FC Multiple N_Port_ID Assignment capability, to obtain a distinct FC N_Port_ID for each of the plurality of OSDs if the FC F_Port supports the FC Multiple N_Port_ID Assignment capability, and to obtain a distinct FC NL_Port_ID for each of the plurality of OSDs if the FC F_Port does not support the FC Multiple N_Port_ID Assignment capability.

32. The shareable FC controller as recited in claim 1, wherein the controller is configured to determine that one of the plurality of OSDs has been reset, and to disclaim the respective one of the plurality of FC port identifiers associated with the one of the plurality of OSDs that has been reset, in response thereto.

33. A Fibre Channel (FC) controller shareable by a plurality of operating system domains (OSDs) within a load-store architecture, comprising:

a plurality of control/status register (CSR) banks, wherein a respective one of said plurality of CSR banks is used by each of the plurality of OSDs to request the shared FC controller to perform I/O operations with remote FC devices;

a fabric interface, coupled to said plurality of CSR banks, for coupling to a switched fabric, configured to receive packets encapsulating load and store transactions from each of the plurality of OSDs, wherein each of the plurality of OSDs comprises a system load-store memory map that is associated with a respective processing complex, wherein the shared FC controller is mapped by each respective processing complex into a location within its associated system load-store memory map, each of said packets including an OSD identifier for identifying one of the plurality of OSDs that initiated said transaction, wherein said fabric interface directs each of said load and store transactions to said respective one of said plurality of CSR banks based on said OSD identifier; and

a FC port, coupled to said fabric interface, configured to obtain a distinct FC port identifier for each of the plurality of OSDs, and to transceive FC frames with said remote FC devices using said distinct FC port identifier for each of the plurality of OSDs in response to said I/O operation requests.

34. The shareable FC controller as recited in claim 33, wherein said switched fabric comprises an Advanced Switching (AS) fabric.

35. A method for sharing a Fibre Channel (FC) controller by a plurality of operating system domains (OSDs) within a load-store architecture, comprising:

obtaining a distinct FC port identifier for each of the plurality of OSDs, wherein each of the plurality of OSDs comprises a system load-store memory map that is associated with a respective processing complex, wherein the shared FC controller is mapped by each respective processing complex into a location within its associated system load-store memory map;

requesting the controller to perform I/O operations with remote FC devices, wherein said requesting comprises the plurality of OSDs executing load-store instructions addressed to a programming interface of the controller, wherein the programming interface comprises a distinct control/status register (CSR) bank for each of the plurality of OSDs;

receiving corresponding load-store transactions in response to said executing, and selecting as a target of each of the load-store transactions the distinct CSR bank for the one of the plurality of OSDs that executed the corresponding load-store instruction; and

transceiving FC frames with the remote FC devices using the distinct FC port identifier for each of the plurality of OSDs to perform the I/O operation requests.

36. The method as recited in claim 35, further comprising:

configuring the programming interface into a location within a system load-store memory map of each of the plurality of OSDs, prior to said requesting.

37. The method as recited in claim 36, further comprising:

decoding a memory address included in each of the load-store transactions;

determining whether the memory address is within the system load-store memory map of the one of the plurality of OSDs that executed the corresponding load-store instruction; and

switching the load-store transaction to the controller if the memory address is within the system load-store memory map of the one of the plurality of OSDs that executed the corresponding load-store instruction.

38. The method as recited in claim 37, wherein the memory address specifies a register in the distinct CSR bank selected as a target of the load-store transaction.

39. The method as recited in claim 36, further comprising:

decoding an I/O address included in each of the load-store transactions;

determining whether the I/O address is within the system load-store memory map of the one of the plurality of OSDs that executed the corresponding load-store instruction; and

switching the load-store transaction to the controller if the I/O address is within the system load-store memory map of the one of the plurality of OSDs that executed the corresponding load-store instruction.

40. The method as recited in claim 39, wherein the I/O address specifies a register in the distinct CSR bank selected as a target of the load-store transaction.

41. The method as recited in claim 35, wherein said obtaining is performed by the controller.

42. The method as recited in claim 41, wherein said obtaining is performed by the controller in response to each of the plurality of OSDs requesting the controller to obtain a port identifier.

43. The method as recited in claim 42, wherein said each of the plurality of OSDs requesting the controller to obtain said port identifier comprises each of the plurality of OSDs executing load-store instructions addressed to the programming interface of the controller.

44. The method as recited in claim 35, wherein said requesting comprises two or more of the plurality of OSDs concurrently requesting the controller to perform the I/O operations.

45. The method as recited in claim 35, wherein each of the load-store transactions includes a memory address specifying a register in the distinct CSR bank selected as a target of the load-store transaction.

46. The method as recited in claim 35, wherein each of the load-store transactions includes an I/O address specifying a register in the distinct CSR bank selected as a target of the load-store transaction.

47. The method as recited in claim 35, wherein said obtaining a distinct FC port identifier for each of the plurality of OSDs comprises obtaining said distinct FC N_Port identifier for each of the plurality of OSDs.

48. The method as recited in claim 35, wherein said obtaining a distinct FC port identifier for each of the plurality of OSDs comprises obtaining said distinct FC NL_Port identifier for each of the plurality of OSDs.

49. The method as recited in claim 35, wherein said receiving corresponding load-store transactions comprises receiving corresponding PCI Express transactions.

50. The method as recited in claim 35, wherein said receiving corresponding load-store transactions comprises receiving corresponding PCI Express transactions including an identifier identifying the one of the plurality of OSDs that executed the corresponding load-store instruction.

51. The method as recited in claim 35, wherein said receiving corresponding load-store transactions comprises receiving corresponding PCI Express+ transactions.

52. The method as recited in claim 35, wherein said receiving corresponding load-store transactions comprises receiving corresponding PCI-family local bus transactions including an identifier identifying the one of the plurality of OSDs that executed the corresponding load-store instruction.

53. The method as recited in claim 35, wherein said receiving corresponding load-store transactions comprises receiving corresponding Advanced Switching (AS) transactions.

54. The method as recited in claim 35, wherein said transceiving comprises:

receiving the FC frames from the remote FC devices, wherein each of the FC frames includes a destination port identifier; and

determining whether the destination port identifier matches any of the distinct FC port identifiers for any of the plurality of OSDs.

55. The method as recited in claim 54, wherein said transceiving further comprises:

dropping the FC frame if the destination port identifier does not match any of the distinct FC port identifiers for any of the plurality of OSDs.

56. The method as recited in claim 54, wherein said transceiving further comprises:

associating a FC frame with one of the plurality of OSDs based on which of the distinct FC port identifiers the frame destination port identifier matches.

57. The method as recited in claim 56, wherein said associating comprises:

looking up the FC frame destination port identifier in an association table to find the one of the plurality of OSDs whose distinct FC port identifiers matches the frame destination port identifier.

58. The method as recited in claim 56, wherein said transceiving further comprises:

transferring at least a portion of the FC frame to the one of the plurality of OSDs associated with the FC frame, after said associating.

59. The method as recited in claim 35, wherein said transceiving comprises:

storing in a FC source identifier field of one of the FC frames the distinct FC port identifier for the one of the plurality of OSDs that performed said requesting the controller to perform the I/O operation; and

transmitting the FC frame to one of the FC remote devices, after said storing.

60. The method as recited in claim 35, wherein said obtaining the distinct FC port ID for each of the plurality of OSDs comprises:

determining whether a FC Fabric_Port (F_Port) to which the controller is linked supports a FC Multiple N_Port identifier (N_Port_ID) Assignment capability;

obtaining a distinct FC N_Port_ID for each of the plurality of OSDs if the FC F_Port supports the FC Multiple N_Port_ID Assignment capability; and

obtaining a distinct FC NL_Port identifier (NL_Port_ID) for each of the plurality of OSDs if the FC F_Port does not support the FC Multiple N_Port_ID Assignment capability.

61. The method as recited in claim 35, wherein said obtaining the distinct FC port identifier for each of the plurality of OSDs comprises:

determining whether a FC Fabric_Port (F_Port) to which the controller is linked supports a FC Multiple N_Port identifier (N_Port_ID) Assignment capability;

obtaining a distinct FC N_Port_ID for each of the plurality of OSDs if the FC F_Port supports the FC Multiple N_Port_ID Assignment capability; and

obtaining a distinct FC NL_Port identifier (NL_Port_ID) for each of the plurality of OSDs if the FC F_Port does not support the FC Multiple N_Port_ID Assignment capability.

62. The method as recited in claim 35, further comprising:

determining that one of the plurality of OSDs has been reset; and

disclaiming the distinct FC port identifier for the one of the plurality of OSDs that has been reset, in response to said determining.

63. The method as recited in claim 62, wherein said disclaiming the distinct FC port identifier for the one of the plurality of OSDs that has been reset comprises logging out of a FC fabric to which the controller is linked for the distinct FC port identifier for the one of the plurality of OSDs that has been reset.

64. The method as recited in claim 63, wherein the controller no longer performs said transceiving FC frames with the remote FC devices using the distinct FC port identifier for the one of the plurality of OSDs that has been reset, in response to said determining.

65. The method as recited in claim 62, wherein said disclaiming the distinct FC port identifier for the one of the plurality of OSDs that has been reset comprises removing the distinct FC port identifier for the one of the plurality of OSDs that has been reset from a mapping table of the controller, wherein said mapping table is for mapping the distinct FC port identifiers to the plurality of OSDs.

66. The method as recited in claim 62, wherein said determining that one of the plurality of OSDs has been reset comprises determining that a device driver of the one of the plurality of OSDs has been reset.