Method and apparatus for a shared I/O network interface controller

• US 20050268137A1
• Filed: 02/03/2005
• Published: 12/01/2005
• Est. Priority Date: 01/21/2003
• Status: Active Grant

• Current Assignee: Mellanox Technologies Limited (NVIDIA Corporation)
• Original Assignee: NextIO Incorporated
• Inventors: Pettey, Christopher J.
• Granted Patent: US 8,032,659 B2
• Time in Patent Office: Days
• US Class Current: 713/400
• CPC Class Codes: H04L 12/4641 Virtual LANs, VLANs, e.g. virtual private networks [VPN] LAN interconnection over a bridge based backbone H04L12/462; encapsulation techniques H04L12/4633; routing of packets H04L45/00; packet switches H04L49/00; virtual private networks for security H04L63/0272

Abstract

A network interface controller is provided which is shareable by a plurality of operating system domains within their load-store architecture. The controller includes local resources for each of the plurality of operating system domains which allow them to communicate uniquely with the controller, and global resources which allow the controller to communicate with each of the plurality of operating systems, as well as with a network fabric. A method and apparatus is provided for distinguishing between the local and global resources, for purposes of reset and configuration. Resets received from ones of the plurality of operating system domains are treated as hot-pluggable events, and utilize a reset DLLP packet to the controller. The controller allows the reset DLLP to reset only those local resources which are associated with the operating system transmitting the reset, while preserving the other local resources. Registration logic is provided to allow one of the operating system domains to register as management (or reset) master, for configuration and reset of global resources. Timer logic is provided to insure that if global configuration fails, the management master is unregistered. Status/Messaging logic is provided to notice other operating system domains of the status of global configuration and/or reset.

Claims

1. A shareable network interface controller to be shared within the load-store architecture of each of a plurality of operating system domains, comprising:

a bus interface, for coupling the shareable network interface controller to each of the plurality of operating system domains; and

registration logic, coupled to said bus interface, for registering one of the plurality of operating system domains as a master of the shareable network interface controller.

2. The shareable network interface controller as recited in claim 1 wherein the shareable network interface controller comprises an Ethernet controller for supporting the plurality of operating system domains.

3. The shareable network interface controller as recited in claim 2 wherein the Ethernet controller further comprises:

a first port for coupling the controller to an Ethernet fabric at a first connection speed; and

a second port for coupling the controller to an Ethernet fabric at a second connection speed.

4. The shareable network interface controller as recited in claim 1 wherein each of the plurality of operating system domains comprises a processing complex executing at least one operating system.

5. The shareable network interface controller as recited in claim 4 wherein said processing complex comprises at least one microprocessor coupled to a memory.

6. The shareable network interface controller as recited in claim 5 wherein the shareable network interface controller is mapped within a region of said memory.

7. The shareable network interface controller as recited in claim 5 wherein the shareable network interface controller is mapped within a region of said memory for each of the plurality of operating system domains.

8. The shareable network interface controller as recited in claim 1 wherein the load-store architecture of each of the plurality of operating system domains references memory space that is accessed through load and store instructions executing on each of the plurality of operating system domains.

9. The shareable network interface controller as recited in claim 1 wherein the bus interface couples the shareable network interface controller to each of the plurality of operating system domains via a load-store link.

10. The shareable network interface controller as recited in claim 9 wherein said load-store link passes loads and stores to the shareable network interface controller, along with OSD information indicating an originating one of the plurality of operating system domains for said loads and stores.

11. The shareable network interface controller as recited in claim 10 wherein said OSD information is embedded within packets transferred on said load-store link.

12. The shareable network interface controller as recited in claim 9 wherein said load-store link comprises PCI-Express.

13. The shareable network interface controller as recited in claim 12 wherein OSD information indicating one of the plurality of operating system domains is embedded in packets on said PCI-Express.

14. The shareable network interface controller as recited in claim 13 wherein said OSD information is embedded in said packets utilizing reserved bits within said PCI-Express.

15. The shareable network interface controller as recited in claim 13 wherein said OSD information is embedded in said packets utilizing undefined bits within said PCI-Express.

16. The shareable network interface controller as recited in claim 13 wherein said OSD information is embedded in said packets utilizing functional unit bits within said PCI-Express.

17. The shareable network interface controller as recited in claim 13 wherein said OSD information is embedded in said packets utilizing an additional OSD header embedded within said PCI-Express.

18. The shareable network interface controller as recited in claim 13 wherein said OSD information is embedded in said packets utilizing address bits within said PCI-Express.

19. The shareable network interface controller as recited in claim 1 wherein said bus interface couples the shareable network interface controller to each of the plurality of operating system domains via a packet based serial link which is within the load-store architecture of each of the plurality of operating system domains.

20. The shareable network interface controller as recited in claim 1 wherein said registration logic comprises a control status register.

21. The shareable network interface controller as recited in claim 1 wherein said registration logic comprises logic for confirming whether a key has been correctly provided by the one of the plurality of operating system domains.

22. The shareable network interface controller as recited in claim 1 wherein said registration logic comprises means for assuring that only one of the plurality of operating system domains is registered as a master at any one time.

23. The shareable network interface controller as recited in claim 1 wherein said one of the plurality of operating system domains which is registered as a master is allowed to globally reset the shareable network interface controller.

24. The shareable network interface controller as recited in claim 1 wherein said one of the plurality of operating system domains which is registered as a master is allowed to reload firmware within the shareable network interface controller.

25. The shareable network interface controller as recited in claim 1 wherein said one of the plurality of operating system domains which is registered as a master is allowed to change the physical link speed of a port attached to network fabric for the shareable network interface controller.

26. The shareable network interface controller as recited in claim 1 wherein said one of the plurality of operating system domains which is registered as a master is allowed to repartition global resources within the shareable network interface controller.

27. The shareable network interface controller as recited in claim 1 wherein said one of the plurality of operating system domains which is registered as a master is allowed to reset global resources within the shareable network interface controller.

28. The shareable network interface controller as recited in claim 1 further comprising:

timer logic, coupled to said registration logic, for providing a timeout signal if said master does not complete a management function within an expected completion time.

29. The shareable network interface controller as recited in claim 28 wherein said timeout signal is provided to said registration logic.

30. The shareable network interface controller as recited in claim 29 wherein upon receipt of said timeout signal by said registration logic, said registration logic un-registers said master, as master.

31. The shareable network interface controller as recited in claim 1 further comprising:

status/messaging logic, coupled to said registration logic, for informing other ones of the plurality of operating system domains when said one of the plurality of operating system domains becomes registered as a master.

32. The shareable network interface controller as recited in claim 31 wherein said status/messaging logic comprises a plurality of control status registers.

33. The shareable network interface controller as recited in claim 32 wherein a first one of said plurality of control status registers is readable by a first one of the plurality of operating system domains, and a second one of said plurality of control status registers is readable by a second one of the plurality of operating system domains.

34. The shareable network interface controller as recited in claim 31 wherein said master provides a status update to said status/messaging logic during performance of a management function.

35. The shareable network interface controller as recited in claim 34 wherein said status update establishes progress of the master in performance of said management function.

36. The shareable network interface controller as recited in claim 35 wherein said status update is readable by other ones of the plurality of operating system domains.

37. The shareable network interface controller as recited in claim 1 further comprising global resources which are only configurable by said master.

38. The shareable network interface controller as recited in claim 1 further comprising global resources which are only reset-able by said master.

39. The shareable network interface controller as recited in claim 38 further comprising a plurality of local resources, each of the plurality associated with at least one of the plurality of operating system domains.

40. The shareable network interface controller as recited in claim 39 wherein said plurality of local resources are reset-able by their associated at least one of the plurality of operating system domains.

41. The shareable network interface controller as recited in claim 1 further comprising:

OSD PCI Config logic, coupled to said bus interface, which is only reset-able by said master.

42. The shareable network interface controller as recited in claim 41 wherein said OSD PCI Config logic comprises a plurality of entries which map each one of the plurality of operating system domains to the shareable network interface controller.

43. The shareable network interface controller as recited in claim 41 wherein said OSD PCI Config logic is not reset by said master when it performs a management function on the shareable network interface controller.

44. A controller, shared by a plurality of operating system domains, accessed by load-store instructions which directly address the controller, comprising:

a bus interface, for coupling the controller to a load-store link which communicates with each of the plurality of operating system domains;

OSD PCI Config logic, coupled to said bus interface, which associates the controller with each of the plurality of operating system domains;

a plurality of local resources, each of the plurality associated with a different one of the plurality of operating system domains;

global resources, utilized by the controller to support all of the plurality of operating system domains; and

registration logic, for registering one of the plurality of operating system domains as a master;

wherein said master is responsible for performing management functions on said global resources.

45. The controller as recited in claim 44 wherein said load-store link comprises a PCI-Express link with OSD information embedded in packets transferred over said link.

46. The controller as recited in claim 44 wherein each of said plurality of local resources comprise control status registers to allow associated ones of the plurality of operating system domains to communicate with the controller.

47. The controller as recited in claim 44 wherein said registration logic comprises a control status register which is writable only by only one OSD at a time to allow said OSD to become master.

48. The controller as recited in claim 44 wherein said management functions comprise:

configuration of link speed for an Ethernet port; and

re-flashing of an EEPROM which holds firmware for the controller.

49. The controller as recited in claim 44 further comprising:

timer logic, coupled to said registration logic, for providing a timeout to the controller if said master does not complete performing its management functions within a specified period of time; and

status/messaging logic, coupled to said registration logic, for communicating status of said management functions to other ones of the plurality of operating system domains.

50. The controller as recited in claim 44 wherein said master communicates with said registration logic if it wishes to continue to maintain a lock on global registration.

51. The controller as recited in claim 50 wherein said master writes to a control status register within said registration logic, and updates said control status register if it wishes to continue to stay registered as master.

52. The controller as recited in claim 44 wherein said registering comprises verifying that said master is authorized to perform said management functions on said global resources.

53. The controller as recited in claim 50 wherein said verifying utilizes a predefined key which must be written to the controller by one of the plurality of operating system domains.

54. A network interface controller which is shareable by a plurality of operating system domains by utilizing load-store instructions within their architecture, the controller comprising:

a plurality of local resources, each of said plurality of local resources associated with a different one of the plurality of operating system domains;

global resources, utilized by the controller for processing of the load-store instructions for each of the plurality of operating system domains; and

a reset from one of the plurality of operating system domains which causes a reset of its associated local resource;

wherein said reset does not cause other ones of the plurality of local resources to be reset.

55. The network interface controller as recited in claim 54 wherein each of said plurality of local resources comprise:

control registers for allowing their associated one of the plurality of operating system domains to communicate with the controller.

56. The network interface controller as recited in claim 55 wherein said plurality of local resources further comprises:

direct memory access controllers for performing data transfer from its associated one of the plurality of operating system domains and the controller.

57. The network interface controller as recited in claim 55 further comprising a task file for storing tasks to be performed by the controller for said associated one of the plurality of operating system domains.

58. The network interface controller as recited in claim 54 wherein said global resources cannot be configured by said reset from one of the plurality of operating system domains.

59. The network interface controller as recited in claim 54 wherein said global resources cannot be reset by said reset from one of the plurality of operating system domains.

60. The network interface controller as recited in claim 54 further comprising:

registration logic, for registering one of the plurality of operating system domains as a master, for purposes of configuring said global resources.

61. The network interface controller as recited in claim 60 wherein said registration logic comprises a control register which only allows one of the plurality of operating system domains to be registered as said master, at one time.

62. The network interface controller as recited in claim 60 wherein said registration logic comprises logic for insuring that a requesting one of the plurality of operating system domains writes a predefined key into said registration logic which authorizes it to become said master.

63. The network interface controller as recited in claim 62 wherein if said requesting one of the plurality of operating system domains does not write said predefined key into said registration logic, it cannot become said master.

64. The network interface controller as recited in claim 60 wherein said configuring of said global resources comprises:

reloading of firmware into the controller.

65. The network interface controller as recited in claim 60 wherein said configuring of said global resources comprises repartitioning of memory utilized by each of the plurality of operating system domains.

66. The network interface controller as recited in claim 60 further comprising:

timer logic, for providing a timeout if said master does not complete configuration of said global resources within a predetermined time; and

status/messaging logic, for allowing said master to update the status of configuration of said global resources to other ones of the plurality of operating system domains.

67. A method for resetting a network interface controller within the load-store architecture of a plurality of operating system domains, the method comprising:

receiving a reset from one of the plurality of operating system domains;

determining which one of the plurality of operating system domains sent the reset; and

utilizing the reset to reset resources associated with the one of the plurality of operating system domains that sent the reset, while not resetting resources associated with other ones of the plurality of operating system domains.

68. The method for resetting a network interface controller as recited in claim 67 wherein the reset is sent via a load-store network coupled to the network interface controller.

69. The method for resetting a network interface controller as recited in claim 68 wherein the load-store network is a packet based network.

70. The method for resetting a network interface controller as recited in claim 69 wherein the packet based network comprises PCI-Express.

71. The method for resetting a network interface controller as recited in claim 69 wherein the packet based network receives packets from the plurality of operating system domains which have OSD information embedded within them that identifies each of the packets with one of the plurality of operating system domains.

72. The method for resetting a network interface controller as recited in claim 67 wherein the reset is a reset DLLP packet received by the network interface controller over a PCI-Express based network.

73. The method for resetting a network interface controller as recited in claim 67 wherein said step of determining comprises:

examining a packet received from one of the plurality of operating system domains; and

extracting OSD information which associates the packet with the one of the plurality of operating system domains.

74. The method for resetting a network interface controller as recited in claim 67 wherein the reset resources comprise resources that are utilized by the one of the plurality of operating system domains that sent the reset, but not by the other ones of the plurality of operating system domains.

75. The method for resetting a network interface controller as recited in claim 74 wherein the resources comprise control status registers utilized by the one of the plurality of operating system domains which allow the one of the plurality of operating system domains to communicate with the controller.

76. The method for resetting a network interface controller as recited in claim 67 further comprising:

receiving a request for registration from one of the plurality of operating system domains to register it as a management master of the network interface controller;

registering the one of the plurality of operating system domains as a management master of the network interface controller;

allowing the management master to configure global resources within the network interface controller.

77. The method for resetting a network interface controller as recited in claim 76 wherein said step of registering comprises:

authenticating the one of the plurality of operating system domains as an authorized management master of the network interface controller; and

only allowing one of the plurality of operating system domains to be a management master of the network interface controller, at one time.

78. The method for resetting a network interface controller as recited in claim 76 said step of allowing comprises:

reloading of firmware within the network interface controller; and

setting the physical link speed of a port coupling the network interface controller to network fabric.

79. A method for insuring that only one of a plurality of operating system domains can reset global resources within a network interface controller that is shared by the plurality of operating system domains within their load-store architecture, the method comprising:

receiving a request by one of the plurality of operating system domains to be a reset master of the network interface controller;

registering as reset master the one of the plurality of operating system domains that sent the request;

upon registering the reset master, allowing the reset master to reset the global resources within the network interface controller.

80. The method as recited in claim 79 wherein the request is transmitted by the one of the plurality of operating system domains by transmitting a packet to the controller via a load-store link.

81. The method as recited in claim 80 wherein the packet identifies which one of the plurality of operating system domains sent the request.

82. The method as recited in claim 79 wherein only one of the plurality of operating system domains can be a reset master at one time.

83. The method as recited in claim 79 wherein said step of allowing comprises:

re-loading of firmware within the network interface controller.

84. The method as recited in claim 79 wherein said step of registering further comprises:

requiring the one of the plurality of operating system domains to provide a unique key to authenticate it as a reset master.

85. The method as recited in claim 79 wherein if the reset master does not complete reset of the global resources within a predetermined time, the reset master becomes unregistered.

86. The method as recited in claim 79 wherein further comprising:

receiving a reset from one of the plurality of operating system domains;

determining which one of the plurality of operating system domains sent the reset; and

resetting local resources associated with the one of the plurality of operating system domains without resetting local resources associated with other ones of the plurality of operating system domains.

87. The method as recited in claim 86 wherein said step of determining comprises:

examining a packet to extract information embedded within the packet, the information containing OSD information associating the packet with one of the plurality of operating system domains.

88. The method as recited in claim 86 wherein the reset comprises a reset DLLP transmitted by one of the plurality of operating system domains.