Method and apparatus for a shared I/O network interface controller

• US 8,346,884 B2
• Filed: 07/30/2004
• Issued: 01/01/2013
• Est. Priority Date: 01/21/2003
• Status: Active Grant

• Current Assignee: Mellanox Technologies Limited (NVIDIA Corporation)
• Original Assignee: NextIO Incorporated
• Inventors: Pettey, Christopher J.
• Primary Examiner(s): KIM, TAE K
• Application Number:
  US10/909,254
• Publication Number:
  US 20050053060A1
• Time in Patent Office: 3,077 Days
• Field of Search: 709/218, 709/244, 709/220, 709/229, 709213-215, 709/223, 709/238, 709248-250, 710/56, 710/308, 710/35, 710/316, 710/317, 711/118, 370/392, 370/463, 370/351, 370/357, 370/360, 370/381, 370/382
• US Class Current: 709/213
• CPC Class Codes: H04L 45/74 Address processing for routing
• CPC Class Codes: H04L 61/00 Network arrangements, protocols or services for addressing or naming
• CPC Class Codes: H04L 61/10 of different types
• CPC Class Codes: H04L 67/1001 for accessing one among a plurality of replicated servers
• CPC Class Codes: H04L 67/1017 based on a round robin mechanism
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/63 Routing a service request depending on the request content or context
• CPC Class Codes: H04L 69/18 Multiprotocol handlers, e.g. single devices capable of handling multiple protocols
• CPC Class Codes: H04L 69/22 Parsing or analysis of headers
• CPC Class Codes: H04L 69/329 in the application layer [OSI layer 7]

Abstract

A shared network interface controller (NIC) interfaces a plurality of operating system domains as part of the load-store architecture of the operating system domains. A bus interface couples the NIC to a load-store domain bus (such as PCI-Express), using header information to associate data on the bus with an originating operating system domain. Transmit/receive logic connects the NIC to the network. Association logic allows the NIC to designate, and later lookup, which destination MAC address (on the Ethernet side) is associated with which operating system domain. Descriptor register files and Control Status Registers (CSR'"'"'s) specific to an operating system domain are duplicated and made available for each domain. Several direct memory access (DMA) engines are provided to improve throughput. Packet replication logic, filters (perfect and hash) and VLAN tables are used for looping back packets originating from one operating system domains to another and other operations.

Claims

1. A network interface controller (NIC) which provides an interface between a network and a plurality of operating system domains (OSDs), each OSD having a logically distinct load-store domain that uses a common memory map to address both random access memory (RAM) and input-output (I/O) devices, the controller existing within the load-store domain of each of the plurality of OSDs, the controller comprising:

a bus interface between the plurality of OSDs and the controller, the bus interface enabling each OSD to directly access the controller from within its own logically distinct load-store domain using load-store instructions that address one or more memory-mapped addresses for the shareable NIC that are within the common memory map of the OSD'"'"'s load-store domain;

a network interface between the network and the controller; and

association logic, coupled to both said bus interface and said network interface to associate data coming from the network with at least one of the plurality of operating system domains.

2. The network interface controller as recited in claim 1 wherein the controller comprises a shared Ethernet controller.

3. The network interface controller as recited in claim 1 wherein the network is an Ethernet network.

4. The network interface controller as recited in claim 1 wherein each of the plurality of operating system domains comprise:

at least one processor;

a memory coupled to said at least one processor; and

an operating system executing on said processor.

5. The network interface controller as recited in claim 1 wherein each of the plurality of operating system domains comprise a computer server.

6. The network interface controller as recited in claim 5 wherein said computer server is a blade server.

7. The network interface controller as recited in claim 1 wherein the load-store domain of each of the plurality of operating system domains comprises memory which is accessed by load and store instructions executing within each of the plurality of operating system domains.

8. The network interface controller as recited in claim 1 wherein each of the plurality of operating system domains comprise:

at least one processor;

a memory coupled to said at least one processor;

an operating system executing on said at least one processor;

a load instruction for loading data from said memory into said at least one processor; and

a store instruction for storing data from said at least one processor into said memory;

wherein each of the plurality of operating system domains communicates with the controller using said load instruction and said store instruction.

9. The network interface controller as recited in claim 1 wherein each of the plurality of operating system domains communicates with the network interface controller using a software driver which is loaded into each of the plurality of operating system domains.

10. The network interface controller as recited in claim 9 wherein said software driver is mapped within the load-store domain of each of the plurality of operating system domains.

11. The network interface controller as recited in claim 1 wherein said bus interface comprises:

an interface to a load-store domain network which transfers data from each of the plurality of operating system domains to the controller; and

OSD identification logic, which determines for said data received from said load-store domain network, which of the plurality of operating system domains is associated with said data.

12. The network interface controller as recited in claim 11 wherein said load-store domain network comprises PCI-Express+.

13. The network interface controller as recited in claim 12 wherein said PCI-Express+comprises PCI-Express with OSD header information.

14. The network interface controller as recited in claim 13 wherein said OSD header information comprises a header included with PCI-Express information which identifies which of the plurality of operating system domains is associated with said PCI-Express information.

15. The network interface controller as recited in claim 1 wherein said network interface comprises transmit/receive logic for buffering transmit data to be transmitted to the network, and for buffering receive data received from the network.

16. The network interface controller as recited in claim 15 wherein said transmit/receive logic comprises:

a virtual fifo for storing said transmit data and said receive data;

wherein said transmit data is tagged to identify which of the plurality of operating system domains said data came from; and

wherein said receive data is tagged to identify which of the plurality of operating system domains should receive said receive data.

17. The network interface controller as recited in claim 15 wherein said transmit/receive logic comprises:

a plurality of transmit fifo'"'"'s, each corresponding to one of the plurality of operating system domains; and

a plurality of receive fifo'"'"'s, each corresponding to one of the plurality of operating system domains.

18. The network interface controller as recited in claim 15 wherein said transmit/receive logic comprises:

a virtual transmit fifo for storing said transmit data from the plurality of operating system domains; and

a virtual receive fifo for storing said receive data received from the network.

19. The network interface controller as recited in claim 18 wherein said receive data is tagged with an OSD identifier after it is received from the network into the controller.

20. The network interface controller as recited in claim 19 wherein said OSD identifier designates which of the plurality of operating system domains should receive the data.

21. The network interface controller as recited in claim 19 wherein said OSD identifier is provided by said association logic.

22. The network interface controller as recited in claim 21 wherein said association logic correlates a MAC address within said receive data with a corresponding OSD.

23. The network interface controller as recited in claim 1 wherein said association logic comprises a lookup table which contains a plurality of MAC address which are correlated to a plurality of OSD'"'"'s.

24. The network interface controller as recited in claim 23 wherein each of said plurality of OSD'"'"'s is associated with at least one of said plurality of MAC addresses.

25. The network interface controller as recited in claim 23 wherein data received from the network contains a destination MAC address.

26. The network interface controller as recited in claim 25 wherein when said data is received from the network into the controller, said destination MAC address is read from said data, and said lookup table is read to determine an associated OSD for said data.

27. The network interface controller as recited in claim 1 further comprising:

a non volatile memory for storing a plurality of MAC addresses.

28. The network interface controller as recited in claim 1 further comprising:

a plurality of direct memory access (DMA) engines; and

DMA arbitration, coupled to said plurality of DMA engines;

wherein said plurality of DMA engines are selected by said DMA arbitration to perform data transfer from the plurality of operating system domains to the controller.

29. The network interface controller as recited in claim 1 further comprising:

a plurality of control status register (CSR) sets;

which are to be associated with the plurality of OSD'"'"'s.

30. The network interface controller as recited in claim 29 wherein said plurality of CSR sets each comprise:

a head pointer register for storing a head pointer to a location within its associated OSD; and

a tail pointer register for storing a tail pointer to a location within its associated OSD.

31. The network interface controller as recited in claim 1 further comprising:

descriptor logic, for storing a plurality of descriptors for the plurality of operating system domains.

32. The network interface controller as recited in claim 31 wherein said descriptor logic comprises:

a virtual descriptor cache, for storing transmit and receive descriptors for the plurality of operating system domains; and

tags, coupled to said virtual descriptor cache, for tagging each of said descriptors with their associated one of the plurality of operating system domains.

33. The network interface controller as recited in claim 31 wherein said descriptor logic comprises:

a virtual transmit descriptor cache for storing transmit descriptors for the plurality of operating system domains; and

a virtual receive descriptor cache for storing receive descriptors for the plurality of operating system domains.

34. The network interface controller as recited in claim 31 wherein said descriptor logic comprises:

a plurality of transmit descriptor caches, each of said caches associated with one of the plurality of operating system domains; and

a plurality of receive descriptor caches, each of said caches associated with one of the plurality of operating system domains;

wherein transmit descriptors are stored in said plurality of transmit descriptor caches; and

wherein receive descriptors are stored in said plurality of receive descriptor caches.

35. A shareable network interface controller (NIC) for a computing device having a plurality of operating system domains (OSDs), each OSD having a logically distinct load-store domain that uses a common memory map to address both random access memory (RAM) and input-output (I/O) devices, the shareable NIC comprising:

transmit/receive logic for coupling the controller to a network to transmit/receive network communication;

bus interface and operating system domain identification logic, for interfacing the controller to the plurality of operating system domains (OSD'"'"'s), and for associating OSD communication to and from said bus interface with ones of the plurality of OSD'"'"'s;

the bus interface enabling each OSD to directly access the shareable NIC from within its own logically distinct load-store domain using load-store instructions that address one or more memory-mapped addresses for the shareable NIC that are within the common memory map of the OSD'"'"'s load-store domain;

association logic, coupled to said transmit/receive logic, for associating said network communication with ones of the plurality of OSD'"'"'s;

wherein by associating said network communication with ones of the plurality of OSD'"'"'s, and associating said OSD communication with ones of the plurality of OSD'"'"'s, the shared controller provides communication between the plurality of OSD'"'"'s and said network.

36. The shareable network interface controller as recited in claim 35 wherein the shareable network interface controller is a shareable Ethernet controller.

37. The shareable network interface controller as recited in claim 36 wherein said shareable Ethernet controller supports 10/100 megabit, 1 gigabit, and/or 10 gigabit network communication.

38. The shareable network interface controller as recited in claim 35 further comprising resources directly accessible by the plurality of OSD'"'"'s using load and/or store instructions.

39. The shareable network interface controller as recited in claim 38 wherein said resources comprise memory spaces for each of the plurality of OSD'"'"'s.

40. The shareable network interface controller as recited in claim 35 wherein each of the plurality of OSD'"'"'s is unaware that it is sharing the shareable network interface controller with another one of the plurality of OSD'"'"'s.

41. The shareable network interface controller as recited in claim 35 wherein each of the plurality of OSD'"'"'s comprise:

a processing complex; and

memory, coupled to said processing complex.

42. The shareable network interface controller as recited in claim 41 wherein said processing complex comprises at least one processor.

43. The shareable network interface controller as recited in claim 35 wherein said transmit/receive logic stores said network communication from/to at least two of the plurality of OSD'"'"'s, simultaneously.

44. The shareable network interface controller as recited in claim 35 wherein said transmit/receive logic comprises:

a transmit fifo for storing said network communication to be transmitted to said network; and

a receive fifo, for storing network communication received from said network.

45. The shareable network interface controller as recited in claim 44 wherein said transmit fifo comprises a virtual fifo for storing said network communication from a plurality of OSD'"'"'s, simultaneously.

46. The shareable network interface controller as recited in claim 44 wherein said receive fifo comprises a virtual fifo for storing said network communication destined for a plurality of OSD'"'"'s, simultaneously.

47. The shareable network interface controller as recited in claim 44 wherein said transmit fifo and said receive fifo each further comprise:

OSD management logic, for associating data within each of said fifo'"'"'s with corresponding OSD'"'"'s.

48. The shareable network interface controller as recited in claim 35 wherein said transmit/receive logic comprises:

a virtual fifo for storing both said network communication to be transmitted to said network, and said network communication received from said network, simultaneously.

49. The shareable network interface controller as recited in claim 48 wherein simultaneously indicates that transmit network communication and receive network communication coexists within said virtual fifo.

50. The shareable network interface controller as recited in claim 48 wherein said virtual fifo further comprises OSD management logic, for associating data within said virtual fifo with corresponding OSD'"'"'s.

51. The shareable network interface controller as recited in claim 35 wherein said transmit/receive logic comprises:

a plurality of transmit fifo'"'"'s, one for each of the plurality of OSD'"'"'s; and

a plurality of receive fifo'"'"'s, one for each of the plurality of OSD'"'"'s.

52. The shareable network interface controller as recited in claim 35 wherein said network is an Ethernet network.

53. The shareable network interface controller as recited in claim 35 wherein said network is a packet based network.

54. The shareable network interface controller as recited in claim 35 wherein said network communication comprises packet based communication.

55. The shareable network interface controller as recited in claim 35 wherein said bus interface and operating system domain identification logic comprises:

a bus interface to a load-store architecture bus; and

operating system domain identification logic, coupled to said bus interface.

56. The shareable network interface controller as recited in claim 55 wherein said load-store architecture bus comprises a packet based serial communication bus.

57. The shareable network interface controller as recited in claim 55 wherein said load-store architecture bus comprises a PCI-Express+bus.

58. The shareable network interface controller as recited in claim 57 wherein said PCI-Express+bus comprises a PCI-Express bus which includes an OSD header.

59. The shareable network interface controller as recited in claim 58 wherein said OSD header identifies an associated OSD for packets transmitted over said PCI-Express bus.

60. The shareable network interface controller as recited in claim 58 wherein said operating system domain identification logic examines said OSD header for packets received by said bus interface to determine which of the plurality of OSD'"'"'s transmitted said packets, and places said OSD header into packets to be transmitted by said bus interface to identify which of the plurality of OSD'"'"'s said transmitted packets are for.

61. The shareable network interface controller as recited in claim 35 wherein said association logic comprises:

a look up table for storing OSD and MAC address associations.

62. The shareable network interface controller as recited in claim 61 wherein said look up table comprises:

a plurality of OSD entries; and

a plurality of MAC address entries;

wherein each of said plurality of OSD entries is associated with at least one of said plurality of MAC address entries.

63. The shareable network interface controller as recited in claim 62 wherein each of said OSD entries comprise a bit representation for identifying an upstream operating system domain.

64. The shareable network interface controller as recited in claim 63 wherein said bit representation comprises 3-bits for designating eight distinct operating system domains.

65. The shareable network interface controller as recited in claim 63 wherein said bit representation comprises 8-bits for designating eight or more distinct operating system domains.

66. The shareable network interface controller as recited in claim 62 wherein each of said plurality of MAC address entries comprise a unique machine address.

67. The shareable network interface controller as recited in claim 62 wherein by associating a MAC address entry with an OSD entry, the controller can associate said network communication with ones of said plurality of OSD entries based on a destination MAC address within said network communication.

68. The shareable network interface controller as recited in claim 62 wherein by associating a MAC address entry with an OSD entry, the controller can associate said OSD communication with ones of said plurality of MAC address entries.

69. The shareable network interface controller as recited in claim 35 further comprising:

a plurality of control status register sets (CSR'"'"'s), coupled to said bus interface and operating system domain identification logic, each available to be associated with a unique one of the plurality of operating system domains.

70. The shareable network interface controller as recited in claim 69 wherein each of said plurality of control register sets comprise:

a head pointer register, for storing a head pointer associated with a descriptor block in a memory associated with one of the plurality of OSD'"'"'s; and

a tail pointer register, for storing a tail pointer associated with said descriptor block in said memory associated with said one of the plurality of OSD'"'"'s.

71. The shareable network interface controller as recited in claim 69 wherein each of the plurality of operating system domains must be associated with a different one of said plurality of control status register sets.

72. The shareable network interface controller as recited in claim 69 wherein unique one of the plurality of operating system domains implies that each of said plurality of control status register sets is associated with no more than one of the plurality of operating system domains.

73. The shareable network interface controller as recited in claim 35 further comprising:

a descriptor cache, coupled to said bus interface and operating system domain identification logic, for storing descriptors retrieved from the plurality of operating system domains.

74. The shareable network interface controller as recited in claim 73 wherein said descriptors define memory locations within memories of the plurality of operating system domains where said OSD communication is transferred to/from the controller.

75. The shareable network interface controller as recited in claim 73 wherein said descriptor cache comprises:

a plurality of descriptor caches, each available to be associated with one of the plurality of operating system domains; and

descriptor arbitration logic, coupled to said plurality of descriptor caches, for associating each of the plurality of descriptor caches with each of the plurality of operating system domains.

76. The shareable network interface controller as recited in claim 73 wherein said descriptor cache comprises:

a virtual descriptor cache, for storing descriptors from the plurality of operating system domains; and

tag logic, coupled to the virtual descriptor cache, for associating each of said descriptors with their associated one of the plurality of operating system domains.

77. The shareable network interface controller as recited in claim 76 wherein said virtual descriptor cache further comprises:

a virtual transmit descriptor cache for storing descriptors for said OSD communication to be transmitted as said network communication; and

a virtual receive descriptor cache for storing descriptors for said network communication to be received as said OSD communication.

78. The shareable network interface controller as recited in claim 35 further comprising:

a plurality of direct memory access (DMA) engines, said engines used by the controller to DMA said OSD communication from the plurality of OSD'"'"'s, and to DMA said network communication to the plurality of OSD'"'"'s.

79. The shareable network interface controller as recited in claim 78 wherein said plurality of DMA engines further comprise:

DMA arbitration, coupled to said plurality of DMA engines, for arbitrating selection of said plurality of DMA engines among the plurality of OSD'"'"'s.

80. The shareable network interface controller as recited in claim 35 further comprising:

packet replication logic, coupled to said bus interface and operating system domain identification logic, for re-storing said OSD communication into said transmit/receive logic for multicast or broadcast communication.

81. The shareable network interface controller as recited in claim 80 wherein said multicast or broadcast communication comprises communication from one of the plurality of OSD'"'"'s to more than one destination MAC address on said network.

82. The shareable network interface controller as recited in claim 80 wherein said packet replication logic places said OSD communication within a receive portion of said transmit/receive logic when a loopback communication occurs.

83. The shareable network interface controller as recited in claim 82 wherein said loopback communication occurs when a destination MAC address for said OSD communication corresponds to a MAC address within said association logic.