Intelligent network processor and method of using intelligent network processor

• US 20070266179A1
• Filed: 05/11/2006
• Published: 11/15/2007
• Est. Priority Date: 05/11/2006
• Status: Active Grant

• Current Assignee: Avago Technologies International Sales Pte Limited (Broadcom, Inc.)
• Original Assignee: Emulex Communications Corporation
• Inventors: Chavan, Mukund, Rupanagunta, Sriram
• Granted Patent: US 7,743,197 B2
• Time in Patent Office: Days
• US Class Current: 709/250
• CPC Class Codes: G06F 13/4022 using switching circuits, e.g. switching matrix, connection or expansion network G06F13/4009 takes precedence
• CPC Class Codes: H04L 12/4625 Single bridge functionality, e.g. connection of two networks over a single bridge

Abstract

An intelligent network processor is disclosed that provides a PCI express (PCIe) host bus adapter with firmware selectable hardware capabilities and firmware enabled emulation of capabilities not supported by hardware. Support for Fibre Channel (FC) and Gigabit Ethernet (GbE) protocols are provided through the same fabric ports, including multiple port trunking for both protocols. On chip protocol conversion is provided for switching and routing between FC and GbE ports. Switching using the same crossbar module is provided for both FC and GbE protocols. The crossbar module is coupled to directly access external DDR memory so that messages from FC, GbE, and PCIe interfaces may be switched directly to the DDR memory.

Claims

1. An intelligent network processor (INP) for connection to a PCI express (PCIe) bus, comprising:

one or more processors for emulating one or more virtual PCIe devices supported by the INP; and

a Root Complex/EndPoint (RC/EP) module communicatively couplable to the PCIe bus and to the one or more processors for receiving communications over the PCIe bus, the RC/EP module including a virtualization engine providing a hardware window through which entities on the PCIe bus can communicate with the one or more processors to configure the one or more virtual PCIe devices and send read and write requests to the one or more virtual PCIe devices.

2. The INP as recited in claim 1, further comprising a host bus interface coupled to the RC/EP module and couplable to the PCIe bus for connecting the RC/EP module to the PCIe bus.

3. The INP as recited in claim 1, further comprising a Bi-directional Crossbar Module (BCM) couplable to the RC/EP module and the one or more processors for routing communications between the RC/EP module and the one or more processors.

4. The INP as recited in claim 1, further comprising firmware executable by the one or more processors for programmably emulating all fields in a configuration header of the one or more virtual PCIe devices.

5. The INP as recited in claim 1:

the RC/EP module further comprising a PCIe core for terminating configuration cycles directed to the INP; and

the virtualization engine within the RC/EP module comprising an aperture table for storing entries corresponding to the one or more virtual PCIe devices supported by the INP and also corresponding to Base Address Registers (BARs) in a configuration space of Operating Systems (OSs) in communication with the INP over the PCIe bus, the aperture table for redirecting communications received over the PCIe bus to a processor appropriate for processing that communication.

6. The INP as recited in claim 5, each aperture table entry comprising:

an address field including a base address field and a length field that together represent an address range for that entry; and

a destination field for storing an address of a processor that will be emulating the virtual PCIe device associated with that entry.

7. The INP as recited in claim 5, each aperture table entry comprising one or more programmable rules.

8. The INP as recited in claim 6:

the one or more processors further comprising a support processor for assisting in configuring the one or more virtual PCIe devices supported by the INP; and

the aperture table including a default entry for redirecting configuration communications received over the PCIe bus to the support processor.

9. The INP as recited in claim 8, the support processor for creating the aperture table entries in response to messages received over the PCIe bus.

10. The INP as recited in claim 8, the support processor for storing addresses in the address fields of the aperture table entries in response to messages received over the PCIe bus.

11. The INP as recited in claim 8, the RC/EP module further comprising logic for comparing addresses in communications received over the PCIe bus to the address field of one or more entries in the aperture table, and if there is a match, redirecting the communications to the address in the destination field of the matching entry.

12. The INP as recited in claim 1, wherein the INP is coupled to a memory for storing Requestor Control Blocks (RCBs) containing configuration information accessible by the one or more processors when responding to communications received over the PCIe bus.

13. The INP as recited in claim 1, the INP formed on a single chip.

14. The INP as recited in claim 13, the INP forming a part of a single-root system configuration comprised of a single root complex coupled to one or more endpoints, the INP representing one of the endpoints.

15. The INP as recited in claim 14, the single root complex comprising a single host partitioned into one or more guest OSs managed by a hypervisor.

16. The INP as recited in claim 13 and located on an I/O blade.

17. The INP as recited in claim 16, the I/O blade forming a part of a multi-root system configuration comprised of multiple coupled root complexes, the INP representing one of the root complexes.

18. The INP as recited in claim 17, the I/O blade further comprising a PCIe switch subsystem coupled between the INP and other root complexes, the PCIe switch subsystem including a PCI switch coupled to the INP and one non-transparent bridge coupled to each of the other root complexes, each non-transparent bridge for storing the configuration space of its associated root complex and for terminating a configuration cycle initiated by the associated root complex.

19. The INP as recited in claim 18, the one or more processors configured for sending read requests to each of the non-transparent bridges in the PCIe switch subsystem to read the BARs of each root complex and create entries in the aperture table corresponding to each BAR.

20. The INP as recited in claim 1, the virtualization engine within the RC/EP module comprising configuration logic for evaluating headers of received PCIe communications to determine a response to that communication.

21. The INP as recited in claim 20, wherein the configuration logic redirects the communication to a processor for processing that communication, and in the case of a configuration cycle, terminating the configuration cycle.

22. The INP as recited in claim 5, the virtualization engine within the RC/EP module comprising configuration logic for evaluating headers of received PCIe communications to determine a response to that communication.

23. The INP as recited in claim 22, wherein the configuration logic redirects the communication to the aperture table for further redirection.

24. A method for virtualizing PCIe devices, comprising:

emulating one or more virtual PCIe devices utilizing one or more processors;

receiving PCIe communications over a PCIe bus;

providing a hardware window through which entities on the PCIe bus can communicate with the one or more processors to configure the one or more virtual PCIe devices and send read and write requests to the one or more virtual PCIe devices.

25. The method as recited in claim 24, further comprising programmably emulating all fields in a configuration header of the one or more virtual PCIe devices.

26. The method as recited in claim 25, further comprising:

terminating configuration cycles at a level of the hardware window;

storing entries in an aperture table acting as the hardware window, the entries corresponding to the one or more virtual PCIe devices and also corresponding to Base Address Registers (BARs) in a configuration space of Operating Systems (OSs) in communication with the virtual PCIe devices over the PCIe bus; and

redirecting communications received over the PCIe bus to a processor for processing that communication utilizing the aperture table entries.

27. The method as recited in claim 26, the step of storing entries in the aperture table comprising:

storing an address field including a base address field and a length field that together represent an address range for that entry; and

storing a destination field for storing an address of a processor that will be emulating the virtual PCIe device associated with that entry.

28. The method as recited in claim 26, the step of storing entries in the aperture table comprising storing one or more programmable rules in each entry.

29. The method as recited in claim 27, further comprising:

assisting in configuring the one or more virtual PCIe devices utilizing a support processor; and

creating a default entry in the aperture table for redirecting configuration communications received over the PCIe bus to the support processor.

30. The method as recited in claim 29, further comprising creating the aperture table entries in response to messages received over the PCIe bus using the support processor.

31. The method as recited in claim 30, further comprising storing addresses in the address fields of the aperture table entries in response to messages received over the PCIe bus.

32. The method as recited in claim 30, further comprising comparing addresses in communications received over the PCIe bus to the address field of one or more entries in the aperture table, and if there is a match, redirecting the communication to the address in the destination field of the matching entry.

33. The method as recited in claim 24, performed within a single chip.

34. The method as recited in claim 33, performed in an endpoint in a single-root system configuration comprised of a single root complex coupled to one or more endpoints.

35. The method as recited in claim 34, performed in an endpoint in a single root system configuration wherein the single root complex includes a single host partitioned into one or more guest OSs managed by a hypervisor.

36. The method as recited in claim 33, performed within an I/O blade.

37. The method as recited in claim 36, performed within a multi-root system configuration comprised of multiple coupled root complexes.

38. The method as recited in claim 37, further comprising:

coupling at least one of the multiple root complexes to a non-transparent bridge within a PCIe switch;

storing the configuration space of a root complex in its associated non-transparent bridge; and

terminating a configuration cycle initiated by its associated root complex within the non-transparent bridge.

39. The method as recited in claim 38, further comprising sending read requests to each of the non-transparent bridges in the PCIe switch to read the BARs of each root complex and create entries in the aperture table corresponding to each BAR.

40. The method as recited in claim 24, further comprising evaluating headers of received PCIe communications in configuration logic acting as the hardware window to determine a response to that communication.

41. The method as recited in claim 40, further comprising redirecting the communication to a processor for processing that communication using the configuration logic, and in the case of a configuration cycle, terminating the configuration cycle.

42. The method as recited in claim 27, further comprising redirecting the communication to the aperture table using the configuration logic for further redirection.

43. One or more storage media including a computer program which, when executed by one or more processors in an intelligent network processor (INP), assists in enabling the INP to emulate multiple virtual PCIe devices by causing the one or more processors to perform the steps of:

emulating one or more fields in a configuration header of one or more virtual PCIe devices; and

storing entries in an aperture table acting as a hardware window through which entities on the PCIe bus can communicate with the one or more processors to configure the one or more virtual PCIe devices and send read and write requests to the one or more virtual PCIe devices, the entries corresponding to the one or more virtual PCIe devices and also corresponding to Base Address Registers (BARs) in a configuration space of Operating Systems (OSs) in communication with the virtual PCIe devices over the PCIe bus.

44. The one or more storage media as recited in claim 43, wherein the computer program, when executed by the one or more processors, further causes the one or more processors to perform the step of storing entries in the aperture table by:

storing an address field including a base address field and a length field that together represent an address range for that entry; and

storing a destination field for storing an address of a processor that will be emulating the virtual PCIe device associated with that entry.

45. The one or more storage media as recited in claim 43, wherein the computer program, when executed by the one or more processors, further causes the one or more processors to perform the step of:

creating a default entry in the aperture table for redirecting configuration communications received over the PCIe bus to a support processor.

46. The one or more storage media as recited in claim 45, wherein the computer program, when executed by the one or more processors, further causes the one or more processors to perform the step of storing addresses in the address fields of the aperture table entries in response to messages received over the PCIe bus.

47. The one or more storage media as recited in claim 46, wherein the computer program, when executed by the one or more processors, further causes the one or more processors to perform the steps of sending read requests to each non-transparent bridge in a PCIe switch in a multi-root system configuration to read Base Address Registers (BARs) of each root complex coupled to the non-transparent bridge, and create entries in the aperture table corresponding to each BAR.

48. An intelligent network processor (INP) for connecting to a Fibre Channel (FC) protocol fabric, a Gigabit Ethernet (GbE) protocol fabric and to a host computer, comprising:

a plurality of port interfaces for connection to a protocol fabric, and adjustable to be selectively compatible with FC or GbE;

a plurality of Combined Media Access Controller (CMAC) modules, with each CMAC connected to a corresponding port interface, and adjustable to be selectively compatible with FC or GbE; and

an Application. Processor (APEX) module connected to each port interface for selecting between support for FC or GbE, and connected to each CMAC for selecting between support for FC or GbE, and enabled to execute firmware for responding to protocol designation messages from the host computer, FC messages from a CMAC through its corresponding port interface, and GbE messages from a CMAC through its corresponding port interface.

49. The INP as recited in claim 48:

wherein a Protocol Coordination Stage (PCS) module is connected to a plurality of CMAC modules and is connected to the APEX module for coordinating an operation of the plurality of CMAC modules in a single trunk connected to the corresponding plurality of port interfaces;

wherein the APEX module provides protocol control signals to the PCS and each of the plurality of CMAC modules and corresponding port interfaces connected to the CMAC modules in order to select between FC and GbE and to command when the PCS module is to start and stop the coordination of the trunk formed by the plurality of CMAC modules connected to the PCS.

50. The INP as recited in claim 49, wherein the PCS is capable of coordinating the plurality of CMAC connected to the PCS to form a FC XAUI trunk for connection to a 10 Gb FC protocol fabric through the plurality of port interfaces corresponding to each CMAC connected to the PCS.

51. The INP as recited in claim 49, wherein the PCS is capable of coordinating the plurality of CMAC connected to the PCS to form a GbE XAUI trunk for connection to a 10 GbE protocol fabric through the plurality of port interfaces corresponding to each CMAC connected to the PCS.

52. The INP as recited in claim 48, wherein each CMAC comprises:

switchable 32-bit FC 8B10b decoding logic and 8-bit GbE 8b10b decoding logic;

switchable FC and GbE Start of Frame Delimiter (SFD) and End of Frame Delimiter (EFD) detection logic;

common Cyclic Redundancy Check (CRC) logic;

common packetizing logic; and

common First In First Out (FIFO) buffering.

53. A method for connecting to a Fibre Channel (FC) protocol fabric, a Gigabit Ethernet (GbE) protocol fabric and to a host computer, comprising:

providing a plurality of adjustable port interfaces selectively compatible with FC or GbE for connection to a protocol fabric;

providing a plurality of adjustable Combined Media Access Controller (CMAC) modules selectively compatible with FC or GbE, each CMAC connected to a corresponding port interface; and

connecting an Application Processor (APEX) module to each port interface and to each CMAC for selecting between support for FC or GbE, the APEX module enabled to execute firmware for responding to protocol designation messages from the host computer, FC messages from a CMAC through its corresponding port interface, and GbE messages from a CMAC through its corresponding port interface.

54. The method as recited in claim 53, further comprising:

connecting a Protocol Coordination Stage (PCS) module to a plurality of CMAC modules and to the APEX module for coordinating an operation of the plurality of CMAC modules in a single trunk connected to the corresponding plurality of port interfaces; and

providing protocol control signals from the APEX module to the PCS, the plurality of CMAC modules and corresponding port interfaces connected to the CMAC modules in order to select between FC and GbE and to command when the PCS module is to start and stop the coordination of the trunk formed by the plurality of CMAC modules connected to the PCS.

55. The method as recited in claim 54, further comprising coordinating the plurality of CMAC modules connected to the PCS to form a FC XAUI trunk for connection to a 10 Gb FC protocol fabric through the plurality of port interfaces corresponding to each CMAC connected to the PCS.

56. The method as recited in claim 54, further comprising coordinating the plurality CMAC modules connected to the PCS to form a GbE XAUI trunk for connection to a 10 GbE protocol fabric through the plurality of port interfaces corresponding to each CMAC connected to the PCS.

57. The method as recited in claim 53, further comprising forming each CMAC by:

providing switchable 32-bit FC 8B10b decoding logic and 8-bit GbE 8b10b decoding logic;

providing switchable FC and GbE Start of Frame Delimiter (SFD) and End of Frame Delimiter (EFD) detection logic;

providing common Cyclic Redundancy Check (CRC) logic;

providing common packetizing logic; and

providing common First In First Out (FIFO) buffering.

58. One or more storage media including a computer program which, when executed by one or more processors in an intelligent network processor (INP), assists in enabling the INP to connect to a Fibre Channel (FC) protocol fabric, a Gigabit Ethernet (GbE) protocol fabric and to a host computer by causing the one or more processors to perform the step of:

responding to protocol designation messages from the host computer, FC messages from a Combined Media Access Controller (CMAC) module through its corresponding port interface, and GbE messages from a CMAC through its corresponding port interface and selectively configuring each port interface and each CMAC for either FC or GbE operation.

59. The one or more storage media as recited in claim 58, wherein the computer program, when executed by the one or more processors, further causes the one or more processors to perform the step of providing protocol control signals to a Protocol Coordination Stage (PCS) module connected to a plurality of CMAC modules, the plurality of CMAC modules and corresponding port interfaces connected to the CMAC modules in order to select between FC and GbE and to command when the PCS module is to start and stop the coordination of the trunk formed by the plurality of CMAC modules connected to the PCS.

60. The one or more storage media as recited in claim 59, wherein the computer program, when executed by the one or more processors, further causes the one or more processors to perform the step of coordinating the plurality of CMAC modules connected to the PCS to form a FC XAUI trunk for connection to a 10 Gb FC protocol fabric through the plurality of port interfaces corresponding to each CMAC connected to the PCS.

61. The one or more storage media as recited in claim 59, wherein the computer program, when executed by the one or more processors, further causes the one or more processors to perform the step of coordinating the plurality CMAC modules connected to the PCS to form a GbE XAUI trunk for connection to a 10 GbE protocol fabric through the plurality of port interfaces corresponding to each CMAC connected to the PCS.

62. An intelligent network processor (INP) having expanded Media Access Controller (MAC) packet filtering, comprising:

one or more MACs, each with a fixed number of packet filtering rule registers; and

a hash engine communicatively coupled to the one or more MACs for receiving a destination MAC address, generating a hash output, and comparing the hash output to a table of hash entries containing valid destination MAC addresses to determine if there is a match;

wherein the table of hash entries is stored external to the INP and is not limited by the fixed number of packet filtering rule registers; and

wherein the hash engine includes programmable offsets for identifying the destination MAC address in a packet formatted according to different protocols.

63. A method for providing expanded Media Access Controller (MAC) packet filtering, comprising:

providing one or more MACs, each with a fixed number of packet filtering rule registers;

storing a table of hash entries in memory not limited by the fixed number of packet filtering rule registers;

selectively adjusting programmable offsets in the hash engine to identify a destination MAC address in a packet formatted according to a particular protocol;

receiving the destination MAC address at a hash engine and generating a hash output;

comparing the hash output to a table of hash entries containing valid destination MAC addresses to determine if there is a match; and

forwarding the packet to a processor for processing if there was a match.

64. An intelligent network processor (INP) for connecting to a first protocol fabric, a second protocol fabric and to a host computer, comprising:

a plurality of port interfaces for connection to a protocol fabric, and adjustable to be selectively compatible with the first or second protocol;

a plurality of Combined Media Access Controller (CMAC) modules, with each CMAC connected to a corresponding port interface, and adjustable to be selectively compatible with the first or second protocol; and

an Application Processor (APEX) module connected to each port interface for selecting between support for the first or second protocol, and connected to each CMAC for selecting between support for the first or second protocol, and enabled to execute firmware for responding to protocol designation messages from the host computer, messages in the first protocol from a CMAC through its corresponding port interface, and messages in a second protocol from a CMAC through its corresponding port interface;

wherein the APEX module is capable of executing firmware for the reception of messages in the first protocol from a first CMAC through its corresponding first port interface, for the translation of such messages in the first protocol to messages in the second protocol, and for the transmission of the messages in the second protocol through a second CMAC through its corresponding second port interface.

65. The INP as recited in claim 64, wherein the first protocol is FC and the second protocol is GbE.

66. An intelligent network processor (INP) on a single silicon die for connection to a Fibre Channel (FC) protocol fabric and to a Gigabit Ethernet (GbE) protocol fabric and to a host computer, comprising:

a plurality of port interfaces for connection to a protocol fabric, and adjustable to be selectively compatible with FC or GbE;

a Bi-directional Crossbar Module (BCM) coupled to receive messages from each of the plurality of port modules, and having crossbar passageways compatible with both FC and GbE messages; and

an application processor (APEX) module coupled to the BCM for controlling the connections between the port interfaces through the BCM, and for allowing simultaneous crossbar connections of FC and GbE messages through the passageways of the BCM.

67. The INP as recited in claim 66, wherein the APEX is capable of executing firmware for switching of messages by detecting a desired destination interface port from contents of a message from an incoming port, and by activating the BCM to connect the incoming port to the desired destination port.

68. The INP as recited in claim 66, wherein the APEX maintains a routing table with entries mapping desired addresses with corresponding interface ports, and is capable of executing firmware for the routing of messages by detecting a desired address from contents of a message from an incoming port, and by activating the BCM to connect the incoming port to the destination port obtained by lookup of the desired address from the routing table.

69. A method for connecting to a first protocol fabric, a second protocol fabric and to a host computer, comprising:

providing a plurality of port interfaces for connection to a protocol fabric, the plurality of port interfaces being adjustable to be selectively compatible with the first or second protocol;

providing a plurality of Combined Media Access Controller (CMAC) modules, each CMAC connected to a corresponding port interface and adjustable to be selectively compatible with the first or second protocol;

responding to protocol designation messages from the host computer, messages in the first protocol from a CMAC through its corresponding port interface, and messages in a second protocol from a CMAC through its corresponding port interface; and

receiving messages in the first protocol from a first CMAC through its corresponding first port interface, translating such messages in the first protocol to messages in the second protocol, and transmitting the messages in the second protocol through a second CMAC through its corresponding second port interface.

70. The method as recited in claim 69, wherein the first protocol is FC and the second protocol is GbE.

71. A method for connection to a Fibre Channel (FC) protocol fabric and to a Gigabit Ethernet (GbE) protocol fabric and to a host computer, comprising:

providing a plurality of port interfaces for connection to a protocol fabric, the plurality of port interfaces being adjustable to be selectively compatible with FC or GbE;

receiving messages from each of the plurality of port modules through a Bi-directional Crossbar Module (BCM) having crossbar passageways compatible with both FC and GbE messages;

controlling the connections between the port interfaces through the BCM; and

allowing simultaneous crossbar connections of FC and GbE messages through the passageways of the BCM.

72. The method as recited in claim 71, further comprising switching messages by detecting a desired destination interface port from contents of a message from an incoming port, and activating the BCM to connect the incoming port to the desired destination port.

73. The method as recited in claim 71, further comprising:

maintaining a routing table with entries mapping desired addresses with corresponding interface ports;

routing messages by detecting a desired address from contents of a message from an incoming port; and

activating the BCM to connect the incoming port to the destination port obtained by lookup of the desired address from the routing table.

74. An intelligent network processor (INP) for connection to a protocol fabric, comprising:

a plurality of port interfaces for connection to a protocol fabric, and adjustable to be selectively compatible with FC or GbE;

a bidirectional crossbar module (BCM) coupled to receive messages from each of the plurality of port modules, and having crossbar passageways compatible with both FC and GbE messages;

a DDR memory interface for connecting to an external DDR memory, for storing and retrieving data messages in queues contained in the DDR memory, and connected to send and receive data messages directly from the BCM; and

an application processor module (APEX) coupled to the BCM for controlling the connections between the port interfaces and the DDR memory interface through the BCM, and enabling direct memory access operations between the DDR memory interface and the BCM without data messages passing through the APEX.

75. A method for connecting a plurality of port interfaces to a protocol fabric, comprising:

providing a plurality of port interfaces for connection to a protocol fabric, the plurality of port interfaces being adjustable to be selectively compatible with FC or GbE;

receiving messages from each of the plurality of port interfaces through a bidirectional crossbar module (BCM) having crossbar passageways compatible with both FC and GbE messages;

connecting to an external DDR memory for storing and retrieving data messages in queues contained in DDR memory, and for sending and receiving data messages directly from the BCM; and

controlling the connections between the port interfaces and the DDR memory interface through the BCM, and enabling direct memory access operations between the DDR memory interface and the BCM without data messages passing through the APEX.