PCI Express Load Sharing Network Interface Controller Cluster

• US 20100115174A1
• Filed: 11/05/2008
• Published: 05/06/2010
• Est. Priority Date: 11/05/2008
• Status: Active Grant

• Current Assignee: Jpmorgan Chase Bank NA As The Agent
• Original Assignee: Aprius Incorporated
• Inventors: Talayco, Daniel, Ekici, Ali, Akyol, Bora
• Granted Patent: US 8,503,468 B2
• Time in Patent Office: Days
• US Class Current: 710/316
• CPC Class Codes: G06F 13/385 for adaptation of a particular data processing system to different peripheral devices

Abstract

Embodiments provide load balancing in a virtual computing environment comprising a plurality of PCI-Express switches (the PCIe switching cloud) coupled to a plurality of network interface devices (NICs). An NIC cluster is added between the PCIe switching cloud and the NICs. The NIC cluster is configured to hide NICs from system images and allow the system images to access functions across multiple NICs. The NIC cluster of an embodiment dynamically load balances network resources by performing a hashing function on a header field of received packets. The NIC cluster of an embodiment performs load balancing and state management in association with driver software, which is embedded in the system image. The driver software adds a tag for flow identification to downstream data packets. The NIC cluster distributes data packets based on information in the tag.

Claims

1. A load balancing apparatus that implements load balancing functions for network traffic between physical network interface controllers (NICs) and system images through a PCI-Express (PCIe) switching cloud, the load balancing apparatus comprising:

at least one PCIe high-speed serial link comprising a plurality of lanes;

a plurality of network data link layer control functions;

a plurality of GbE physical layer inputs and outputs (IOs); and

a data processing unit coupled to a set of data buffers for parsing, processing, and storing a plurality of data packets, wherein each data packet encapsulates a data frame.

2. The load balancing apparatus of claim 1, wherein the network data link layer control functions comprise at least a first flow control mechanism and a second flow control mechanism.

3. The load balancing apparatus of claim 2, wherein the first flow control mechanism is based on a hash with a uniform distribution.

4. The load balancing apparatus of claim 2, wherein the second flow control mechanism is based on link aggregation.

5. The load balancing apparatus of claim 2, wherein the network data link layer control functions comprise a media access control (MAC) layer function, wherein the MAC layer function is based on a MAC protocol standard including error processing.

6. The load balancing apparatus of claim 1, wherein the PCIe high-speed serial link is a PCIe endpoint comprising a set of physical functions, wherein the set of physical functions is used by a single-root and multi-root PCIe manager to manage a plurality of virtual functions.

7. The load balancing apparatus of claim 1, wherein a first data packet is a PCIe transaction layer packet (TLP), wherein the first data packet embeds a second data packet.

8. The load balancing apparatus of claim 7, wherein the data processing unit performs a cyclic redundancy check (CRC) on the TLP to verify that the TLP is free of errors, parses the second data packet if the TLP is error-free, performs the first flow control mechanism, stores the second data packet in the set of data buffers, and selects a first MAC address based on the result of the first flow control mechanism.

9. The load balancing apparatus of claim 8, wherein the set of data buffers comprise a plurality of first-in first-out (FIFO) buffers.

10. The load balancing apparatus of claim 8, wherein the data processing unit inserts the selected first MAC address into the second data packet and sends the second data packet to a first GbE physical layer IO.

11. The load balancing apparatus of claim 1, wherein the data frame comprises an Ethernet frame format.

12. The load balancing apparatus of claim 1, wherein each network data link layer control function comprises a MAC layer function capable of being coupled to a GbE physical layer device via a GMII or an RGMII interface.

13. A network input/output (I/O) clustering system for load balancing in a system of networking virtual machines or system images, the network IO clustering system comprising:

a shim driver; and

a network interface card (NIC) cluster.

14. The network IO clustering system of claim 13, wherein the shim driver comprises a machine usable program which can be embedded in at least one of an operating system and a system image.

15. The network IO clustering system of claim 14, wherein the machine usable program inserts a tag having a fixed integer number of octets (8-bit byte) for flow identification, and wherein the tag is inserted at a fixed location within a downstream data packet.

16. The network IO clustering system of claim 15, wherein the tag is divided into a plurality of flow identification fields.

17. The network IO clustering system of claim 16, wherein the plurality of flow identification fields is editable by a network operator.

18. The network IO clustering system of claim 13, wherein the NIC cluster comprises:

at least one PCIe port in accordance with a PCI Express physical format;

a plurality of network data link layer control functions which include a media access control (MAC) layer function;

a plurality of GbE physical layer inputs/outputs (IOs); and

a data processing unit comprising a cyclic redundancy check (CRC) engine and an evaluation mechanism for parsing the flow identification fields.

19. The network IO clustering system of claim 18, wherein the NIC cluster receives a first PCIe transport layer packet (TLP) at the PCIe port, computes a CRC of the first TLP to obtained a first LCRC value, compares the first LCRC value with an LCRC encapsulated in the first TLP, evaluates a first flow identification field if there is a match, and selects a flow control procedure based on a value obtained from the evaluation.

20. The network IO clustering system of claim 19, wherein the flow control procedure comprises at least a first flow control scheme, a second flow control scheme, and a third flow control scheme.

21. The network IO clustering system of claim 20, wherein the first flow control scheme directly forwards a data field within the TLP to an associated GbE physical layer IO.

22. The network IO clustering system of claim 20, wherein the second flow control scheme stores the data field in a FIFO.

23. The network IO clustering system of claim 20, wherein the third flow control scheme teams at least two GbE physical layer IOs.

24. A method for making a cluster of network interface controllers (NICs) appear as a single virtual NIC to a system of clustered system images that operates in conjunction with that NIC cluster through a PCIe switching cloud, the method comprising:

receiving a data packet;

verifying that the received data packet is free of errors;

storing the error-free data packet in a receive First-In First-Out (FIFO) buffer;

executing a flow control mechanism to obtained a first hash value; and

sending the data packet to a first NIC associated with the first hash value.

25. The method of claim 24, wherein the data packet is a PCIe transaction layer packet comprising a PCIe transaction layer frame, a PCIe data layer frame and a data field which comprises a transport layer frame, the transport layer frame being encapsulated in a network layer frame, and the network layer frame being encapsulated in a network data link layer frame.

26. The method of claim 25, wherein the network data link layer frame is an Ethernet frame.

27. The method of claim 24, wherein the verifying further comprises:

performing a cyclic redundancy check (CRC) on the received data layer frame to obtain a first CRC value; and

comparing the first CRC value with an encapsulated LCRC value embedded in the data layer frame.

28. The method of claim 24, wherein the flow control mechanism comprises:

extracting a plurality of headers within the data field into a layer 2 header, a layer 3 header, and a layer 4 header;

performing a hashing function on a first header segment to obtain the first hashed value; and

selecting the first NIC associated with the first hash value.

29. The method of claim 28, wherein the first header segment comprises at least one of the layer 2 header, the layer 3 header, and a combination of the layer 3 header with the layer 4 header.

30. The method of claim 28, wherein the layer 2 header comprises a MAC header.

31. The method of claim 28, wherein the layer 3 header comprises an IPv4 header comprising a 32-bit destination address and a 32-bit source address in accordance with an Internet Protocol (IP) packet format.

32. The method of claim 28, wherein the layer 4 header comprises a TCP/UDP header in accordance with the IP packet format.

33. The method of claim 28, wherein the hashing function performs a cyclic redundancy check (CRC).

34. The method of claim 24, wherein the single virtual NIC has a PCIe bus number and a virtual MAC address.

35. A method for assigning a plurality of network interface cards (NICs) to a system image in a server computer cluster through a PCIe switching cloud, the method comprising:

providing a machine usable program code to the system image, wherein the machine usable program code inserts a tag into a downstream data packet; and

providing a NIC cluster device which parses the tag to direct the data packet to an associated output.

36. The method of claim 35, wherein the tag comprises a fixed integer number of octets (8-bit byte) for flow identification, wherein the tag is inserted at a predetermined location within the downstream data packet.

37. The method of claim 36, wherein the flow identification comprises a plurality of flow identification fields.

38. The method of claim 35, wherein the NIC cluster comprises:

at least one PCIe link in accordance with a PCI Express physical format;

a plurality of NICs, wherein each NIC is associated with a MAC address; and

a data processor coupled to a set of data buffer.

39. The method of claim 38, wherein the plurality of NICs comprise PCIe IOV-(IO virtualization) capable endpoints.

40. The method of claim 38, wherein the data processor handles load distribution and state management based on information obtained from the plurality of flow identification fields.