Controller integration

• US 9,256,560 B2
• Filed: 01/14/2010
• Issued: 02/09/2016
• Est. Priority Date: 07/29/2009
• Status: Active Grant

• Current Assignee: Xilinx, Inc. (Advanced Micro Devices, Inc.)
• Original Assignee: SolarFlare Communications Incorporated (Advanced Micro Devices, Inc.)
• Inventors: Pope, Steven L., Riddoch, David, Roberts, Derek
• Primary Examiner(s): CATTUNGAL, AJAY P
• Application Number:
  US12/687,382
• Publication Number:
  US 20110029734A1
• Time in Patent Office: 2,217 Days
• Field of Search: 370/350, 370/389, 370/392, 370/395.1, 370/401, 370/410, 370/412, 370/419, 370/426, 709/228, 709/230, 709/245
• US Class Current: 1/1
• CPC Class Codes: G06F 13/385 for adaptation of a particular data processing system to different peripheral devices
• CPC Class Codes: G06F 13/4295 using an embedded synchronisation
• CPC Class Codes: H04L 12/4633 Interconnection of networks using encapsulation techniques, e.g. tunneling
• CPC Class Codes: H04L 2101/622 Layer-2 addresses, e.g. medium access control [MAC] addresses
• CPC Class Codes: H04L 49/90 Buffering arrangements
• CPC Class Codes: H04L 61/2596 Translation of addresses of the same type other than IP, e.g. translation from MAC to MAC addresses
• CPC Class Codes: H04L 69/22 Parsing or analysis of headers

Abstract

Roughly described, a data processing system comprises a central processing unit and a split network interface functionality, the split network interface functionality comprising: a first sub-unit collocated with the central processing unit and configured to at least partially form a series of network data packets for transmission to a network endpoint by generating data link layer information for each of those packets; and a second sub-unit external to the central processing unit and coupled to the central processing unit via an interconnect, the second sub-unit being configured to physically signal the series of network data packets over a network.

Claims

1. A data processing system comprising a central processing unit and a split network interface functionality, the split network interface functionality comprising:

a first sub-unit collocated with or part of the central processing unit and configured to partially form a series of network data packets for transmission to an endpoint of the network by generating data link layer information for each of those packets according to a data link layer protocol of the network; and

a second sub-unit external to the central processing unit and coupled to the central processing unit via an interconnect having a data link layer protocol different from that of the network, the second sub-unit being configured to receive the partially formed network data packets and to physically signal the series of network data packets over the network using the data link layer information generated by the first sub-unit.

2. A data processing system as claimed in claim 1, wherein the first sub-unit is configured to encapsulate each network data packet of the series in one or more interconnect data packets in accordance with the data protocol in use over the interconnect.

3. A data processing system as claimed in claim 1, wherein the first sub-unit is configured to transmit the interconnect data packets to the second sub-unit so as to cause the second sub-unit to signal the respective network data packets over the network.

4. A data processing system as claimed in claim 1, wherein the second sub-unit is configured to complete the formation of each of the series of network data packets at the data link layer of the network protocol in use over the network.

5. A data processing system as claimed in claim 4, wherein the second sub-unit is configured to complete the formation of each of the series of network data packets at the data link layer by calculating one or more checksums for each network data packet.

6. A data processing system as claimed in claim 1, wherein the second sub-unit is configured to physically signal the series of completed data packets over the network in accordance with the physical layer requirements of the network protocol in use over the network.

7. A data processing system as claimed in claim 1, further comprising a memory and/or cache coupled to the central processing unit and, at the central processing unit, a memory management unit configured to provide access to the memory and/or cache for the central processing unit and the first sub-unit.

8. A data processing system as claimed in claim 7, wherein the memory and/or cache is configured to hold a transmit queue comprising data for transmission over the network and the first sub-unit is configured to access the transmit queue by means of a read request to the memory management unit and form the series of network data packets using the data for transmission over the network.

9. A data processing system as claimed in claim 1, wherein the second sub-unit is operable to receive a series of network data packets over the network and the first sub-unit is configured to at least partially process each received network data packet at the data link layer of the network so as to extract data carried within the network data packet.

10. A data processing system as claimed in claim 9, further comprising a memory and/or cache coupled to the central processing unit and, at the central processing unit, a memory management unit configured to provide access to the memory and/or cache for the central processing unit and the first sub-unit,

wherein the first sub-unit is configured to store at the memory a forwarding table comprising information such as to allow the first sub-unit to, for each received network data packet, route extracted data to one or more receive queues at the memory and/or another network device in dependence on the content of the received network data packet.

11. A data processing system as claimed in claim 9, wherein the second sub-unit is configured to encapsulate each received network data packet in one or more interconnect data packets in accordance with the data protocol in use over the data link of the interconnect proximal to the second sub-unit.

12. A data processing system as claimed in claim 9, wherein the second sub-unit is configured to transmit the interconnect data packets to the first sub-unit so as to cause the first sub-unit to at least partially process each received network data packet at the data link layer of the network and extract the data carried within received network data packet.

13. A data processing system comprising:

a central processing unit, the central processing unit including a network interface controller, the network interface controller being configured to form intermediate data units by at least partially framing at the data link layer data for transmission over a network in accordance with a network protocol, each intermediate data unit including a destination network address for the data for transmission;

a memory coupled to the central processing unit;

a media interface connected to the network and configured to transmit fully-formed data packets over the network; and

a first interconnect extending between the central processing unit and the media interface, the interconnect being formed of one or more data links;

wherein the network interface controller is further configured to encapsulate said intermediate data units of the network protocol in interconnect data packets for transmission over the first interconnect, and the media interface is configured to extract the intermediate data units from the interconnect data packets received over the first interconnect and process the intermediate data units at the physical layer so as to complete the formation of data packets in accordance with the network protocol.

14. A data processing system as claimed in claim 13, wherein the network interface controller is configured to encapsulate each intermediate data unit in one or more interconnect data packets in accordance with the data protocol in use over the data link of the first interconnect proximal to the central processing unit.

15. A data processing system as claimed in claim 13, wherein the network interface controller is configured to cause the one or more interconnect data packets to be transmitted to the media interface so as to cause the media interface to transmit a fully-formed network data packet over the network.

16. A data processing system as claimed in claim 13, wherein the network interface controller is configured to write an encapsulation header to each of the interconnect data packets, the encapsulation header comprising information such as to allow the media interface to reassemble the respective intermediate data unit from the interconnect data packets.

17. A data processing system as claimed in claim 13, wherein the first interconnect comprises two data links:

a first data link extending between the central processing unit and a translation unit; and

a second data link extending between the translation unit and the media interface,

the translation unit being configured to translate interconnect data packets framed in accordance with the data protocol in use over the first data link into interconnect data packets framed in accordance with the data protocol in use over the second data link, and vice versa.

18. A data processing system as claimed in claim 17, wherein the translation unit is configured to translate interconnect data packets by:

removing framing data of the data protocol in use over the first data link and re-framing the one or more interconnect data packets in accordance with the data protocol in use over the second data link;

or

adding an additional framing layer about the one or more interconnect data packets in accordance with the data protocol in use over the second data link.

19. A data processing system as claimed in claim 17, wherein the first data link is a CPU interconnect and the second data link is a peripheral data bus.

20. A data processing system as claimed in claim 19, wherein the media interface is supported at the peripheral data bus.

21. A data processing system as claimed in claim 19, wherein the CPU interconnect is one of a HyperTransport interconnect , a QuickPath interconnect and a PCIe interconnect.

22. A data processing system as claimed in claim 19, wherein the peripheral data bus is PCIe.

23. A data processing system as claimed in claim 13, wherein the central processing unit further comprises a memory management unit configured to provide access to the memory for the central processing unit and the network interface controller.

24. A data processing system as claimed in claim 13, wherein the network interface controller is configured to, if the destination network address belongs to a process supported at the central processing unit, write the data for transmission directly to a receive queue at the memory without encapsulating that data in one or more interconnect data packets, the receive queue corresponding to the destination network address.

25. A data processing system as claimed in claim 24, wherein the network interface controller is further configured to not frame that data at the data link layer in accordance with the network protocol.

26. A data processing system as claimed in claim 23, wherein the memory is configured to hold one or more transmit queues comprising data for transmission over the network and the network interface controller is operable to access said data for transmission at its respective transmit queue by means of a read request to the memory management unit.

27. A data processing system as claimed in claim 23, wherein the memory management unit comprises a cache operable to store a copy of at least some of the data held at the memory, the cache being accessible to both the central processing unit and the network interface controller.

28. A data processing system as claimed in claim 27, wherein the cache is configured to store data recently accessed at the memory by the central processing unit.

29. A data processing system as claimed in claim 27, wherein the network interface controller is configured to, for any data for transmission that is available at both the memory and the cache, read that data for transmission from the cache in preference to reading the data from the memory.

30. A data processing system as claimed in claim 27, wherein the network interface controller is configured to, for data for transmission that is not available at the cache, read the said data from the memory and not write the data into the cache.

31. A data processing system as claimed in claim 29, wherein the network interface controller is configured to, on reading the data for transmission from the cache, invalidate the corresponding lines of the cache so as to allow other processes executing at the central processing unit to overwrite those lines.

32. A data processing system as claimed in claim 13, wherein:

the media interface is operable to receive network data packets over the network and encapsulate the received network data packets in interconnect data packets for transmission over the first interconnect; and

the network interface controller is configured to extract the received network data packets from the interconnect data packets and at least partially process at the data link layer each received network data packet in accordance with the network protocol so as to extract the data carried within each received network data packet.

33. A data processing system as claimed in claim 32, wherein the media interface is configured to encapsulate each received network data packet in one or more interconnect data packets in accordance with the data protocol in use over the data link of the first interconnect proximal to the media interface.

34. A data processing system as claimed in claim 32, wherein the media interface is configured to transmit the interconnect data packets to the network interface controller in order to cause the network interface controller to at least partially process each received network data packet at the data link layer and write the data carried within each received network data packet to at least one receive queue at the memory identified by a destination network address included within that received network data packet.

35. A data processing system as claimed in claim 34, wherein the central processing unit further comprises a memory management unit configured to provide access to the memory for the central processing unit and the network interface controller,

wherein the memory management unit comprises a cache operable to store a copy of at least some of the data held at the memory, the cache being accessible to both the central processing unit and the network interface controller,

and wherein the network interface controller is configured to, on writing the data carried within a received network data packet to a receive queue at the memory, write that data to the cache.

36. A data processing system as claimed in claim 32, wherein the network interface controller is configured to store at the memory a forwarding table comprising information such as to allow the network interface controller to, for each received network data packet, select in dependence on the destination network address one or more receive queues at the memory and/or another network device to which data of the received network data packet is to be routed.

37. A data processing system as claimed in claim 32, wherein the media interface is configured to write an encapsulation header to each of the interconnect data packets, the encapsulation header comprising information such as to allow the network interface controller to reassemble the respective network data packet from the interconnect data packets.

38. A data processing system as claimed in claim 24, wherein the central processing unit further comprises a memory management unit configured to provide access to the memory for the central processing unit and the network interface controller,

wherein the memory management unit is configured to, on receiving a request to access a virtual memory address from either the central processing unit or the network interface controller, translate that virtual memory address into a physical memory address of the data processing system and access that physical memory address.

39. A data processing system as claimed in claim 38, wherein the memory management unit is configured to, if a translation of a virtual memory address on behalf of the network interface controller fails, raise a page fault to the network interface controller.

40. A data processing system as claimed in claim 39, wherein the network interface controller is configured to, responsive to the page fault:

if the virtual memory address relates to a transmit queue, cause the transmit queue corresponding to that virtual memory address to be paused until the respective page has been rewritten to memory;

or

if the virtual memory address relates to a receive queue, write the data of the received network data packet in respect of which the request was issued to a reserved buffer for retrieval by a protocol processing entity of the data processing system.

41. A data processing system as claimed in claim 13, further comprising:

a second central processing unit including a second network interface controller, the second network interface controller also being configured to form intermediate data units by at least partially framing at the data link layer data for transmission in accordance with the network protocol, each intermediate data unit including a destination network address for the data for transmission; and

a second interconnect extending between the second central processing unit and the media interface, the second interconnect being formed of one or more data links,

wherein the second network interface controller is configured to encapsulate intermediate data units of the network protocol in interconnect data packets for transmission over the second interconnect, and the media interface is configured to extract intermediate data units from interconnect data packets received over the interconnect and process the intermediate data units at the physical layer so as to complete the formation of data packets in accordance with the network protocol.

42. A data processing system as claimed in claim 41, wherein one or more data links of the second interconnect are shared with one or more data links of the first interconnect.

43. A data processing system as claimed in claim 41, wherein the network interface controller is configured to, if the destination network address of intermediate data units formed at the central processing unit is a network address supported at the second central processing unit, cause encapsulated interconnect data packets comprising the intermediate data units to be transmitted to the second network interface controller.

44. A data processing system as claimed in claim 41, wherein the media interface includes a management controller configured to handle management commands received for the network interface devices formed by (a) the media interface and the network interface controller and (b) the media interface and the second network interface controller.

45. A data processing system as claimed in claim 13, wherein the data processing system is a blade server comprising one or more blades supported at a chassis, each blade having one or more central processing units at least one of which includes a network interface controller, and the media interface being located at the chassis of the blade server.

46. A data processing system as claimed in claim 45, wherein the central processing unit and second central processing unit are located at the same blade of the blade server.

47. A data processing system as claimed in claim 46, wherein the central processing unit and second central processing unit are accessible to one another over a CPU interconnect comprising a data link of the first interconnect and a data link of the second interconnect.

48. A data processing system as claimed in claim 45, wherein the central processing unit and second central processing unit are located at different blades of the blade server.

49. A data processing system as claimed in claim 45, wherein the network interface controller is configured to, if the destination network address of intermediate data units formed at the central processing unit is a network address belonging to a process supported at the second central processing unit, cause interconnect data packets encapsulating the intermediate data units to be transmitted over the CPU interconnect to the second network interface controller not via the media interface.

50. A data processing system as claimed in claim 45, wherein the media interface is supported at a peripheral data bus and the blades of the blade server are configured to communicate with one another over the peripheral data bus.

51. A data processing system as claimed in claim 50, further comprising a switch at the chassis of the blade server, the switch being configured to route interconnect data packets between the blades of the blade server.

52. A data processing system as claimed in claim 51, wherein the switch is configured to route interconnect data packets in dependence on their framing information at the level of the data protocol in use over the peripheral data bus.

53. A data processing system as claimed in claim 51, wherein the switch is configured to route interconnect data packets encapsulating intermediate data units in dependence on framing information of the intermediate data units at the level of the network protocol.

54. A data processing system as claimed in claim 51, wherein the peripheral data bus is PCIe and the switch is a PCIe switch.

55. A data processing system as claimed in claim 13, wherein the media interface is one of a regular PCIe device, an SR-IOV PCIe device and an MR-IOV PCIe device.

56. A data processing system as claimed in claim 54, wherein the media interface is an SR-IOV PCIe device and the switch is an MR-IOV PCIe switch.

57. A data processing system as claimed in claim 13, wherein the media interface and each network interface controller of the data processing system are configured to exchange control data by means of one or more interconnect data packets.

58. A data processing system as claimed in claim 13, wherein the network protocol is Ethernet.

59. A method for transmitting data over a network from a data processing system comprising a media interface at the interface to the network, a central processing unit having a memory management unit for accessing a memory, and an interconnect connected between the central processing unit and the media interface, the method comprising:

a network interface controller located at the central processing unit;

retrieving data for transmission over the network from a transmit queue at the memory by means of the memory management unit;

partially framing, according to the data link layer of a network protocol, data packets containing the retrieved data to form intermediate data units; and

in accordance with the data protocol in use over the interconnect, encapsulating the intermediate data packets of the network protocol in data packets for transmission over the interconnect and transmitting the encapsulated intermediate data packets over the interconnect to the media interface,

and the media interface;

receiving the encapsulated intermediate data packets;

extracting the intermediate data packets of the network protocol from the encapsulated intermediate data packets;

processing the intermediate data packets of the network protocol at the physical layer of the network protocol so as to complete the framing of the intermediate data packets in accordance with the network protocol; and

transmitting the fully-formed data packets of the network protocol onto the network.

60. A method for receiving data over a network at a data processing system comprising a media interface at the interface to the network, a central processing unit having a memory management unit for accessing a memory, and an interconnect connected between the central processing unit and the media interface, the method comprising:

the media interface;

receiving network data packets over the network; and

in accordance with the data protocol in use over the interconnect, encapsulating the received network data packets in data packets for transmission over the interconnect and transmitting the data packets over the interconnect to a network interface controller located at the central processing unit,

and the network interface controller;

receiving the encapsulated data packets;

extracting the received network data packets from the encapsulated data packets;

at least partially processing each received network data packet at the data link layer in accordance with the network protocol so as to extract the data carried within each received network data packet; and

writing the extracted data to a receive queue at the memory by means of the memory management unit.

61. A method for communicating data between central processing units at a data processing system having a first central processing unit and a second central processing unit coupled together by means of an interconnect, each central processing unit comprising a respective network interface controller operable to (a) form intermediate data units by at least partially framing data for communication at the data link layer in accordance with a network protocol, and (b) at least partially process at the data link layer in accordance with the network protocol intermediate data units so as to extract the data for communication carried within each intermediate data unit, the method comprising:

the first central processing unit passing data for communication to its first network interface controller,

the first network interface controller;

forming intermediate data units comprising the data for communication by at least partially framing the data for communication at the data link layer in accordance with the network protocol;

encapsulating the intermediate data units of the network protocol in data packets for transmission over the interconnect; and

transmitting the encapsulated data packets over the interconnect to the second network interface controller,

and the second network interface controller;

receiving the encapsulated data packets;

extracting the intermediate data units from the encapsulated data packets;

at least partially processing each intermediate data unit at the data link layer in accordance with the network protocol so as to extract the data carried within each intermediate data unit; and

passing the extracted data to the second central processing unit.

62. A central processing unit for a data processing system, the central processing unit comprising:

a processing core;

a link module for communicating data packets of a predetermined data protocol over a data bus;

a memory management unit operable to retrieve data from a memory coupled to the central processing unit; and

a network interface controller configured to;

retrieve data for transmission over a network by means of the memory management unit;

frame at the data link layer of a network protocol data packets that contain retrieved data; and

cause those packets to be encapsulated in data packets of the predetermined data protocol and communicated over the data bus by means of the link module.

63. A central processing unit as claimed in claim 62, wherein the network interface controller does not frame data packets at the physical layer of the network protocol.

64. A central processing unit for a data processing system, the central processing unit comprising:

a processing core;

a cache;

a memory management unit operable to retrieve data from a memory coupled to the central processing unit; and

a network interface controller configured to at least partially form a series of network data packets for transmission by generating data link layer information for each of those packets, the network interface controller further configured to, when in use, store at least some of its state at a memory coupled to the central processing unit;

wherein the processing core and network interface controller are configured to access a memory coupled to the central processing unit by means of the memory management unit, and the cache is configured to cache data stored at a memory coupled to the central processing unit.

65. A central processing unit as claimed in claim 64, wherein the state of the network interface controller includes one or both of a forwarding table and an address translation table.

66. A peripheral electronic device supported at a data processing system and presenting itself as a PCIe device to the data processing system, the data processing system comprising a central processing unit and a PCIe data bus, and the peripheral electronic device having two components:

a core component located at the central processing unit, the core component being operable to handle I/O requests for the peripheral electronic device, and

a peripheral component accessible to the core component over the PCIe data bus;

wherein the core component is configured to be responsive to I/O requests directed to the peripheral component.

67. A peripheral electronic device as claimed in claim 66, wherein the core component is allocated a hardware address range defining its control aperture and the peripheral component is configured to provide a Base Address Register indicating a range of hardware addresses that includes the base address of the control aperture of the core component such that I/O requests directed to the range of hardware addresses indicated by the Base Address Register are handled at the control aperture of the core component.

68. A peripheral electronic device as claimed in claim 66, wherein the peripheral component is additionally a component of one or more other PCIe devices and the peripheral component is configured to provide an additional Base Address Register for each other PCIe device, each additional Base Address Register being configured such that I/O requests directed to the range of hardware addresses indicated by that Base Address Register are handled at a respective core component of that other PCIe device.

69. A peripheral electronic device as claimed in claim 66, wherein the peripheral component is configured to forward interrupt configuration requests it receives to the core component for handling at the core component.

70. A peripheral electronic device as claimed in claim 66, wherein the peripheral component further provides a legacy Base Address Register supporting a set of legacy registers, the peripheral component being configured to, when a legacy register is read or written, map the read or write to the legacy register to a read or write to the control aperture of the core component such that legacy I/O requests directed to the set of legacy registers are handled at the control aperture of the core component.