Host bus access by add-on devices via a network interface controller

• US 10,152,441 B2
• Filed: 05/14/2016
• Issued: 12/11/2018
• Est. Priority Date: 05/18/2015
• Status: Active Grant

• Current Assignee: Mellanox Technologies Limited (NVIDIA Corporation)
• Original Assignee: Mellanox Technologies Limited (NVIDIA Corporation)
• Inventors: Liss, Liran, Raindel, Shachar, Raikin, Shlomo, Menachem, Adi, Itkin, Yuval
• Primary Examiner(s): Peyton, Tammara R
• Application Number:
  US15/154,945
• Publication Number:
  US 20160342547A1
• Time in Patent Office: 941 Days
• CPC Class Codes: G06F 13/28 using burst mode transfer, e.g. direct memory access DMA, cycle steal G06F13/32 takes precedence
• CPC Class Codes: G06F 13/385 for adaptation of a particular data processing system to different peripheral devices
• CPC Class Codes: G06F 13/4022 using switching circuits, e.g. switching matrix, connection or expansion network G06F13/4009 takes precedence
• CPC Class Codes: G06F 13/4282 on a serial bus, e.g. I2C bus, SPI bus on daisy chain buses G06F13/4247
• CPC Class Codes: H04L 49/3009 Header conversion, routing tables or routing tags
• CPC Class Codes: H04L 49/9068 in the network interface card

Abstract

Peripheral apparatus for use with a host computer includes an add-on device, which includes a first network port coupled to one end of a packet communication link and add-on logic, which is configured to receive and transmit packets containing data over the packet communication link and to perform computational operations on the data. A network interface controller (NIC) includes a host bus interface, configured for connection to the host bus of the host computer and a second network port, coupled to the other end of the packet communication link. Packet processing logic in the NIC is coupled between the host bus interface and the second network port, and is configured to translate between the packets transmitted and received over the packet communication link and transactions executed on the host bus so as to provide access between the add-on device and the resources of the host computer.

Claims

1. Peripheral apparatus for use with a host computer that includes a host bus and host resources, including a host processor, connected to the host bus, the apparatus comprising:

a packet communication link having first and second ends;

an add-on device, which comprises;

a first network port coupled to the first end of the packet communication link; and

add-on logic, which is configured to receive and transmit packets containing data over the packet communication link and to perform computational operations on the data; and

a network interface controller (NIC), which comprises;

a host bus interface, configured for connection to the host bus;

a second network port, coupled to the second end of the packet communication link; and

packet processing logic, which is coupled between the host bus interface and the second network port, and is configured to translate between the packets transmitted and received over the packet communication link and transactions executed on the host bus so as to provide access between the add-on device and the host resources,

wherein the host resources comprise a system memory, and wherein the NIC comprises a direct memory access (DMA) engine, which is configured to write and read data to and from the system memory via the host bus, including the data in the packets transmitted from the add-on device over the packet communication link.

5. The apparatus according to claim 1, wherein the first and second network ports are configured to transmit the packets over the packet communication link in accordance with an Ethernet protocol.

6. The apparatus according to claim 1, wherein a range of addresses on the host bus is assigned to the host bus interface, and a sub-range within the assigned range is assigned to the add-on device, and wherein the packet processing logic is configured to translate the transactions on the host bus that are directed to the addresses in the sub-range into corresponding packets for transmission over the packet communication link to the add-on device.

7. The apparatus according to claim 1, wherein the add-on logic and the packet processing logic are configured to transmit and receive the packets over the packet communication link in accordance with a remote direct memory access (RDMA) protocol, which specifies addresses in the system memory to and from which the data in the packets are to be written and read by the DMA engine.

2. Peripheral apparatus for use with a host computer that includes a host bus and host resources, including a host processor, connected to the host bus, the apparatus comprising:

a packet communication link having first and second ends;

an add-on device, which comprises;

a first network port coupled to the first end of the packet communication link; and

add-on logic, which is configured to receive and transmit packets containing data over the packet communication link and to perform computational operations on the data; and

a network interface controller (NIC), which comprises;

a host bus interface, configured for connection to the host bus;

a second network port, coupled to the second end of the packet communication link;

packet processing logic, which is coupled between the host bus interface and the second network port, and is configured to translate between the packets transmitted and received over the packet communication link and transactions executed on the host bus so as to provide access between the add-on device and the host resources; and

a third network port, which is configured to convey communications between the host computer and a packet data network while the second network port communicates with the add-on device over the packet communication link.

3. Peripheral apparatus for use with a host computer that includes a host bus and host resources, including a host processor, connected to the host bus, the apparatus comprising:

a packet communication link having first and second ends;

an add-on device, which comprises;

a first network port coupled to the first end of the packet communication link; and

add-on logic, which is configured to receive and transmit packets containing data over the packet communication link and to perform computational operations on the data; and

a network interface controller (NIC), which comprises;

a host bus interface, configured for connection to the host bus;

a second network port, coupled to the second end of the packet communication link; and

packet processing logic, which is coupled between the host bus interface and the second network port, and is configured to translate between the packets transmitted and received over the packet communication link and transactions executed on the host bus so as to provide access between the add-on device and the host resources,

wherein the add-on device comprises a switch, which is coupled between the first network port and the add-on logic, and is configured to convey communications, transmitted and received via the second network port, between the host computer and a packet data network, while transferring the packets to and from the add-on logic.

4. Peripheral apparatus for use with a host computer that includes a host bus and host resources, including a host processor, connected to the host bus, the apparatus comprising:

a packet communication link having first and second ends;

an add-on device, which comprises;

a first network port coupled to the first end of the packet communication link; and

add-on logic, which is configured to receive and transmit packets containing data over the packet communication link and to perform computational operations on the data;

a network interface controller (NIC), which comprises;

a host bus interface, configured for connection to the host bus;

a second network port, coupled to the second end of the packet communication link; and

packet processing logic, which is coupled between the host bus interface and the second network port, and is configured to translate between the packets transmitted and received over the packet communication link and transactions executed on the host bus so as to provide access between the add-on device and the host resources; and

a switch, which comprises;

a third port, which is connected to the first end of the packet communication link;

a fourth port, which is connected to communicate with the first port of the add-on device; and

a fifth port, which is connected to communicate with another entity over a further packet communication link.

8. Peripheral apparatus for use with a host computer that includes a host bus and host resources, including a host processor, connected to the host bus, the apparatus comprising:

a packet communication link having first and second ends;

an add-on device, which comprises;

a first network port coupled to the first end of the packet communication link; and

add-on logic, which is configured to receive and transmit packets containing data over the packet communication link and to perform computational operations on the data; and

a network interface controller (NIC), which comprises;

a host bus interface, configured for connection to the host bus;

a second network port, coupled to the second end of the packet communication link; and

packet processing logic, which is coupled between the host bus interface and the second network port, and is configured to translate between the packets transmitted and received over the packet communication link and transactions executed on the host bus so as to provide access between the add-on device and the host resources,

wherein the host resources comprise a system memory, and wherein the packet processing logic is configured to read and execute work items posted in work queues in the system memory by software processes running on the host computer, including at least one queue containing add-on work items specifying operations to be performed by the add-on device, wherein execution of the add-on work items causes the packet processing logic to transmit, over the packet communication link, at least one packet that causes the add-on logic to perform the specified operations.

9. The apparatus according to claim 8, wherein the at least one queue comprises two or more queues containing different, respective add-on work items directed to the add-on device, and wherein the NIC comprises scheduling logic, which is configured to transmit the packets corresponding to the work items in the two or more queues in accordance with predefined scheduling criteria.

10. The apparatus according to claim 9, wherein the scheduling criteria comprise different, respective levels of quality of service that are assigned to the two or more queues.

11. The apparatus according to claim 8, wherein the add-on logic is configured, upon completion of the specified operations, to transmit a reporting message over the packet communication link, and wherein the NIC logic is configured, in response to the reporting message, to post a completion item in a completion queue in the system memory, to be accessed by the software processes running on the host processor.

12. The apparatus according to claim 8, wherein the add-on logic is configured, upon completion of the specified operations, to write a completion status to the system memory using a remote direct memory access (RDMA) protocol.

13. A computing method, comprising:

connecting a first network port of an add-on device to a second network port of a network interface controller (NIC) via a packet communication link;

connecting a host bus interface of the NIC to a host bus of a host computer comprising host resources, including a host processor, connected to the host bus;

in the add-on device, receiving and transmitting packets containing data from and to the NIC over the packet communication link and performing computational operations on the data; and

translating, in the NIC, between the packets transmitted and received over the packet communication link and transactions executed on the host bus so as to provide access between the add-on device and the host resources,

wherein the host resources comprise a system memory, and wherein translating between the packets and the transactions comprises writing and reading the data in the packets transmitted from and to the add-on device over the packet communication link to and from the system memory via the host bus using a direct memory access (DMA) engine in the NIC.

15. The method according to claim 13, wherein the add-on device comprises a switch, which is coupled between the first network port and the add-on logic, and wherein the method comprises conveying communications via the second network port and the switch between the host computer and a packet data network, while transferring the packets via the switch to and from the add-on logic.

16. The method according to claim 13, wherein connecting the first network port comprises connecting the packet communication link between the second network port of the NIC and a third port of a switch while connecting a fourth port of the switch to communicate with the first port of the add-on device, and

wherein the method comprises connecting a fifth port of the switch to communicate with another entity over a further packet communication link.

17. The method according to claim 13, wherein receiving and transmitting the packets comprises conveying the packets over the packet communication link in accordance with an Ethernet protocol.

19. The method according to claim 13, wherein receiving and transmitting the packets comprises exchanging the packets over the packet communication link between the add-on device and the NIC in accordance with a remote direct memory access (RDMA) protocol, which specifies addresses in the system memory to and from which the data in the packets are to be written and read by the DMA engine.

14. A computing method, comprising:

connecting a first network port of an add-on device to a second network port of a network interface controller (NIC) via a packet communication link;

connecting a host bus interface of the NIC to a host bus of a host computer comprising host resources, including a host processor, connected to the host bus;

in the add-on device, receiving and transmitting packets containing data from and to the NIC over the packet communication link and performing computational operations on the data;

translating, in the NIC, between the packets transmitted and received over the packet communication link and transactions executed on the host bus so as to provide access between the add-on device and the host resources; and

conveying communications between the host computer and a packet data network via a third network port of the NIC while the second network port communicates with the add-on device over the packet communication link.

18. A computing method, comprising:

connecting a first network port of an add-on device to a second network port of a network interface controller (NIC) via a packet communication link;

connecting a host bus interface of the NIC to a host bus of a host computer comprising host resources, including a host processor, connected to the host bus;

in the add-on device, receiving and transmitting packets containing data from and to the NIC over the packet communication link and performing computational operations on the data; and

translating, in the NIC, between the packets transmitted and received over the packet communication link and transactions executed on the host bus so as to provide access between the add-on device and the host resources,

wherein a range of addresses on the host bus is assigned to the host bus interface, and wherein the method comprises assigning a sub-range within the assigned range to the add-on device, and

wherein translating between the packets and the transactions comprises translating the transactions on the host bus that are directed to the addresses in the sub-range into corresponding packets for transmission over the packet communication link to the add-on device.

20. A computing method, comprising:

connecting a first network port of an add-on device to a second network port of a network interface controller (NIC) via a packet communication link;

connecting a host bus interface of the NIC to a host bus of a host computer comprising host resources, including a host processor, connected to the host bus;

in the add-on device, receiving and transmitting packets containing data from and to the NIC over the packet communication link and performing computational operations on the data; and

translating, in the NIC, between the packets transmitted and received over the packet communication link and transactions executed on the host bus so as to provide access between the add-on device and the host resources,

wherein the host resources comprise a system memory, and wherein the method includes reading and executing in the NIC work items posted in work queues in the system memory by software processes running on the host computer, including at least one queue containing add-on work items specifying operations to be performed by the add-on device,

wherein execution of the add-on work items causes NIC to transmit, over the packet communication link, at least one packet that causes the add-on device to perform the specified operations.

21. The method according to claim 20, wherein the at least one queue comprises two or more queues containing different, respective reading and executing the work items comprises applying scheduling logic in the NIC to transmit the packets corresponding to the work items in the two or more queues in accordance with predefined scheduling criteria.

22. The method according to claim 21, wherein the scheduling criteria comprise different, respective levels of quality of service that are assigned to the two or more queues.

23. The method according to claim 20, and comprising, upon completion of the specified operations in the add-on device, transmitting a reporting message from the add-on device over the packet communication link to the NIC, and

wherein translating the packets comprises, in response to the reporting message, posting a completion item from the NIC to a completion queue in the system memory, to be accessed by the software processes running on the host processor.

24. The method according to claim 20, and comprising, upon completion of the specified operations in the add-on device, writing a completion status from the add-on device to the system memory using a remote direct memory access (RDMA) protocol.