Cross-channel network operation offloading for collective operations

• US 8,811,417 B2
• Filed: 11/15/2010
• Issued: 08/19/2014
• Est. Priority Date: 11/15/2009
• Status: Active Grant

• Current Assignee: Mellanox Technologies Limited (NVIDIA Corporation)
• Original Assignee: Mellanox Technologies Limited (NVIDIA Corporation)
• Inventors: Bloch, Noam, Bloch, Gil, Shachar, Ariel, Chapman, Hillel, Rabinovitz, Ishai, Shamis, Pavel, Shainer, Gilad
• Primary Examiner(s): Sheikh, Ayaz
• Assistant Examiner(s): CHAU, PETER P
• Application Number:
  US12/945,904
• Publication Number:
  US 20110119673A1
• Time in Patent Office: 1,373 Days
• Field of Search: None
• US Class Current: 370/412
• CPC Class Codes: G06F 2209/509 Offload
• CPC Class Codes: G06F 9/546 Message passing systems or structures, e.g. queues
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

A Network Interface (NI) includes a host interface, which is configured to receive from a host processor of a node one or more cross-channel work requests that are derived from an operation to be executed by the node. The NI includes a plurality of work queues for carrying out transport channels to one or more peer nodes over a network. The NI further includes control circuitry, which is configured to accept the cross-channel work requests via the host interface, and to execute the cross-channel work requests using the work queues by controlling an advance of at least a given work queue according to an advancing condition, which depends on a completion status of one or more other work queues, so as to carry out the operation.

Claims

1. A Network Interface device (NI), comprising:

a host interface, which is configured to communicate with a host processor of a node of a network;

a network interface, which is configured to be coupled to the network; and

circuitry, which is coupled between the host interface and the network interface and is configured to;

maintain multiple work queues that correspond to transport channels between the node and one or more peer nodes on the network,

receive from the host processor via the host interface one or more cross-channel work requests,

execute work queue entries from the multiple work queues;

disable the execution of work queue entries in a first work queue from the multiple work queues until conditions required for execution of the work queue entries of the first work queue are met, responsive to the received one or more cross-channel work requests; and

enable the execution of work queue entries in the first work queue, responsive to identifying a completion status of one or more second work queues such that the conditions required for execution of the work queue entries of the first work queue are met,

wherein the circuitry is configured to place in the first work queue, responsive to the received one or more cross-channel work requests, a sequence of entries including a wait entry, followed by a calculation entry, which when executed performs a calculation on data received in the one or more second work queues, and to disable the execution of work queue entries in the first work queue when the wait entry is executed.

2. The NI according to claim 1, and comprising a computation unit that is configured to execute a calculation that is specified in the one or more cross-channel work requests, on data received by one or more of the one or more second work queues, wherein the circuitry is configured to send one or more results of the calculation to at least one target.

3. The NI according to claim 2, wherein the calculation comprises at least one operation selected from a group of operations comprising maximum, minimum, sum, product, logical AND, bit-wise AND, logical OR, bit-wise OR, logical exclusive OR (XOR), and bit-wise exclusive OR (XOR).

4. The NI according to claim 2, wherein the calculation comprises arguments of at least one data type selected from a group of data types comprising a vector that is contiguously organized in memory, a vector that is non-contiguously organized in memory and a multidimensional vector.

5. The NI according to claim 2, wherein the at least one target comprises a peer node.

6. The NI according to claim 2, wherein the at least one target comprises a host memory.

7. The NI according to claim 1, wherein the first work queue comprises a send queue.

8. The NI according to claim 1, wherein the first work queue comprises a receive queue, and the circuitry is configured to reject a message received by a disabled receive queue.

9. The NI according to claim 1, wherein the circuitry is configured to identify the completion status of the one or more second work queues according to one or more counting objects that are indicative of the completion status.

10. The NI according to claim 1, wherein the network comprises an Infiniband network and the multiple work queues comprise Infiniband work queues.

11. The NI according to claim 1, and comprising one or more completion queues, wherein the circuitry is configured to estimate the completion status of the one or more second work queues according to a status of the one or more completion queues.

12. The NI according to claim 1, wherein the circuitry is configured to reuse at least one of the multiple work queues for carrying out multiple transport channels over the at least one of the multiple work queues.

13. The NI according to claim 1, wherein the circuitry and the multiple work queues are comprised in a chip-set.

14. The NI according to claim 1, wherein the circuitry and the multiple work queues are comprised in a single chip.

15. The NI according to claim 1, wherein the sequence of entries includes a send entry, following the calculation entry, which sends a result of the calculation entry to a peer node.

16. The NI according to claim 1, wherein the circuitry is configured to enable the execution of work queue entries in the first work queue, responsive to identifying a completion status of a plurality of second work queues.

17. The NI according to claim 1, wherein the circuitry is configured to place an enable entry in the one or more second work queues and to enable the first work queue, responsive to executing the enable entry in the one or more second work queues.

18. The NI according to claim 1, wherein identifying the completion status of the one or more second work queues comprises identifying reception of messages from specific peer nodes associated with the one or more second work queues, such that the circuitry is configured to enable the execution of work queue entries in the first work queue, responsive to receiving messages from the peer nodes.

19. The NI according to claim 1, wherein the circuitry identifies the completion status by receiving an indication through a doorbell of the Network Interface device.

20. The NI according to claim 1, wherein the circuitry identifies the completion status by receiving an interrupt.

21. The NI according to claim 1, wherein the circuitry is configured to disable execution of work queue entries of the first work queue by preventing an index indicating work queue entries for execution from pointing at the disable work queue entries.

22. The NI according to claim 1, wherein the circuitry is configured to enable the execution of work queue entries on the first work queue responsive to receiving a message required for the execution, in the one or more second work queues.

23. A method, comprising:

using a Network Interface device (NI) that is coupled between a network and a host processor of a node, maintaining in the NI multiple work queues that correspond to transport channels between the node and one or more peer nodes on the network;

execute, by the NI, work queue entries from the multiple work queues;

receiving in the NI from the host processor one or more cross-channel work requests,

placing in a first work queue from the multiple work queues, responsive to the received one or more cross-channel work requests, a sequence of entries including a wait entry, followed by a calculation entry;

disabling the execution of work queue entries in the first work queue until conditions required for execution of the work queue entries of the first work queue are met, when the wait entry is executed; and

enabling the execution of work queue entries in the first work queue, responsive to identifying a completion status of one or more second work queues such that the conditions required for execution of the work queue entries of the first work queue are met,

wherein the calculation entry, when executed, performs a calculation on data received in the one or more second work queues.

24. The method according to claim 23, wherein receiving one or more cross-channel work requests comprises receiving an indication of a collective operation to be executed by the node together with the one or more peer nodes.

25. The method according to claim 23, wherein receiving one or more cross-channel work requests comprises receiving instructions to carry out one or more Message Passing Interface (MPI) commands.

26. The method according to claim 25, wherein the one or more MPI commands comprise non-blocking commands.

27. The method according to claim 23, wherein receiving one or more cross-channel work requests comprises receiving instructions to carry out one or more Shared Memory (SHMEM) commands.

28. The method according to claim 23, comprising executing a calculation that is specified in the one or more cross-channel work requests, and sending one or more results of the calculation to at least one target.

29. The method according to claim 28, wherein the calculation comprises at least one operation selected from a group of operations comprising maximum, minimum, sum, product, logical AND, bit-wise AND, logical OR, bit-wise OR, logical exclusive OR (XOR), and bit-wise exclusive OR (XOR).

30. The method according to claim 28, wherein the calculation comprises arguments of at least one data type selected from a group of data types comprising a vector that is contiguously organized in memory, a vector that is non-contiguously organized in memory and a multidimensional vector.

31. The method according to claim 28, wherein the at least one target comprises a peer node.

32. The method according to claim 28, wherein the at least one target comprises a host memory.

33. The method according to claim 23, wherein enabling the execution of work queue entries in the first work queue responsive to identifying the completion status comprises enabling sending to a peer node a message that is stored at a head of the first work queue only upon identifying of the completion status.

34. The method according to claim 23, wherein enabling the execution of work queue entries in the first work queue comprises enabling receiving in the first work queue a message from a peer node.

35. The method according to claim 23, wherein identifying the completion status comprises identifying the completion status of the one or more second work queues according to one or more counting objects that are indicative of the completion status.

36. The method according to claim 23, wherein the network comprises an Infiniband network and the multiple work queues comprise Infiniband work queues.

37. The method according to claim 23, wherein the NI includes one or more completion queues, and wherein identifying the completion status comprises identifying the completion status of the one or more second work queues according to a status of the one or more completion queues.

38. The method according to claim 23, comprising reusing at least one of the multiple work queues for carrying out multiple transport channels over the at least one of the multiple work queues.

39. A computer software product, comprising a non-transitory computer-readable storage medium in which program instructions are stored, which instructions, when read by a processor that is coupled between a network and a host processor of a node, cause the processor:

to maintain multiple work queues that correspond to transport channels between the node and one or more peer nodes on the network,

to execute work queue entries from the multiple work queues,

to receive from the host processor one or more cross-channel work requests,

to place in a first work queue from the multiple work queues, responsive to the received one or more cross-channel work requests, a sequence of entries including a wait entry, followed by a calculation entry,

to disable the execution of work queue entries in the first work queue when the wait entry is executed until conditions required for execution of the work queue entries of the first work queue are met, and

to enable the execution of work queue entries in the first work queue, responsive to identifying a completion status of one or more second work queues, such that the conditions required for execution of the work queue entries of the first work queue are met,

wherein the calculation entry, when executed, performs a calculation on data received in the one or more second work queues.