NUMA-AWARE SCALING FOR NETWORK DEVICES

• US 20110153935A1
• Filed: 12/17/2009
• Published: 06/23/2011
• Est. Priority Date: 12/17/2009
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Li, Yadong
• Granted Patent: US 8,446,824 B2
• Time in Patent Office: Days
• US Class Current: 711/118
• CPC Class Codes: G06F 15/17331 Distributed shared memory [DSM], e.g. remote direct memory access [RDMA]
• CPC Class Codes: H04L 45/00 Routing or path finding of packets in data switching networks routing or path finding in wireless networks H04W40/00
• CPC Class Codes: H04L 49/9036 Common buffer combined with individual queues

Abstract

The present disclosure describes a method and apparatus for network traffic processing in a non-uniform memory access architecture system. The method includes allocating a Tx/Rx Queue pair for a node, the Tx/Rx Queue pair allocated in a local memory of the node. The method further includes routing network traffic to the allocated Tx/Rx Queue pair. The method may include designating a core in the node for network traffic processing. Of course, many alternatives, variations and modifications are possible without departing from this embodiment.

Claims

1. A method for distributing network traffic in a non-uniform memory access (NUMA) architecture system comprising a plurality of nodes, each node comprising a plurality cores, and a node local memory associated with each node, said method comprising:

allocating at least one Tx/Rx Queue pair in each node local memory, the at least one Tx/Rx Queue pair configured to store one or more packets associated with an application, wherein the application is assigned to at least one said core in the associated node; and

routing received network traffic to at least one allocated Tx/Rx Queue pair based, at least in part, on a packet flow ID related to the received network traffic.

2. The method of claim 1, further comprising designating one core of the plurality of cores for each allocated Tx/Rx Queue pair in each associated node for receiving an interrupt associated with network traffic.

3. The method of claim 1, further comprising sharing the at least one Tx/Rx Queue pair between threads running on one or more of the cores in the associated node.

4. The method of claim 1, further comprising determining a number of Tx/Rx Queue pairs based, at least in part, on a network adapter speed.

5. The method of claim 1, further comprising:

determining a number of nodes in the NUMA system; and

determining a number of cores in each node of the number of nodes.

6. The method of claim 1, further comprising:

receiving an indication that an application running on at least one of the cores in one of the plurality of nodes has data or a message to transmit;

identifying one of the at least one core and identifying the one of the plurality of nodes;

selecting one Tx/Rx Queue pair from the at least one Tx/Rx Queue pair allocated in the associated node local memory for the identified node; and

providing one or more packets corresponding to the data or message to the selected Tx/Rx Queue pair.

7. The method of claim 6, wherein the selecting comprises a round robin scheme if a plurality of Tx/Rx Queue pairs is allocated in the associated node local memory of the identified node.

8. The method of claim 6, wherein, if a plurality of Tx/Rx Queue pairs is allocated in the associated node local memory of the identified node, the selecting comprises using a hashing algorithm and is based, at least in part, on the packet flow ID.

9. The method of claim 2, further comprising:

receiving a packet;

identifying the packet flow;

identifying one node of the plurality of nodes and one Tx/Rx Queue pair of the at least one Tx/Rx Queue pair allocated in the associated node local memory for the one node of the plurality of nodes, based, at least in part, on the packet flow ID;

providing the packet to the identified Tx/Rx Queue pair; and

providing an interrupt to the designated core in the identified node.

10. A non-uniform memory access (NUMA) architecture system comprising:

a plurality of nodes;

a node local memory associated with each node, each node local memory configured to store at least one Tx/Rx Queue pair allocated for the associated node; and

a network adapter in communication with the plurality of nodes, the network adapter configured to transmit and receive network traffic and to route the received network traffic to at least one allocated Tx/Rx Queue pair based, at least in part, on a packet flow ID related to the received network traffic,

wherein each node of the plurality of nodes comprises;

a processor comprising a plurality of cores,

a memory controller coupled to the processor, the memory controller configured to couple the plurality of cores to the node local memory, and

a cache memory coupled to the processor.

11. The NUMA system of claim 10, wherein a number of Tx/Rx Queue pairs allocated for each associated node is determined based, at least in part, on a speed of the network adapter.

12. The NUMA system of claim 10, wherein at least one of the plurality of cores in each respective node is configured to receive an interrupt associated with the network traffic.

13. The NUMA system of claim 10, wherein the network adapter is configured to:

identify the packet flow, identify one node of the plurality of nodes and one Tx/Rx Queue pair of the least one Tx/Rx Queue pair allocated in the associated node local memory for the one node of the plurality of nodes, based, at least in part, on the packet flow ID, provide the received network traffic to the identified Tx/Rx Queue pair, and provide an interrupt to a designated core in the identified node.

14. A system comprising one or more storage mediums having stored thereon, individually or in combination, instructions that when executed by one or more processors a non-uniform memory access (NUMA) architecture system comprising a plurality of nodes, each node comprising a plurality cores, and a node local memory associated with each node, results in the following:

allocating at least one Tx/Rx Queue pair in each node local memory, the at least one Tx/Rx Queue pair configured to store one or more packets associated with an application, wherein the application is assigned to at least one said core in the associated node; and

routing received network traffic to at least one allocated Tx/Rx Queue pair based, at least in part, on a packet flow ID related to the received network traffic.

15. The system of claim 14, wherein the instructions that when executed by one or more of the processors result in the following additional operations comprising:

designating one core of the plurality of cores for each allocated Tx/Rx Queue pair in each associated node for receiving an interrupt associated with network traffic.

16. The system of claim 14, wherein the instructions that when executed by one or more of the processors result in the following additional operations comprising:

sharing the at least one Tx/Rx Queue pair between threads running on one or more of the cores in the associated node.

17. The system of claim 14, wherein the instructions that when executed by one or more of the processors result in the following additional operations comprising:

determining a number of Tx/Rx Queue pairs based, at least in part, on a network adapter speed.

18. The system of claim 14, wherein the instructions that when executed by one or more of the processors result in the following additional operations comprising:

determining a number of nodes in the NUMA system; and

determining a number of cores in each respective node of the number of nodes.

19. The system of claim 14, wherein the instructions that when executed by one or more of the processors result in the following additional operations comprising:

receiving an indication that an application running on at least one of the cores in one of the plurality of nodes has data or a message to transmit;

identifying one of the at least one core and identifying the one of the plurality of nodes;

selecting one Tx/Rx Queue pair from the at least one Tx/Rx Queue pair allocated in the associated node local memory for the identified node; and

providing one or more packets corresponding to the data or message to the selected Tx/Rx Queue pair.

20. The system of claim 15, wherein the instructions that when executed by one or more of the processors result in the following additional operations comprising:

identifying the packet flow;

identifying one node of the plurality of nodes and one Tx/Rx Queue pair of the at least one Tx/Rx Queue pair allocated in the associated node local memory for the one node of the plurality of nodes, based, at least in part, on the packet flow ID;

providing a received packet to the identified Tx/Rx Queue pair; and

providing an interrupt to the designated core in the identified node.