System and method for receive queue provisioning

• US 7,508,837 B2
• Filed: 10/17/2003
• Issued: 03/24/2009
• Est. Priority Date: 10/18/2002
• Status: Expired due to Fees

• Current Assignee: Avago Technologies General IP PTE Limited (Broadcom, Inc.)
• Original Assignee: Broadcom Corporation (Broadcom, Inc.)
• Inventors: Elzur, Uri
• Primary Examiner(s): ABELSON, RONALD B
• Application Number:
  US10/688,392
• Publication Number:
  US 20040095949A1
• Time in Patent Office: 1,985 Days
• Field of Search: 370/229, 370412-418, 370/428, 370/468, 370/465, 710/22, 710/23, 710/26, 710/52, 710/54, 709/235, 709/213
• US Class Current: 370/413
• CPC Class Codes: H04L 47/621 Individual queue per connection or flow, e.g. per VC
• CPC Class Codes: H04L 49/90 Buffering arrangements
• CPC Class Codes: H04L 49/9036 Common buffer combined with individual queues
• CPC Class Codes: H04L 69/16 Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
• CPC Class Codes: H04L 69/32 Architecture of open systems interconnection [OSI] 7-layer type protocol stacks, e.g. the interfaces between the data link level and the physical level

Abstract

Systems and methods that provide receive queue provisioning are provided. In one embodiment, a communications system may include, for example, a first queue pair (QP), a second QP and a general pool. The first QP may be associated with a first connection and may include, for example, a first send queue (SQ). The second QP may be associated with a second connection and may include, for example, a second SQ. The general pool may include, for example, a shared receive queue (SRQ) that may be shared, for example, by the first QP and the second QP.

Claims

1. A communications system, comprising:

a first queue pair (QP) associated with a first connection, the first QP comprising a first send queue (SQ);

a second QP associated with a second connection, the second QP comprising a second SQ; and

a general pool comprising a shared receive queue (SRQ), the SRQ being shared by the first QP and the second QP,

wherein the first QP comprises a first limit queue (LQ) that limits an amount of resources of the SRQ that the first QP can access,

wherein the second QP comprises a second limit queue (LQ) that limits an amount of resources of the SRQ that the second QP can access, and

wherein at least one of the first LQ and the second LQ comprises a maximum value relating to a size of a hole in a transport protocol sequence space.

2. The communications system according to claim 1, wherein the first QP and the second QP are part of a particular node.

3. The communications system according to claim 1, wherein the SRQ comprises an amount of resources that is statistically determined.

4. The communications system according to claim 3, wherein the amount of resources is statistically determined based upon empirical resource requirements of the first connection and the second connection.

5. The communications system according to claim 3, wherein the amount of resources is statistically determined based upon dynamic statistics.

6. The communications system according to claim 1, wherein the SRQ comprises a plurality of posted buffers.

7. The communications system according to claim 1, wherein the first QP does not have its own dedicated received queue (RQ).

8. The communications system according to claim 1, wherein the first QP conducts communications over the first connection as if the first QP has access to more resources of the SRQ than a statistical determination of resource requirements of the first connection.

9. The communications system according to claim 1, wherein if the first QP exceeds a limit as set forth in the first LQ, then the first connection is dropped.

10. The communications system according to claim 1, wherein the first LQ and the second LQ are managed locally.

11. The communications system according to claim 10, wherein the first LQ and the second LQ are managed locally without communications with other nodes.

12. The communications system according to claim 1, wherein at least one of the first LQ and the second LQ is a soft limit.

13. The communications system according to claim 12,

wherein, if the soft limit is reached, then a connection behavior of the first connection or the second connection is analyzed before a response is generated, and

wherein the generated response is based on the analyzed connection behavior.

14. The communications system according to claim 1, wherein at least one of the first LQ and the second LQ is a hard limit.

15. The communications system according to claim 14, wherein, if the hard limit is reached, then an automatic response is generated.

16. The communications system according to claim 1, wherein the first LQ or the second LQ comprises a limit that relates to at least one of an in-order message, an out-of-order message, a segment of the in-order message and a segment of the out-of-order message.

17. The communications system according to claim 1, wherein resource allocation for the SRQ is managed locally.

18. The communications system according to claim 1,

wherein the first QP is associated with a first completion queue (CQ), and

wherein the second QP is associated with a second CQ.

19. The communications system according to claim 1, wherein the general pool comprises a shared CQ (SCQ), the SCQ being associated with the first QP and the second QP.

20. The communications system according to claim 1, wherein the general pool comprises a memory translation and protection table (TPT) associated with resources of the general pool.

21. The communications system according to claim 1, wherein at least one of the first connection and the second connection is a remote direct memory access (RDMA) connection.

22. The communications system according to claim 1, wherein the at least one of the first connection and the second connection is an Internet small computer system interface (iSCSI) over RDMA (iSER) connection.

23. A communications system, comprising:

a network interface card interface (NI) comprising a network interface card (NIC) and a NIC driver, the NIC being coupled to the NIC driver; and

a consumer coupled to the NI,

wherein the NIC comprises a first queue pair (QP), a second QP and a shared receive queue (SRQ), and

wherein the first QP and the second QP share the SRQ,

wherein the first QP comprises a first limit queue (LQ) that limits an amount of resources of the SRQ that the first QP can access,

wherein the second QP comprises a second limit queue (LQ) that limits an amount of resources of the SRQ that the second QP can access, and

wherein the first LQ or the second LQ comprises a limit that relates to at least one of an in-order message, an out-of-order message, a segment of the in-order message and a segment of the out-of-order message.

24. The communications system according to claim 23, the consumer communicates with the NI via verbs.

25. The communications system according to claim 23, wherein the consumer comprises a user space application.

26. The communications system according to claim 23, wherein the consumer comprises a kernel space application.

27. The communications system according to claim 23, wherein at least one of the consumer and the NI comprises an SRQ manager that employs statistical provisioning in managing resources of the SRQ.

28. The communications system according to claim 23, wherein the NIC comprises a remote-direct-memory-access-enabled NIC.

29. The communications system according to claim 23, wherein at least one of the consumer and the NI locally manages provisioning of the SRQ using statistical information.

30. The communications system according to claim 23, wherein the first QP comprises a send queue (SQ).

31. The communications system according to claim 23, wherein the consumer comprises a verb consumer.

32. A method for communicating, comprising:

establishing a first connection associated with a first queue pair (QP), the first QP comprising a first limit queue (LQ) that limits an amount of resources of the SRQ that the first QP can access;

establishing a second connection associated with a second QP, the second QP comprising a second limit queue (LQ) that limits an amount of resources of the SRQ that the second QP can access, at least one of the first LQ and the second LQ comprising a maximum value relating to a size of a hole in a transport protocol sequence space;

concurrently sharing a single receive queue (RQ) between the first QP and the second QP; and

provisioning the single RQ using statistical information.

33. The method according to claim 32, comprising:

managing locally resources of the single RQ.

34. The method according to claim 32, comprising:

managing dynamically resources of the single RQ.

35. The method according to claim 32, comprising:

managing locally RQ resources available to the first QP.

36. The method according to claim 35, wherein managing comprises limiting RQ resources available to the first QP.

37. The method according to claim 32, wherein the first QP is used by a user space application.

38. The method according to claim 32, wherein the first QP is used by a kernel space application.

39. A communications system, comprising:

a first queue pair (QP) associated with a first connection, the first QP comprising a first send queue (SQ);

a second QP associated with a second connection, the second QP comprising a second SQ; and

a general pool comprising a shared receive queue (SRQ), the SRQ being shared by the first QP and the second QP,

wherein the first QP comprises a first limit queue (LQ) that limits an amount of resources of the SRQ that the first QP can access,

wherein the second QP comprises a second limit queue (LQ) that limits an amount of resources of the SRQ that the second QP can access,

wherein the first LQ or the second LQ comprises a limit that relates to at least one of an in-order message, an out-of-order message, a segment of the in-order message and a segment of the out-of-order message.

40. A communications system, comprising:

a network interface card interface (NI) comprising a network interface card (NIC) and a NIC driver, the NIC being coupled to the NIC driver; and

a consumer coupled to the NI,

wherein the NIC comprises a first queue pair (QP), a second QP and a shared receive queue (SRQ), and

wherein the first QP and the second QP share the SRQ,

wherein the first QP comprises a first limit queue (LQ) that limits an amount of resources of the SRQ that the first QP can access,

wherein the second QP comprises a second limit queue (LQ) that limits an amount of resources of the SRQ that the second QP can access, and

wherein at least one of the first LQ and the second LQ comprises a maximum value relating to a size of a hole in a transport protocol sequence space.

41. A method for communicating, comprising:

establishing a first connection associated with a first queue pair (QP), the first QP comprising a first limit queue (LQ) that limits an amount of resources of the SRQ that the first QP can access;

establishing a second connection associated with a second QP, the second QP comprising a second limit queue (LQ) that limits an amount of resources of the SRQ that the second QP can access, the first LQ or the second LQ comprising a limit that relates to at least one of an in-order message, an out-of-order message, a segment of the in-order message and a segment of the out-of-order message;

concurrently sharing a single receive queue (RQ) between the first QP and the second QP; and

provisioning the single RQ using statistical information.