Queue pair partitioning in distributed computer system

• US 6,718,392 B1
• Filed: 10/24/2000
• Issued: 04/06/2004
• Est. Priority Date: 10/24/2000
• Status: Expired due to Fees

• Current Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Krause, Michael R.
• Primary Examiner(s): Harrell, Robert B.
• Application Number:
  US09/695,634
• Time in Patent Office: 1,260 Days
• Field of Search: 709/200, 709/201, 709/220, 709/221, 709/222, 709/223, 709/238, 709/249
• US Class Current: 709/238
• CPC Class Codes: H04L 2101/677 Multiple interfaces, e.g. multihomed nodes
• CPC Class Codes: H04L 61/35 involving non-standard use of addresses for implementing network functionalities, e.g. coding subscription information within the address or functional addressing, i.e. assigning an address to a function
• CPC Class Codes: H04L 63/104 Grouping of entities
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 69/22 Parsing or analysis of headers

Abstract

A distributed computer system has a first subnet including a first group of endnodes and a second subnet including a second group of endnodes. Each endnode in the first and second groups of endnodes includes at least one process which produces and/or consumes message data, and queue pairs. Each queue pair includes a send work queue having work queue elements that describe message data for sending, and a receive work queue having work queue elements that describe where to place incoming message data. A communication fabric is physically coupled to the first group of endnodes and the second group of endnodes. A partitioning mechanism associates a first partition key representing endnodes in a first partition to a first group of queue pairs and a second partition key representing endnodes in a second partition to a second group of queue pairs for enabling communication between endnodes over the communication fabric.

Claims

1. A distributed computer system comprising:

a first subnet including a first group of endnodes and a second subnet including a second group of endnodes, each endnode in the first and second groups of endnodes including;

at least one process which produces and/or consumes message data; and

queue pairs, each queue pair including;

a send work queue having work queue elements that describe message data for sending, and

a receive work queue having work queue elements that describe where to place incoming message data;

a communication fabric physically coupled to the first group of endnodes and the second group of endnodes; and

a partitioning mechanism associating a first partition key representing endnodes in a first partition to a first group of queue pairs and a second partition key representing endnodes in a second partition to a second group of queue pairs for enabling communication between endnodes over the communication fabric.

2. The distributed computer system of claim 1, wherein each subnet includes a partition manager providing definition and setup of partitions on the subnet.

3. The distributed computer system of claim 2, wherein at least one subnet includes multiple partition managers.

4. The distributed computer system of claim 2, wherein at least one partition manager spans a plurality of subnets, and each of the plurality of subnets includes a partition manager agent being controlled by the partition manager to manage partitions on the subnet.

5. The distributed computer system of claim 1, wherein the partitioning mechanism inserts a partition key associated to a send work queue of a source endnode in a frame sent from the send work queue, stores a partition key at a destination endnode, and compares the partition key in the frame to the stored partition key at the destination endnode.

6. The distributed computer system of claim 5, wherein if the partition key in the frame matches the stored partition key at the destination endnode, the frame is accepted and processed normally.

7. The distributed computer system of claim 5, wherein if the partition key in the frame does not match the stored partition key at the destination endnode the frame is rejected.

8. The distributed computer system of claim 1, wherein the at least one endnode includes queue pairs in different partitions.

9. The distributed computer system of claim 1, wherein the at least one endnode includes queue pairs in the same partition.

10. The distributed computer system of claim 1, wherein a queue pair must have a partition key associated to the queue pair before the queue pair is used.

11. The distributed computer system of claim 1, wherein the endnodes include ports and the ports each include a partition key table having at least one partition key entry.

12. The distributed computer system of claim 1, wherein the endnodes include ports and the ports each include a partition key table having at least one partition key entry, wherein each subnet includes a partition manager providing definition and setup of partitions on the subnet, and wherein the partition manager controls the content of the partition key table.

13. A method of partitioning endnodes in a distributed computer system including a first subnet having a first group of endnodes, a second subnet having a second group of endnodes, and a communication fabric physically coupled to the first group of endnodes and the second group of endnodes, the method comprising:

producing message data at source endnodes;

consuming messages at destination endnodes;

establishing queue pairs at each endnode, each queue pair including;

a send work queue having work queue elements that describe message data for sending, and

a receive work queue having work queue elements that describe where to place incoming message data; and

associating a first partition key representing endnodes in a first partition to a first group of queue pairs and associating a second partition key representing endnodes in a second partition to a second group of queue pairs for enabling communication between endnodes over the communication fabric.

14. The method of claim 13 further comprising:

inserting a partition key associated to a send work queue of a source endnode in a frame sent from the send work queue;

storing a partition key at a destination endnode; and

comparing the partition key in the frame to the stored partition key at the destination endnode.

15. The method of claim 14, further comprising:

accepting the frame and processing the frame normally if the partition key in the frame matches the stored partition key at the destination endnode.

16. The method of claim 14, further comprising:

rejecting the frame if the partition key in the frame does not match the stored partition key at the destination endnode.

17. The method of claim 13 further comprising:

defining and establishing partitions on each subnet with a partition manager.

18. The method of claim 17 further comprising:

defining and establishing partitions on at least one subnet with multiple partition managers.

19. The method of claim 13, wherein the at least one endnode includes queue pairs in different partitions.

20. The method of claim 13, wherein the at least one endnode includes queue pairs in the same partition.

21. The method of claim 13, wherein a queue pair must have a partition key associated to the queue pair before the queue pair is used.

22. The method of claim 13, wherein the endnodes include ports and the method further comprises:

establishing a partition key table having at least one partition key entry at each port.

23. The method of claim 13 further comprising:

managing partitions on a plurality of subnets via one partition manager which controls a partition manager agent on each of the plurality of subnets.

24. The method of claim 13, wherein the endnodes include ports and the method further comprises:

establishing a partition key table having at least one partition key entry at each port;

defining and establishing partitions on each subnet with a partition manager; and

controlling the content of-the partition key table with the partition manager.