Global Queue Pair Management in a Point-to-Point Computer Network

• US 20130097600A1
• Filed: 10/18/2011
• Published: 04/18/2013
• Est. Priority Date: 10/18/2011
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Cardona, Omar, Jain, Vinit, Recio, Renato J., Sharma, Rakesh
• Granted Patent: US 8,782,128 B2
• Time in Patent Office: Days
• US Class Current: 718/1
• CPC Class Codes: G06F 2009/45595 Network integration; Enabling network access in virtual machine instances
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects
• CPC Class Codes: H04L 47/20 Traffic policing
• CPC Class Codes: H04L 63/20 for managing network security; network security policies in general filtering policies H04L63/0227

Abstract

An approach is provided in which a local module receives a data frame initiated by a first virtual machine and has a target destination at a second virtual machine, which executes on a destination host system. The local module identifies a destination local port ID and a destination global queue pair number corresponding to the second virtual machine. In one embodiment, the destination local port ID corresponds to the destination host, but the destination global queue pair number is independent of the destination host. The local module includes the destination global queue pair number and the destination local port ID in an overlay header and encapsulates the data frame with the overlay header, which results in an encapsulated frame. In turn, the local module sends the encapsulated frame through a computer network to the second virtual machine.

Claims

1. A method comprising:

receiving, at a local module, a data frame initiated by a first virtual machine, wherein the data frame has a target destination at a second virtual machine that executes on a destination host;

identifying a destination global queue pair number and a destination local port ID that correspond to the second virtual machine, wherein the destination global queue pair number is independent of the destination host;

including the destination global queue pair number and the destination local port ID in an overlay header;

encapsulating the data frame with the overlay header, the encapsulating resulting in an encapsulated frame; and

sending the encapsulated frame through a computer network to the second virtual machine.

2. The method of claim 1 wherein the first virtual machine executes on a source host and wherein the source host and the destination host are included in a computer subnetwork, the method further comprising:

migrating the second virtual machine from the destination host to a new destination host that is included in the computer subnetwork; and

updating, by a distributed policy service, a subnetwork table entry corresponding to the destination global queue pair number, wherein the updating includes maintaining association of the destination global queue pair number to the first virtual machine and replacing the destination local port ID with a subsequent destination local port ID that corresponds to the new destination host.

3. The method of claim 2 further comprising:

determining, by the local module, that the destination local port ID is not valid;

querying the distributed policy service by the local module for an updated local port ID that corresponds to the destination global queue pair number;

receiving the updated destination local port ID from the distributed policy service at the local module in response to the querying; and

storing the updated destination local port ID in a local table entry corresponding to the destination global queue pair number.

4. The method of claim 2 further comprising:

receiving an allocation request from a third virtual machine at the local module for a new global queue pair number, the third virtual machine executing on the source host;

querying the distributed policy service for the new global queue pair number;

receiving, from the distributed policy service, the new global queue pair number at the local module;

identifying a source local port ID that corresponds to an adapter located on the source host;

generating a local table entry in a local storage area that includes the new global queue pair number and the source local port ID; and

notifying the distributed policy service of the source local port ID that corresponds to the new global queue pair number.

5. The method of claim 4 further comprising:

receiving a de-allocation request from the third virtual machine at the local module to de-allocate the new global queue pair number;

removing the local table entry from the local storage area; and

notifying the distributed policy service to de-allocate the new global queue pair number from the third virtual machine.

6. The method of claim 2 wherein the computer subnetwork includes a plurality of virtual domains that each correspond to one of a plurality of heterogeneous tenants, and wherein the source host and the destination host belong to one of the plurality of virtual domains, the method further comprising:

identifying one or more physical path translations that are based upon a logical connectivity between the first virtual machine and the second virtual machine, wherein the logical connectivity is independent of one or more physical topology constraints of the computer subnetwork; and

including the one or more physical path translations in the overlay header.

7. The method of claim 1 further comprising:

receiving, at the destination host, the encapsulated frame;

decapsulating the encapsulated frame and extracting the destination global queue pair number from the overlay header;

identifying a virtual function that corresponds to the destination global queue pair number; and

forwarding the data frame to the second virtual machine through the identified virtual function.

8. The method of claim 1 wherein:

the data frame includes a local routing header, a base transport header, and is devoid of a global routing header; and

the destination global queue pair number corresponds to a send queue, a receive queue, and a completion queue associated with the second virtual machine.

9. An information handling system comprising:

one or more processors;

a memory coupled to at least one of the processors;

a set of computer program instructions stored in the memory and executed by at least one of the processors in order to perform actions of;

receiving, at a local module, a data frame initiated by a first virtual machine, wherein the data frame has a target destination at a second virtual machine that executes on a destination host;

identifying a destination global queue pair number and a destination local port ID that correspond to the second virtual machine, wherein the destination global queue pair number is independent of the destination host;

including the destination global queue pair number and the destination local port ID in an overlay header;

encapsulating the data frame with the overlay header, the encapsulating resulting in an encapsulated frame; and

sending the encapsulated frame through a computer network to the second virtual machine.

10. The information handling system of claim 9 wherein the first virtual machine executes on a source host and wherein the source host and the destination host are included in a computer subnetwork, the processors performing additional actions comprising:

migrating the second virtual machine from the destination host to a new destination host that is included in the computer subnetwork; and

updating, by a distributed policy service, a subnetwork table entry corresponding to the destination global queue pair number, wherein the updating includes maintaining association of the destination global queue pair number to the first virtual machine and replacing the destination local port ID with a subsequent destination local port ID that corresponds to the new destination host.

11. The information handling system of claim 10 wherein the processors perform additional actions comprising:

determining, by the local module, that the destination local port ID is not valid;

querying the distributed policy service by the local module for an updated local port ID that corresponds to the destination global queue pair number;

receiving the updated destination local port ID from the distributed policy service at the local module in response to the querying; and

storing the updated destination local port ID in a local table entry corresponding to the destination global queue pair number.

12. The information handling system of claim 10 wherein the processors perform additional actions comprising:

receiving an allocation request from a third virtual machine at the local module for a new global queue pair number, the third virtual machine executing on the source host;

querying the distributed policy service for the new global queue pair number;

receiving, from the distributed policy service, the new global queue pair number at the local module;

identifying a source local port ID that corresponds to an adapter located on the source host;

generating a local table entry in a local storage area that includes the new global queue pair number and the source local port ID; and

notifying the distributed policy service of the source local port ID that corresponds to the new global queue pair number.

13. The information handling system of claim 12 wherein the processors perform additional actions comprising:

receiving a de-allocation request from the third virtual machine at the local module to de-allocate the new global queue pair number;

removing the local table entry from the local storage area; and

notifying the distributed policy service to de-allocate the new global queue pair number from the third virtual machine.

14. The information handling system of claim 10 wherein the computer subnetwork includes a plurality of virtual domains that each correspond to one of a plurality of heterogeneous tenants, and wherein the source host and the destination host belong to one of the plurality of virtual domains, the processors performing additional actions comprising:

identifying one or more physical path translations that are based upon a logical connectivity between the first virtual machine and the second virtual machine, wherein the logical connectivity is independent of one or more physical topology constraints of the computer subnetwork; and

including the one or more physical path translations in the overlay header.

15. The information handling system of claim 9 wherein the processors perform additional actions comprising:

receiving, at the destination host, the encapsulated frame;

decapsulating the encapsulated frame and extracting the destination global queue pair number from the overlay header;

identifying a virtual function that corresponds to the destination global queue pair number; and

forwarding the data frame to the second virtual machine through the identified virtual function.

16. The information handling system of claim 9 wherein:

the data frame includes a local routing header, a base transport header, and is devoid of a global routing header; and

the destination global queue pair number corresponds to a send queue, a receive queue, and a completion queue associated with the second virtual machine.

17. A computer program product stored in a computer readable storage medium, comprising computer program code that, when executed by an information handling system, causes the information handling system to perform actions comprising:

receiving, at a local module, a data frame initiated by a first virtual machine, wherein the data frame has a target destination at a second virtual machine that executes on a destination host;

identifying a destination global queue pair number and a destination local port ID that correspond to the second virtual machine, wherein the destination global queue pair number is independent of the destination host;

including the destination global queue pair number and the destination local port ID in an overlay header;

encapsulating the data frame with the overlay header, the encapsulating resulting in an encapsulated frame; and

sending the encapsulated frame through a computer network to the second virtual machine.

18. The computer program product of claim 17 wherein the first virtual machine executes on a source host and wherein the source host and the destination host are included in a computer subnetwork, the information handling system performing additional actions comprising:

migrating the second virtual machine from the destination host to a new destination host that is included in the computer subnetwork; and

updating, by a distributed policy service, a subnetwork table entry corresponding to the destination global queue pair number, wherein the updating includes maintaining association of the destination global queue pair number to the first virtual machine and replacing the destination local port ID with a subsequent destination local port ID that corresponds to the new destination host.

19. The computer program product of claim 18 wherein the information handling system performs additional actions comprising:

determining, by the local module, that the destination local ID is not valid;

querying the distributed policy service by the local module for an updated local port ID that corresponds to the destination global queue pair number;

receiving the updated destination local port ID from the distributed policy service at the local module in response to the querying; and

storing the updated destination local port ID in a local table entry corresponding to the destination global queue pair number.

20. The computer program product of claim 18 wherein the information handling system performs additional actions comprising:

receiving an allocation request from a third virtual machine at the local module for a new global queue pair number, the third virtual machine executing on the source host;

querying the distributed policy service for the new global queue pair number;

receiving, from the distributed policy service, the new global queue pair number at the local module;

identifying a source local port ID that corresponds to an adapter located on the source host;

generating a local table entry in a local storage area that includes the new global queue pair number and the source local port ID; and

notifying the distributed policy service of the source local port ID that corresponds to the new global queue pair number.

21. The computer program product of claim 20 wherein the information handling system performs additional actions comprising:

receiving a de-allocation request from the third virtual machine at the local module to de-allocate the new global queue pair number;

removing the local table entry from the local storage area; and

notifying the distributed policy service to de-allocate the new global queue pair number from the third virtual machine.

22. The computer program product of claim 18 wherein the computer subnetwork includes a plurality of virtual domains that each correspond to one of a plurality of heterogeneous tenants, and wherein the source host and the destination host belong to one of the plurality of virtual domains, the information handling system performing additional actions comprising:

identifying one or more physical path translations that are based upon a logical connectivity between the first virtual machine and the second virtual machine, wherein the logical connectivity is independent of one or more physical topology constraints of the computer subnetwork; and

including the one or more physical path translations in the overlay header.

23. The computer program product of claim 17 wherein the information handling system performs additional actions comprising:

receiving, at the destination host, the encapsulated frame;

decapsulating the encapsulated frame and extracting the destination global queue pair number from the overlay header;

identifying a virtual function that corresponds to the destination global queue pair number; and

forwarding the data frame to the second virtual machine through the identified virtual function.

24. The computer program product of claim 17 wherein:

the data frame includes a local routing header, a base transport header, and is devoid of a global routing header; and

the destination global queue pair number corresponds to a send queue, a receive queue, and a completion queue associated with the second virtual machine.

25. A method comprising:

receiving, at a local module, a data frame initiated by a first virtual machine that executes on a first host, wherein the data frame has a target destination at a second virtual machine that executes on a second host;

identifying a destination global queue pair number and a destination local port ID that correspond to the second virtual machine, wherein the destination global queue pair number is independent of the destination host;

determining, by the local module, that the destination local port ID is not valid;

querying a distributed policy service by the local module for an updated local port ID that corresponds to the destination global queue pair number;

receiving the updated destination local port ID from the distributed policy service at the local module in response to the querying, wherein the updated destination local port ID corresponds to a third host to which the second virtual machine migrated;

including the destination global queue pair number and the updated destination local port ID in an overlay header;

encapsulating the data frame with the overlay header, the encapsulating resulting in an encapsulated frame; and

sending the encapsulated frame through a computer network to the second virtual machine executing on the third host.