Remote direct memory access adapter state migration in a virtual environment

• US 10,467,182 B2
• Filed: 05/19/2016
• Issued: 11/05/2019
• Est. Priority Date: 10/31/2011
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Sharp, Robert O., Keels, Kenneth G.
• Primary Examiner(s): Alriyashi, Abdulkader M
• Application Number:
  US15/159,209
• Publication Number:
  US 20160267053A1
• Time in Patent Office: 1,265 Days
• CPC Class Codes: G06F 15/17331 Distributed shared memory [DSM], e.g. remote direct memory access [RDMA]
• CPC Class Codes: G06F 2009/4557 Distribution of virtual machine instances; Migration and load balancing
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects
• CPC Class Codes: G06F 9/5016 the resource being the memory

Abstract

In an embodiment of the present invention, a method includes partitioning a plurality of remote direct memory access context objects among a plurality of virtual functions, establishing a remote direct memory access connection between a first of the plurality of virtual functions, and migrating the remote direct memory access connection from the first of the plurality of virtual functions to a second of the plurality of virtual functions without disconnecting from the remote peer.

Claims

1. An apparatus comprising:

a network adapter that includes an onboard memory;

a processor to execute a virtual machine monitor to;

partition an onboard memory space of the network adapter into a plurality of regions to distribute a plurality of remote direct memory access (RDMA) context objects among a plurality of virtual functions;

assign a first region of the plurality of regions to a first virtual function of the plurality of virtual functions;

store a segment table in a segment table space of the network adapter;

store at least a first RDMA context object of the plurality of RDMA context objects in the first region, at a location referenced by a first entry in the segment table; and

assign the first entry to the first virtual function.

2. The apparatus of claim 1, wherein the network adapter is to:

establish an RDMA connection with the remote peer with the first RMDA context object; and

allocate a second region in the onboard memory space to a second of the plurality of virtual functions.

3. The apparatus of claim 2, wherein the network adapter is further to copy contents from the first region to the second region to migrate the RDMA connection from the first virtual function to the second virtual function.

4. The apparatus of claim 3, wherein to migrate the RDMA connection comprises to copy at least one of a local network address or a network routing configuration from the first region to the second region.

5. The apparatus of claim 2, wherein the network adapter is further to migrate the RDMA connection from the first virtual function to the second virtual function without disconnection of the RDMA connection.

6. The apparatus of claim 2, wherein the network adapter is further to:

stop a packet flow associated with the RDMA connection;

flush a state associated with the RDMA connection;

copy RDMA parameters from the first context object to the second region; and

continue the packet flow after the RDMA parameters have been copied to the second region.

7. The apparatus of claim 2, wherein the network adapter is further to:

determine whether a threshold amount of resources are available in the network adapter to perform a migration to the second virtual function; and

migrate, in response to a determination that the threshold amount of resources are available, the RDMA connection from the first virtual function to the second virtual function.

8. One or more non-transitory machine-readable storage media comprising a plurality of instructions stored thereon that, when executed, cause an apparatus to:

partition, with a virtual machine monitor, an onboard memory space of a network adapter of the apparatus into a plurality of regions for use in distributing a plurality of remote direct memory access (RDMA) context objects among a plurality of virtual functions;

assign, with the virtual machine monitor, a first region of the plurality of regions to a first virtual function of the plurality of virtual functions;

store, with the virtual machine monitor, a segment table in a segment table space of the network adapter;

store, with the virtual machine monitor, at least a first RDMA context object of the plurality of RDMA context objects in the first region, at a location referenced by a first entry in the segment table; and

assign, with the virtual machine monitor, the first entry to the first virtual function.

9. The one or more non-transitory machine-readable storage media of claim 8, wherein the plurality of instructions, when executed, further cause the apparatus to:

establish, with the network adapter, an RDMA connection with a remote peer with the first RDMA context object; and

allocate, with the network adapter, a second region in the onboard memory space to a second of the plurality of virtual functions.

10. The one or more non-transitory machine-readable storage media of claim 9, wherein plurality of instructions, when executed, further cause the apparatus to copy contents from the first region to the second region to migrate the RDMA connection from the first virtual function to the second virtual function.

11. The one or more non-transitory machine-readable storage media of claim 10, wherein the plurality of instructions, when executed, further cause the apparatus to copy at least one of a local network address or a network routing configuration from the first region to the second region to migrate the RDMA connection.

12. The one or more non-transitory machine-readable storage media of claim 9, wherein the plurality of instructions, when executed, further cause the apparatus to migrate the RDMA connection from the first virtual function to the second virtual function without disconnection of the RDMA connection.

13. The one or more non-transitory machine-readable storage media of claim 9, wherein the plurality of instructions, when executed, further cause the apparatus to:

stop, with the network adapter, a packet flow associated with the RDMA connection;

flush, with the network adapter, a state associated with the RDMA connection;

copy, with the network adapter, RDMA parameters from the first context object to the second region; and

continue, with the network adapter, the packet flow after the RDMA parameters have been copied to the second region.

14. The one or more non-transitory machine-readable storage media of claim 9, wherein the plurality of instructions, when executed, further cause the apparatus to:

determine, with the network adapter, whether a threshold amount of resources are available in the network adapter to perform a migration to the second virtual function; and

migrate, with the network adapter and in response to a determination that the threshold amount of resources are available, the RDMA connection from the first virtual function to the second virtual function.

15. An apparatus comprising:

circuitry to;

partition an onboard memory space of a network adapter into a plurality of regions for use in distributing a plurality of remote direct memory access (RDMA) context objects among a plurality of virtual functions;

assign a first region of the plurality of regions to a first virtual function of the plurality of virtual functions;

store a segment table in a segment table space of the network adapter;

store at least a first RDMA context object of the plurality of RDMA context objects in the first region, at a location referenced by a first entry in the segment table; and

assign the first entry to the first virtual function.

16. The apparatus of claim 15, wherein the circuitry is further to:

establish an RDMA connection with a remote peer with the first RDMA context object; and

allocate a second region in the onboard memory space to a second of the plurality of virtual functions.

17. The apparatus of claim 16, wherein the circuitry is further to migrate the RDMA connection from the first virtual function to the second virtual function by copying contents from the first region to the second region.

18. The apparatus of claim 17, wherein to migrate the RDMA connection comprises to copy at least one of a local network address or a network routing configuration from the first region to the second region.

19. The apparatus of claim 16, wherein the circuitry is further to migrate the RDMA connection from the first virtual function to the second virtual function without disconnecting the RDMA connection.

20. The apparatus of claim 16, wherein the circuitry is further to:

stop a packet flow associated with the RDMA connection;

flush a state associated with the RDMA connection;

copy RDMA parameters from the first context object to the second region; and

continue the packet flow after the RDMA parameters have been copied to the second region.

21. A method comprising:

partitioning, by a virtual machine monitor, an onboard memory space of a network adapter into a plurality of regions for use in distributing a plurality of remote direct memory access (RDMA) context objects among a plurality of virtual functions;

assigning, by the virtual machine monitor, a first region of the plurality of regions to a first virtual function of the plurality of virtual functions;

storing, by the virtual machine monitor, a segment table in a segment table space of the network adapter;

storing, by the virtual machine monitor, at least a first RDMA context object of the plurality of RDMA context objects in the first region, at a location referenced by a first entry in the segment table; and

assigning, by the virtual machine monitor, the first entry to the first virtual function.

22. The method of claim 21, further comprising:

establishing, by the network adapter, an RDMA connection with a remote peer with the first RDMA context object; and

allocating, by the network adapter, a second region in the onboard memory space to a second of the plurality of virtual functions.

23. The method of claim 22, further comprising migrating, by the network adapter, the RDMA connection from the first virtual function to the second virtual function by copying contents from the first region to the second region.

24. The method of claim 23, wherein migrating the RDMA connection comprises copying at least one of a local network address or a network routing configuration from the first region to the second region.