Local access DMA with shared memory pool

• US 10,157,146 B2
• Filed: 02/12/2015
• Issued: 12/18/2018
• Est. Priority Date: 02/12/2015
• Status: Active Grant

• Current Assignee: Red Hat Israel Ltd. (International Business Machines Corporation)
• Original Assignee: Red Hat Israel Ltd. (International Business Machines Corporation)
• Inventors: Tsirkin, Michael, Apfelbaum, Marcel
• Primary Examiner(s): Aquino, Wynuel S
• Application Number:
  US14/620,906
• Publication Number:
  US 20160239430A1
• Time in Patent Office: 1,405 Days
• Field of Search: None
• CPC Class Codes: G06F 12/109 for multiple virtual address spaces, e.g. segmentation G06F12/1036 takes precedence
• CPC Class Codes: G06F 2009/45583 Memory management, e.g. access or allocation
• CPC Class Codes: G06F 2212/1024 Latency reduction
• CPC Class Codes: G06F 2212/152 Virtualized environment, e.g. logically partitioned system
• CPC Class Codes: G06F 2212/154 Networked environment
• CPC Class Codes: G06F 2212/656 Address space sharing
• CPC Class Codes: G06F 2212/657 Virtual address space management
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A processing device receives a first request from a virtual machine to register a memory region to a hardware device. The processing device generates a first key for the memory region, wherein the memory region is not registered to the hardware device. The processing device generates a second key for a shared memory pool that is pinned and registered to the hardware device. The processing device generates a mapping of the first key to the second key. The processing device sends a response to the virtual machine that the memory region has been registered to the hardware device, the notification comprising the first key.

Claims

1. A method comprising:

receiving, by a processing device executing a hypervisor, a first request from a virtual machine to register a memory region with a hardware device, wherein registering the memory region comprises pinning one or more underlying memory pages of the memory region and generating a key for the memory region;

generating, by the processing device executing the hypervisor, a first key for the memory region, wherein the memory region is not registered with the hardware device;

generating, by the processing device, a second key to access a shared memory pool, the shared memory pool being pinned and registered to the hardware device;

generating a mapping of the first key to the second key;

in response to generating the mapping of the first key to the second key, sending a response to the virtual machine that the memory region has been registered to the hardware device, the response comprising the first key that has been mapped to the second key, the first key comprising an indication that the first key is a virtual key that is to be replaced with the second key that is mapped to the first key;

receiving a second request for the hardware device to perform an operation associated with the memory region, wherein the request identifies the first key; and

copying data from a first memory location associated with the first key to a second memory location associated with the second key, wherein the operation is to be performed by the hardware device with the data at the second memory location.

2. The method of claim 1, further comprising:

generating an additional memory region for the virtual machine, wherein the additional memory region is controlled by the hypervisor and overlaps the shared memory pool; and

registering the additional memory region with the hardware device, wherein the second key for the shared memory pool comprises a key for the additional memory region that overlaps the shared memory pool.

3. The method of claim 2, wherein the virtual machine is associated with a protection domain, wherein the memory region is in the protection domain, and wherein the additional memory region is in the protection domain.

4. The method of claim 1, further comprising:

determining whether permissions associated with the first request allow remote access to the memory region;

responsive to determining that the permissions allow remote access to the virtual memory region, disabling overcommitment of virtual memory for the virtual machine; and

responsive to determining that the permissions do not allow remote access to the memory region, proceeding with generating the first key.

5. The method of claim 1,

wherein the hardware device is a network interface controller (NIC), the second request comprising the first key, a first memory address of the first location in the memory region and a length, and wherein the method further comprises:

providing the hardware device with direct memory access (DMA) to the second location of the shared pool, the providing comprising;

determining that the first key is associated with the second key in view of the indication that the first key is a virtual key;

allocating the second location of the shared pool for the second request, the second location having the second memory address and the length;

modifying the second request by replacing the first key and the first memory address in the second request with the second key and the second memory address; and

sending the modified second request to the NIC.

6. The method of claim 5, wherein the second request is a send request, and

wherein the NIC is to read the contents of the second memory region via DMA and send the contents from the second memory location to a remote device.

7. The method of claim 5, wherein the second request is a receive request, and wherein the NIC is to receive data from a remote device and write the data to the second location in the shared memory pool via DMA, the method further comprising:

copying the data from the second location of the shared memory pool to the first location of the memory region of the virtual machine.

8. The method of claim 5, further comprising:

determining that the operation associated with the second request has completed; and

deallocating the second memory location of the shared memory pool.

9. The method of claim 5, further comprising:

responsive to determining that there is insufficient available space in the shared memory pool to satisfy the second request, adding an entry into a queue for the second request; and

performing the allocating of the second memory location responsive to the entry for the second request reaching a start of the queue and sufficient available space in the shared memory pool becoming available.

10. The method of claim 1, further comprising:

receiving a request to access the memory region, the request comprising a third key;

determining whether the third key corresponds to the second key; and

responsive to determining that the third key corresponds to the second key, returning an error.

11. A non-transitory computer readable storage medium comprising instructions that, when executed by a processing device executing a hypervisor, cause the processing device to:

receive, by the processing device executing the hypervisor, a first request from a virtual machine to register a memory region with a hardware device, wherein registering the memory region comprises pinning one or more underlying memory pages of the memory region and generating a key for the memory region;

generate, by the processing device executing the hypervisor, a first key for the memory region, wherein the memory region is not registered with the hardware device;

generate, by the processing device, a second key to access a shared memory pool, the shared memory pool being pinned and registered to the hardware device;

generate a mapping of the first key to the second key;

in response to generating the mapping of the first key to the second key, send a response to the virtual machine that the memory region has been registered to the hardware device, the response comprising the first key that has been mapped to the second key, the first key comprising an indication that the first key is a virtual key that is to be replaced with the second key that is mapped to the first key;

receive a second request for the hardware device to perform an operation associated with the memory region, wherein the request identifies the first key; and

copy data from a first memory location associated with the first key to a second memory location associated with the second key, wherein the operation is to be performed by the hardware device with the data at the second memory location.

12. The non-transitory computer readable storage medium of claim 11, wherein the instructions further cause the processing device to:

generate an additional memory region for the virtual machine, wherein the additional memory region is controlled by the hypervisor and overlaps the shared memory pool; and

register the additional memory region with the hardware device, wherein the second key for the shared memory pool comprises a key for the additional memory region that overlaps the shared memory pool.

13. The non-transitory computer readable storage medium of claim 12, wherein the virtual machine has a protection domain, wherein the memory region is in the protection domain, and wherein the additional memory region is in the protection domain.

14. The non-transitory computer readable storage medium of claim 11, wherein the instructions further cause the processing device to:

determine whether permissions associated with the first request allow remote access to the memory region;

responsive to determining that the permissions allow remote access to the virtual memory region, disable overcommitment of virtual memory for the virtual machine; and

responsive to determining that the permissions do not allow remote access to the memory region, proceed with generating the first key.

15. The non-transitory computer readable storage medium of claim 11,

wherein the hardware device is a network interface controller (NIC), the second request comprising the first key, the first memory address of a first location in the memory region and a length, and wherein the processing device is further to:

provide the hardware device with direct memory access (DMA) to the second location of the shared pool, wherein to provide the hardware device with the DMA, the processing device is further to;

determine that the first key is associated with the second key in view of the indication that the first key is a virtual key;

allocate the second location of the shared pool for the second request, the second location having the second memory address and the length;

modify the second request by replacing the first key and the first memory address in the second request with the second key and the second memory address; and

send the modified second request to the NIC.

16. The non-transitory computer readable storage medium of claim 15,

wherein the NIC is to read the contents of the second memory region via DMA and send the contents from the second memory location to a remote device.

17. The non-transitory computer readable storage medium of claim 15, wherein the second request is a receive request, and wherein the NIC is to receive data from a remote device and write the data to the second location in the shared memory pool via DMA, and wherein the instructions further cause the processing device to:

copy the data from the second location of the shared memory pool to the first location of the memory region of the virtual machine.

18. The non-transitory computer readable storage medium of claim 15, wherein the instructions further cause the processing device to:

determine that the operation associated with the second request has completed; and

deallocate the second memory location of the shared memory pool.

19. The non-transitory computer readable storage medium of claim 11, wherein the instructions further cause the processing device to:

responsive to determining that there is insufficient available space in the shared memory pool to satisfy the second request, add an entry into a queue for the second request; and

perform the allocating of the second memory location responsive to the entry for the second request reaching a start of the queue and sufficient available space in the shared memory pool becoming available.

20. A system comprising:

a memory;

a network interface controller (NIC) coupled to the memory; and

a processing device, operatively coupled with the memory and the NIC, the processing device to;

receive a first request from a virtual machine to register a memory region to the NIC, wherein registering the memory region comprises pinning one or more underlying memory pages of the memory region and generating a key for the memory region;

generate a first key for the memory region, wherein the memory region is not registered with the NIC;

generate a second key to access a shared memory pool, the shared memory pool being pinned and registered to the NIC;

generate a mapping of the first key to the second key;

in response to generating the mapping of the first key to the second key, send a response to the virtual machine that the memory region has been registered with the NIC, the response comprising the first key that has been mapped to the second key, the first key comprising an indication that the first key is a virtual key that is to be replaced with the second key that is mapped to the first key;

receive a second request for the NIC to perform an operation associated with the memory region, wherein the request identifies the first key; and

copy data from a first memory location associated with the first key to a second memory location associated with the second key, wherein the operation is to be performed by the NIC with the data at the second memory location.