Pre-registering memory regions for remote direct memory access in a distributed file system

• US 10,713,211 B2
• Filed: 01/13/2016
• Issued: 07/14/2020
• Est. Priority Date: 01/13/2016
• Status: Active Grant

• Current Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Original Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Inventors: Parambil, Mohammed Rafi Kavungal Chundattu, Talur, Raghavendra
• Primary Examiner(s): Joo, Joshua
• Application Number:
  US14/994,959
• Publication Number:
  US 20170199841A1
• Time in Patent Office: 1,644 Days
• Field of Search: None
• CPC Class Codes: G06F 15/17331 Distributed shared memory [DSM], e.g. remote direct memory access [RDMA]
• CPC Class Codes: G06F 9/54 Interprogram communication
• CPC Class Codes: H04L 49/9005 using dynamic buffer space allocation
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

The subject matter of this specification can be implemented in, among other things, a method that includes pre-registering, by a processing device at a client device, multiple input/output (IO) buffers at the client device with a remote direct memory access (RDMA) interface at the client device. The client device accesses multiple server devices of a distributed file system using the RDMA interface. The method further includes receiving a request to access a file in the distributed file system from an application at the client device. The method further includes designating a first IO buffer among the IO buffers as a cache for data from the file. The method further includes receiving the data for the file in the first IO buffer from the distributed file system using the RDMA interface.

Claims

1. A method comprising:

pre-registering, by at least one processing device at a client device, multiple input/output (IO) buffers comprising memory regions at the client device for a remote direct memory access (RDMA) interface at the client device, wherein the client device provides an application executed by the client device with access to a distributed file system comprising multiple server devices using the RDMA interface, and wherein the IO buffers are pre-registered during initialization of the RDMA interface prior to providing the application with access to the distributed file system using the RDMA interface;

receiving a request to access a file in the distributed file system from the application at the client device;

responsive to determining that data of the file has been designated for caching, designating a first IO buffer among the IO buffers as a cache for the data from the file; and

receiving the data for the file in the first IO buffer from the distributed file system using the RDMA interface.

2. The method of claim 1, wherein the IO buffers comprise a first set of IO buffers designated for caching data received from the distributed file system and a second set of IO buffers that are not designated for caching data received from the distributed file system, and wherein the first set of IO buffers comprises the first IO buffer.

3. The method of claim 2, wherein the cached data for the file in the first IO buffer is used again for a subsequent request to access the file from the client device, wherein the method further comprises determining that the cached data for the file in the first IO buffer relative to the data for the file in the distributed file system has not changed prior to using the cached data for the file in the first IO buffer again.

4. The method of claim 1, wherein pre-registering comprises specifying multiple levels of access for the IO buffers due to a level of access subsequently used by the application not yet being known at a time the IO buffers are pre-registered.

5. The method of claim 1, wherein the request to access the file comprises a system call or a file operation of an operating system, and wherein the system call or the file operation accesses a virtual file system.

6. The method of claim 5, wherein the operating system directs the system call or the file operation for the virtual file system to a kernel module in kernel space that provides access to the distributed file system from user space.

7. A non-transitory computer-readable medium having instructions stored thereon that, when executed by at least one processing device, cause the processing device to:

pre-register, by the processing device at a client device, multiple input/output (IO) buffers comprising memory regions at the client device for a remote direct memory access (RDMA) interface at the client device, wherein the client device provides an application executed by the client device with access to a distributed file system comprising multiple server devices using the RDMA interface, and wherein the IO buffers are pre-registered during initialization of the RDMA interface prior to providing the application with access to the distributed file system using the RDMA interface, and wherein pre-registering comprises specifying multiple levels of access for the IO buffers due to a level of access subsequently used by the application not yet known at a time the IO buffers are pre-registered;

receive a request to access a file in the distributed file system from the application at the client device;

receive data for the file in a first IO buffer among the IO buffers from the distributed file system using the RDMA interface; and

responsive to determining that the data of the file has been designated for caching, copying the data from the first IO buffer to a cache at the client device.

8. The computer-readable medium of claim 7, wherein the processing device is to copy the data using a thread that is separate from a thread that receives the data from the distributed file system.

9. The computer-readable medium of claim 7, wherein the cached data for the file in the cache is used again for a subsequent request to access the file from the client device, wherein the instructions are further to cause the processing device to determine that the cached data for the file in the cache relative to the data for the file in the distributed file system has not changed prior to use of the cached data for the file in cache again.

10. The computer-readable medium of claim 7, wherein the request to access the file comprises a system call or a file operation of an operating system, and wherein the system call or the file operation accesses a virtual file system.

11. The computer-readable medium of claim 10, wherein the operating system directs the system call or the file operation for the virtual file system to a kernel module in kernel space that provides access to the distributed file system from user space.

12. A system comprising:

a remote direct memory access (RDMA) interface at a client device;

a memory, at the client device, comprising multiple memory regions; and

at least one processing device, at the client device, to communicate with the RDMA interface and the memory to;

pre-register multiple input/output (IO) buffers comprising the memory regions for the RDMA interface, wherein the client device provides an application executed by the client device with access to a distributed file system comprising multiple server devices using the RDMA interface, and wherein the IO buffers are pre-registered during initialization of the RDMA interface prior to providing the application with access to the distributed file system using the RDMA interface;

receive a request to access a file in the distributed file system from the application at the client device;

responsive to determining that data of the file has been designated for caching, designate a first IO buffer among the IO buffers as a cache for the data from the file; and

receive the data for the file in the first IO buffer from the distributed file system using the RDMA interface.

13. The system of claim 12, wherein the IO buffers comprise a first set of IO buffers designated for caching data received from the distributed file system and a second set of IO buffers that are not designated for caching data received from the distributed file system, and wherein the first set of IO buffers comprises the first IO buffer.

14. The system of claim 13, wherein the cached data for the file in the first IO buffer is used again for a subsequent request to access the file from the client device, wherein the processing device is further to determine that the cached data for the file in the first IO buffer relative to the data for the file in the distributed file system has not changed prior to use of the cached data for the file in the first IO buffer again.

15. The system of claim 12, wherein, to pre-register, the processing is further to specify multiple levels of access for the IO buffers due to a level of access subsequently used by the application not yet being known at a time the IO buffers are pre-registered.

16. The system of claim 12, wherein the request to access the file comprises a system call or a file operation of an operating system, and wherein the system call or the file operation accesses a virtual file system.

17. The system of claim 16, wherein the operating system directs the system call or the file operation for the virtual file system to a kernel module in kernel space that provides access to the distributed file system from user space.