SCALABLE DATA USING RDMA AND MMIO

• US 20160034418A1
• Filed: 04/28/2015
• Published: 02/04/2016
• Est. Priority Date: 07/30/2014
• Status: Active Grant

• Current Assignee: NVIDIA Corporation
• Original Assignee: Excelero, Inc. (NVIDIA Corporation)
• Inventors: ROMEM, Yaniv, MANN, Omri, OSHRI, Ofer
• Granted Patent: US 9,658,782 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 12/1081 for peripheral access to main memory, e.g. direct memory access [DMA]
• CPC Class Codes: G06F 15/167 using a common memory, e.g. mailbox
• CPC Class Codes: G06F 15/17331 Distributed shared memory [DSM], e.g. remote direct memory access [RDMA]
• CPC Class Codes: G06F 2212/65 Details of virtual memory and virtual address translation
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0611 in relation to response time
• CPC Class Codes: G06F 3/0637 Permissions
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0647 Migration mechanisms
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0652 Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0688 Non-volatile semiconductor memory arrays
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD
• CPC Class Codes: H04L 12/06 Answer-back mechanisms or circuits
• CPC Class Codes: H04L 12/6418 Hybrid transport
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

To improve upon some of the characteristics of current storage systems in general and block data storage systems in particular, exemplary embodiments combine state-of-the art networking techniques with state-of-the-art data storage elements in a novel way. To accomplish this combination in a highly effective way, it is proposed to combine networking remote direct memory access (RDMA) technique and storage-oriented memory mapped input output (MMIO) technique in a system to provide direct access from a remote storage client to a remote storage system with little to no central processing unit (CPU) intervention of the remote storage server. In some embodiments, this technique may reduce the required CPU intervention on the client side. These reductions of CPU intervention potentially reduce latency while providing performance improvements, and/or providing more data transfer bandwidth and/or throughput and/or more operations per second compared to other systems with equivalent hardware.

Claims

1. A computerized method for retrieving data from a memory mapped input/output (MMIO) enabled storage device of a control server communicatively coupled with a client device through a remote direct memory access (RDMA) enabled network interface controller (rNIC), the method comprising:

receiving from the client device, through a first communication protocol, a first entry for a submission queue, the first entry using a second communication protocol that permits communication between the control server and the MMIO enabled storage device;

receiving from the client device, through the first communication protocol, a second entry for a send queue, the second entry using the first communication protocol to send a data block from a memory reserved for the client device to the client device;

receiving a trigger for the first entry; and

receiving, using the second communication protocol, the data block from the MMIO enabled storage device in response to executing the first entry; and

transmitting, using the first communication protocol, the data block to the client device in response to the data block being received from the MMIO enabled storage device.

2. The computerized method of claim 1, wherein the first communication protocol includes one from among Infiniband, RDMA over Converged Ethernet (RoCE), and iWARP.

3. The computerized method of claim 1, wherein the second communication protocol is NVMexpress.

4. The computerized method of claim 1, wherein the first entry comprises an instruction set that when executed by the storage device causes the storage device to read a block of data into the memory reserved for the client device.

5. The computerized method of claim 1, wherein the second entry comprises an instruction set that when executed causes the data block to be written to a memory in the client device.

6. The computerized method of claim 1, wherein the trigger is a doorbell.

7. The computerized method of claim 6, further comprising executing the first entry for in response to writing to the doorbell.

8. The computerized method of claim 1, further comprising generating an interrupt by the storage device.

9. The computerized method of claim 8, further comprising receiving instructions, at the storage device, to generate the interrupt to invoke the trigger.

10. The computerized method of claim 1, further comprising sending the client device an instruction to generate an interrupt to indicate the first entry was executed.

11. The computerized method of claim 1, further comprising receiving from the client device a poll to determine if the first entry was executed.

12. The computerized method of claim 1, further comprising reading a fixed data block from the storage device into the trigger.

13. The computerized method of claim 8, further comprising:

receiving the interrupt by a processing element of the control server; and

invoking the trigger in response to receiving the interrupt.

14. A computerized method for storing data in a memory mapped input/output (MMIO) enabled storage device of a control server communicatively coupled with a client device through a remote direct memory access (RDMA) enabled network interface controller (rNIC), the method comprising:

receiving through a first communication protocol from the client device a first data block for storing on the storage device;

receiving through the first communication protocol from the client device a first entry for a submission queue, the first entry for communicating using a second communication protocol;

receiving through the first communication protocol from the client device a second entry for a send queue to send a second data block from a memory reserved for the client device to the client device over the first communication protocol; and

receiving a trigger in response to receiving the second entry, and sending the second data block to the client device using the first communication protocol.

15. The computerized method of claim 14, wherein the first communication protocol includes one from among Infiniband, RoCE, and iWARP.

16. The computerized method of claim 14, wherein the second communication protocol is NVMexpress.

17. The computerized method of claim 14, wherein the second entry for the submission queue comprises an instruction set that when executed by the storage device causes the storage device to write the second data block from the memory reserved for the client device.

18. The computerized method of claim 14, further comprising generating an interrupt by the storage device.

19. The computerized method of claim 18, wherein the storage device receives instructions to generate the interrupt to invoke the trigger.

20. The computerized method of claim 14, further comprising sending the client device an instruction to generate an interrupt to indicate the second entry was executed.

21. The computerized method of claim 14, further comprising receiving from the client device a poll to determine if the second entry was executed.

22. The computerized method of claim 14, further comprising reading a fixed data block from the storage device into the trigger.

23. The computerized method of claim 18, further comprising:

receiving the interrupt by a processing element of the control server; and

invoking the trigger.

24. A non-transitory computer readable medium having stored thereon instructions that when executed by one or more processing elements perform a method of retrieving data from a memory mapped input/output (MMIO) enabled storage device of a control server communicatively coupled with a client device through a remote direct memory access (RDMA) enabled network interface controller (rNIC), the method comprising:

receiving from the client device, through a first communication protocol, a first entry for a submission queue, the first entry using a second communication protocol that permits communication between the control server and the MMIO enabled storage device;

receiving from the client device, through the first communication protocol, a second entry for a send queue, the second entry using the first communication protocol to send a data block from a memory reserved for the client device to the client device;

receiving a trigger for the first entry; and

receiving, using the second communication protocol, the data block from the MMIO enabled storage device in response to executing the first entry; and

transmitting, using the first communication protocol, the data block to the client device in response to the data block being received from the MMIO enabled storage device.

25. A non-transitory computer readable medium having stored thereon instructions that when executed by one or more processing elements perform a method for storing data in a memory mapped input/output (MMIO) enabled storage device of a control server communicatively coupled with a client device through a remote direct memory access (RDMA) enabled network interface controller (rNIC), the method comprising:

receiving through a first communication protocol from the client device a first data block for storing on the storage device;

receiving through the first communication protocol from the client device a first entry for a submission queue, the first entry for communicating using a second communication protocol;

receiving through the first communication protocol from the client device a second entry for a send queue to send a second data block from a memory reserved for the client device to the client device over the first communication protocol; and

receiving a trigger in response to receiving the second entry, and sending the second data block to the client device using the first communication protocol.

26. A server including:

a remote direct memory access (RDMA) enabled network interface controller (rNIC) configured to communicate with a client using a first communication protocol and to communicate with a storage device using a second communication protocol; and

a memory configured to store a data block received from the storage device using the second communication protocol,

wherein the rNIC is further configured to transmit the data block stored in the memory to the client device using the first communication protocol.

27. The server according to claim 26, wherein the rNIC includes the memory.

28. A server including:

a remote direct memory access (RDMA) enabled network interface controller (rNIC) configured to communicate with a client device using a first communication protocol and to communicate with a storage device using a second communication protocol, the storage device including a memory reserved for the client device;

a memory configured to store a first data block that is for storing on the storage device, in response to the rNIC receiving the data block from the client device;

a submission queue configured to store a first entry in response to receiving the first entry from the client device, the first entry facilitating communication between the rNIC and the storage device using the second communication protocol;

a send queue configured to store a second entry in response to receiving the second entry from the client device, the second entry facilitating sending a second data block from the memory reserved for the client device to the client device over the first communication protocol; and

a doorbell configured to send the second data block to the client device using the first communication protocol in response to being triggered.