System And Method Of Providing Multiple Virtual Machines With Shared Access To Non-Volatile Solid-State Memory Using RDMA

• US 20100083247A1
• Filed: 09/26/2008
• Published: 04/01/2010
• Est. Priority Date: 09/26/2008
• Status: Abandoned Application

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Miller, Steven C., Kanevsky, Arkady
• Application Number:
  US12/239,092
• Publication Number:
  US 20100083247A1
• Time in Patent Office: Days
• US Class Current: 718/1
• CPC Class Codes: G06F 13/28 using burst mode transfer, e.g. direct memory access DMA, cycle steal G06F13/32 takes precedence
• CPC Class Codes: G06F 2009/45583 Memory management, e.g. access or allocation
• CPC Class Codes: G06F 2009/45587 Isolation or security of virtual machine instances
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects

Abstract

A processing system includes a plurality of virtual machines which have shared access to a non-volatile solid-state memory (NVSSM) subsystem, by using remote direct memory access (RDMA). The NVSSM subsystem can include flash memory and other types of non-volatile solid-state memory. The processing system uses scatter-gather lists to specify the RDMA read and write operations. Multiple reads or writes can be combined into a single RDMA read or write, respectively, which can then be decomposed and executed as multiple reads or writes, respectively, in the NVSSM subsystem. Memory accesses generated by a single RDMA read or write may be directed to different memory devices in the NVSSM subsystem, which may include different forms of non-volatile solid-state memory.

Claims

1. A processing system comprising:

a plurality of virtual machines;

a non-volatile solid-state memory shared by the plurality of virtual machines;

a hypervisor operatively coupled to the plurality of virtual machines; and

a remote direct memory access (RDMA) controller operatively coupled to the plurality of virtual machines and the hypervisor, to access the non-volatile solid-state memory on behalf of the plurality of virtual machines by using RDMA operations.

2. A processing system as recited in claim 1, wherein each of the virtual machines and the hypervisor synchronize write accesses to the non-volatile solid-state memory through the RDMA controller by using atomic memory access operations.

3. A processing system as recited in claim 1, wherein the virtual machines access the non-volatile solid-state memory by communicating with the non-volatile solid-state memory through the RDMA controller without involving the hypervisor.

4. A processing system as recited in claim 1, wherein the hypervisor generates tags to determine a portion of the non-volatile solid-state memory which each of the virtual machines can access.

5. A processing system as recited in claim 4, wherein the hypervisor uses tags to control read and write privileges of the virtual machines to different portions of the non-volatile solid-state memory.

6. A processing system as recited in claim 4, wherein the hypervisor generates the tags to implement load balancing across the non-volatile solid-state memory.

7. A processing system as recited in claim 4, wherein the hypervisor generates the tags to implement fault tolerance between the virtual machines.

8. A processing system as recited in claim 1, wherein the hypervisor implements fault tolerance between the virtual machines by configuring the virtual machines each to have exclusive write access to a separate portion of the non-volatile solid-state memory.

9. A processing system as recited in claim 8, wherein the hypervisor has read access to the portions of the non-volatile solid-state memory to which the virtual machines have exclusive write access.

10. A processing system as recited in claim 1, wherein the non-volatile solid-state memory comprises non-volatile random access memory and a second form of non-volatile solid-state memory;

and

wherein, when writing data to the non-volatile solid-state memory, the RDMA controller stores in the non-volatile random access memory, metadata associated with data being stored in the second form of non-volatile solid-state memory.

11. A processing system as recited in claim 1, further comprising a second memory;

wherein the RDMA controller uses scatter-gather lists of the non-volatile solid-state memory and the second memory to perform an RDMA data transfer between the non-volatile solid-state memory and the second memory.

12. A processing system as recited in claim 1, wherein the RDMA controller combines a plurality of write requests from one or more of the virtual machines into a single RDMA write targeted to the non-volatile solid-state memory, wherein the single RDMA write is executed at the non-volatile solid-state memory as a plurality of individual writes.

13. A processing system as recited in claim 12, wherein the RDMA controller suppresses completion status indications for individual ones of the plurality of RDMA writes, and generates only a single completion status indication after the plurality of individual writes have completed successfully.

14. A processing system as recited in claim 13, wherein the non-volatile solid-state memory comprises a plurality of erase blocks, wherein the single RDMA write affects at least one erase block of the non-volatile solid-state memory, and wherein the RDMA controller combines the plurality of write requests so that the single RDMA write substantially fills each erase block affected by the single RDMA write.

15. A processing system as recited in claim 1, wherein the RDMA controller initiates an RDMA write targeted to the non-volatile solid-state memory, the RDMA write comprising a plurality of sets of data, including:

write data,

resiliency metadata associated with the write data, and

file system metadata associated with the client write data;

and wherein the RDMA write causes the plurality of sets of data to be written into different sections of the non-volatile solid-state memory according to an RDMA scatter list generated by the RDMA controller.

16. A processing system as recited in claim 15, wherein the different sections include a plurality of different types of non-volatile solid-state memory.

17. A processing system as recited in claim 16, wherein the plurality of different types include flash memory and non-volatile random access memory.

18. A processing system as recited in claim 17, wherein the RDMA write causes the client write data and the resiliency metadata to be stored in the flash memory and causes the other metadata to be stored in the non-volatile random access memory.

19. A processing system as recited in claim 1, wherein the RDMA controller combines a plurality of read requests from one or more of the virtual machines into a single RDMA read targeted to the non-volatile solid-state memory.

20. A processing system as recited in claim 19, wherein the single RDMA read is executed at the non-volatile solid-state memory as a plurality of individual reads.

21. A processing system as recited in claim 1, wherein the RDMA controller uses RDMA to read data from the non-volatile solid-state memory in response to a request from one of the virtual machines, including generating, from the read request, an RDMA read with a gather list specifying different subsets of the non-volatile solid-state memory as read sources.

22. A processing system as recited in claim 21, wherein at least two of the different subsets are different types of non-volatile solid-state memory.

23. A processing system as recited in claim 22, wherein the different types of non-volatile solid-state memory include flash memory and non-volatile random access memory.

24. A processing system as recited in claim 1, wherein the non-volatile solid-state memory comprises a plurality of memory devices, and wherein the RDMA controller uses RDMA to implement a RAID redundancy scheme to distribute data for a single RDMA write across the plurality of memory devices.

25. A processing system as recited in claim 24, wherein the RAID redundancy scheme is transparent to each of the virtual machines.

26. A processing system comprising:

a plurality of virtual machines;

a non-volatile solid-state memory;

a second memory;

a hypervisor operatively coupled to the plurality of virtual machines, to configure the virtual machines to have exclusive write access each to a separate portion of the non-volatile solid-state memory, wherein the hypervisor has at least read access to each said portion of the non-volatile solid-state memory, and wherein the hypervisor generates tags, for use by the virtual machines, to control which portion of the non-volatile solid-state memory each of the virtual machines can access; and

a remote direct memory access (RDMA) controller operatively coupled to the plurality of virtual machines and the hypervisor, to access the non-volatile solid-state memory on behalf of each of the virtual machines, by creating scatter-gather lists associated with the non-volatile solid-state memory and the second memory to perform an RDMA data transfer between the non-volatile solid-state memory and the second memory, wherein the virtual machines access the non-volatile solid-state memory by communicating with the non-volatile solid-state memory through the RDMA controller without involving the hypervisor.

27. A processing system as recited in claim 26, wherein the hypervisor uses RDMA tags to control access privileges of the virtual machines to different portions of the non-volatile solid-state memory.

28. A processing system as recited in claim 26, wherein the non-volatile solid-state memory comprises non-volatile random access memory and a second form of non-volatile solid-state memory;

and

wherein, when writing data to the non-volatile solid-state memory, the RDMA controller stores in the non-volatile random access memory, metadata associated with data being stored in the second form of non-volatile solid-state memory.

29. A processing system as recited in claim 26, wherein the RDMA controller combines a plurality of write requests from one or more of the virtual machines into a single RDMA write targeted to the non-volatile solid-state memory, wherein the single RDMA write is executed at the non-volatile solid-state memory as a plurality of individual writes.

30. A processing system as recited in claim 26, wherein the RDMA controller uses RDMA to read data from the non-volatile solid-state memory in response to a request from one of the virtual machines, including generating, from the read request, an RDMA read with a gather list specifying different subsets of the non-volatile solid-state memory as read sources.

31. A processing system as recited in claim 30, wherein at least two of the different subsets are different types of non-volatile solid-state memory.

32. A method comprising:

operating a plurality of virtual machines in a processing system; and

using remote direct memory access (RDMA) to enable the plurality of virtual machines to have shared access to a non-volatile solid-state memory, including using RDMA to implement fault tolerance between the virtual machines in relation to the non-volatile solid-state memory.

33. A method as recited in claim 32, wherein using RDMA to implement fault tolerance between the virtual machines comprises using a hypervisor to configure the virtual machines to have exclusive write access each to a separate portion of the non-volatile solid-state memory.

34. A method as recited in claim 33, wherein the virtual machines access the non-volatile solid-state memory without involving the hypervisor in accessing the non-volatile solid-state memory.

35. A method as recited in claim 33, wherein using a hypervisor comprises the hypervisor generating tags to determine a portion of the non-volatile solid-state memory which each of the virtual machines can access and to control read and write privileges of the virtual machines to different portions of the non-volatile solid-state memory.

36. A method as recited in claim 32, wherein said using RDMA operations further comprises using RDMA to implement at least one of:

wear-leveling across the non-volatile solid-state memory;

load balancing across the non-volatile solid-state memory;

or

37. A method as recited in claim 32, wherein said using RDMA operations comprises:

combining a plurality of write requests from one or more of the virtual machines into a single RDMA write targeted to the non-volatile solid-state memory, wherein the single RDMA write is executed at the non-volatile solid-state memory as a plurality of individual writes.

38. A method as recited in claim 32, wherein said using RDMA operations comprises:

using RDMA to read data from the non-volatile solid-state memory in response to a request from one of the virtual machines, including generating, from the read request, an RDMA read with a gather list specifying different subsets of the non-volatile solid-state memory as read sources.

39. A method as recited in claim 38, wherein at least two of the different subsets are different types of non-volatile solid-state memory.

40. A method as recited in claim 32, wherein the non-volatile solid-state memory comprises a plurality of memory devices, and wherein using RDMA to implement fault tolerance comprises:

using RDMA to implement a RAID redundancy scheme which is transparent to each of the virtual machines to distribute data for a single RDMA write across the plurality of memory devices of the non-volatile solid-state memory.