Network storage system with shared software stack for LDMA and RDMA

• US 7,493,424 B1
• Filed: 04/30/2004
• Issued: 02/17/2009
• Est. Priority Date: 04/30/2004
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Budhia, Ravi K., Bali, Naveen
• Primary Examiner(s): Patel; Niketa I
• Assistant Examiner(s): Franklin; Richard
• Application Number:
  US10/837,435
• Time in Patent Office: 1,754 Days
• Field of Search: 709/212, 710/22, 710/27, 710/33, 719/321, 719325-327
• US Class Current: 710/22
• CPC Class Codes: G06F 13/28 using burst mode transfer, e.g. direct memory access DMA, cycle steal G06F13/32 takes precedence

Abstract

A network storage system includes a non-volatile memory to store data including a log of received data access requests, and a cluster interconnect adapter through which to send data to a cluster partner. The nonvolatile memory and the cluster interconnect adapter are implemented in a single device connected to an expansion bus in the network storage system. Communication with the nonvolatile memory is carried out using LDMA, and communication with the cluster partner is carried out using RDMA via the cluster interconnect adapter. LDMA and RDMA functionality are merged in a common software stack.

Claims

1. A method comprising:

using a driver software module and a software stack in a network storage system to store data in a non-volatile memory in the network storage system, wherein the network storage system contains a cluster interconnect adapter, the driver software module contains code for accessing the non-volatile memory and the cluster interconnect adapter, and the software stack provides a shared interface for invoking the code; and

using the driver software module and the software stack to store data in a second memory in a cluster partner of the network storage system, by controlling the cluster interconnect adapter in the network storage system, wherein the software stack invokes the code in the driver software module to control the cluster interconnect adapter and to store data in the second memory of the cluster partner.

2. A method as recited in claim 1, wherein using the driver software module in the network storage system to store data in the non-volatile memory comprises using a local direct memory access (DMA) operation to store the data in the non-volatile memory.

3. A method as recited in claim 2, wherein using the driver software module to store data in the second memory comprises using a remote DMA operation to store the data in the second memory.

4. A method as recited in claim 3, wherein the local DMA operation and the remote DMA operation each comprises sending data in an INFINIBAND message.

5. A method as recited in claim 3, wherein the local DMA operation and the remote DMA operation are executed in response to receiving, at a network storage system, a data access request from a remote client of the network storage system.

6. A method as recited in claim 5, wherein the data stored in the non-volatile memory and the second memory represents the request.

7. A method as recited in claim 6, wherein the request is a write request relating to data stored by the network storage system.

8. A method as recited in claim 7, wherein the non-volatile memory is a non-volatile random access memory (NVRAM) of the network storage system.

9. A method as recited in claim 8, wherein the data is stored in the non-volatile memory and the second memory as part of a log of data access requests received by the network storage system.

10. A method of operating a network storage system, the method comprising:

managing access to data stored in a plurality of mass storage devices on behalf of a plurality of clients;

receiving a data access request at the network storage system from one of the plurality of clients, the data access request being directed to a portion of the stored data;

using a driver software module and a software stack in the network storage system to store data representing the data access request in a non-volatile memory in the network storage system, by performing a local direct memory access (DMA) operation, wherein the network storage system contains a cluster interconnect adapter, the driver software module contains code for accessing the non-volatile memory and the cluster interconnect adapter, and the software stack provides a shared interface for invoking the code; and

using the driver software module and the software stack to store data representing the data access request in a second memory in a cluster partner of the network storage system, by controlling the cluster interconnect adapter in the network storage system, wherein the software stack invokes the code in the driver software module to control the cluster interconnect adapter and to store data in the second memory of the cluster partner.

11. A method as recited in claim 10, wherein the local DMA operation and the remote DMA operation each comprises sending data in an INFINIBAND message.

12. A method as recited in claim 10, wherein the data representing the data access request is stored in the non-volatile memory and the second memory as part of a log of data access requests received by the network storage system from the plurality of clients.

13. A method as recited in claim 10, wherein using the driver software module to store data representing the data access request in the second memory in the cluster partner comprises performing a remote DMA operation to store the data representing the data access request.

14. An apparatus for use in a network storage system, the apparatus comprising:

a non-volatile memory to store log data representing data access requests received by the network storage system;

a cluster interconnect adapter through which the network storage system can communicate with a cluster partner; and

a driver software module and a software stack to control operation of both the non-volatile memory and the cluster interconnect adapter, wherein the software module contains code for accessing the non-volatile memory and the cluster interconnect adapter, and the software stack provides a shared interface for invoking the code.

15. An apparatus as recited in claim 14, wherein the driver software module is to store data in the non-volatile memory during a local direct memory access (DMA) operation executed in the network storage system, and to store the data in a memory of the cluster partner via the cluster interconnect adapter during a remote DMA executed from the network storage system.

16. An apparatus as recited in claim 15, wherein the local DMA operation and the remote DMA operation are executed concurrently.

17. An apparatus as recited in claim 15, wherein the local DMA operation and the remote DMA operation are both in response to a data access request received from a client of the data storage system.

18. An apparatus as recited in claim 14, further comprising an expansion bus coupled to the processor, wherein the non-volatile memory, the cluster interconnect adapter, and the driver software are all implemented in a single circuit module coupled to the expansion bus.

19. An apparatus as recited in claim 18, wherein the driver software is to store the data in the non-volatile memory and to store the data in a memory of the cluster partner, using a single transaction on the expansion bus.

20. An apparatus as recited in claim 14, wherein the non-volatile memory is a non-volatile random access memory.

21. A network storage system comprising:

a processor;

a storage adapter coupled to the processor;

a network adapter coupled to the processor;

a non-volatile memory coupled to the processor;

a cluster interconnect adapter coupled to the processor, through which the storage system can communicate with a cluster partner; and

an operating system, executable by the processor, the operating system including a driver and a software stack to control operation of both the non-volatile memory and the cluster interconnect adapter, wherein the software module contains code for accessing the non-volatile memory and the cluster interconnect adapter, and the software stack provides a shared interface for invoking the code.

22. A network storage system as recited in claim 21, wherein the driver is configured to execute a local direct memory access (DMA) operation to store data in the non-volatile memory and to initiate an operation to store the data in a memory of the cluster partner via the cluster interconnect adapter.

23. A network storage system as recited in claim 22, wherein the operation to store the data in the memory of the cluster partner is a remote DMA operation.

24. A network storage system as recited in claim 22, wherein the local DMA operation and the remote DMA operation are both in response to a data access request received from a client of the data storage system.

25. A network storage system as recited in claim 22, wherein the local DMA operation and the remote DMA operation each comprises sending data in an INFINIBAND message.

26. A network storage system as recited in claim 21, wherein the operating system further comprises a file system layer to manage a file system of the network storage system, the file system representing data stored in a set of mass storage devices, wherein the driver is responsive to the file system layer.

27. A network storage system as recited in claim 26, wherein the operating system further comprises a storage layer to manage storage operations performed upon the set of mass storage devices, wherein the driver is further responsive to the storage layer.

28. A network storage system as recited in claim 21, wherein the non-volatile memory is a non-volatile random access memory.

29. A network storage system as recited in claim 21, wherein the network storage system comprises a filer.

30. A network storage system comprising:

a processor;

a storage adapter coupled to the processor;

a network adapter coupled to the processor;

a non-volatile random access memory (NVRAM) coupled to the processor;

a cluster interconnect adapter coupled to the processor, through which the storage system can communicate with a cluster partner; and

a second memory containing a driver and a software stack, wherein the driver contains code for accessing the non-volatile memory and the cluster interconnect adapter, the software stack provides a shared interface for invoking the code, the driver and the software stack are to execute local direct memory access (DMA) operations to store data representing one or more write requests in the NVRAM and to initiate remote DMA operations to store the data representing the one or more write requests in a memory of the cluster partner via the cluster interconnect adapter.

31. A network storage system as recited in claim 30, wherein the local DMA operations and the remote DMA operations comprise sending the data in INFINIBAND messages.

32. A network storage system as recited in claim 30, wherein the second memory further contains:

a file system layer to manage a file system of the network storage system, the file system representing data stored in a set of mass storage devices; and

a storage layer to manage storage operations performed upon the set of mass storage devices;

wherein the driver is responsive to both the file system layer and the storage layer.

33. A network storage system as recited in claim 30, wherein the network storage system comprises a filer.

34. A network storage system comprising:

a non-volatile memory;

a cluster interconnect adapter;

means for using a driver software module and a software stack in the network storage system to store data in the non-volatile memory; and

means for using the driver software module and the software stack to store data in a second memory in a cluster partner of the network storage system, by controlling the cluster interconnect adapter, wherein the software module contains code for accessing the non-volatile memory and the cluster interconnect adapter, and the software stack provide a shared interface for invoking the code.

35. A method comprising:

using a driver software module and a software stack to access a device in a network storage system, wherein the device contains a non-volatile memory and a cluster interconnect adapter, the driver software module contains code for accessing the non-volatile memory and the cluster interconnect adapter, and the software stack provides a shared interface for invoking the code;

storing data in the non-volatile memory of the device by interfacing with the software stack, wherein the software stack invokes the code in the driver software module to access the non-volatile memory; and

storing data in a second memory in a cluster partner of the network storage system by interfacing with the software stack, wherein the software stack invokes the code in the driver software module to access the cluster interconnect adaptor and communicate with the cluster partner.

36. The method as recited in claim 35, wherein the non-volatile memory is embedded in the cluster interconnect adapter.

37. The method as recited in claim 35, wherein the software module further contains code for accessing the second memory of the cluster partner via the cluster interconnect adaptor.

38. The method as recited in claim 35, wherein the data stored in the non-volatile memory of the device contains a log for data access requests not yet saved to the network storage system.

39. The method as recited in claim 35, wherein said storing data in the non-volatile memory and said storing data in the second memory in the cluster partner are in response to a data storage request received from a client of the network storage system.