Network storage system with NVRAM and cluster interconnect adapter implemented in a single circuit module

• US 7,895,286 B1
• Filed: 04/30/2004
• Issued: 02/22/2011
• Est. Priority Date: 04/30/2004
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Kilbourne, Allen J. II, Valin, Steven J., Budhia, Ravi K.
• Primary Examiner(s): Phillips; Hassan
• Application Number:
  US10/836,980
• Time in Patent Office: 2,489 Days
• Field of Search: None
• US Class Current: 709/212
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 69/12 Protocol engines

Abstract

A network storage system includes a non-volatile memory to store data including a log of received data access requests, and a cluster interconnect adapter through which to send data to a cluster partner. The nonvolatile memory and the cluster interconnect adapter are implemented in a single device connected to an expansion bus in the network storage system. Communication with the nonvolatile memory is carried out using LDMA, and communication with the cluster partner is carried out using RDMA via the cluster interconnect adapter. LDMA and RDMA functionality are merged in a common software stack.

Claims

1. A circuit module comprising:

a circuit medium designed to be installed within a host device;

a non-volatile memory disposed on the circuit medium, to store log data representing data access requests received by a network storage system; and

a cluster interconnect adapter disposed on the circuit medium, through which to send the log data to a cluster partner of the network storage system, the cluster interconnect adapter including

control circuitry to control access to the non-volatile memory and to control communication with the cluster partner, the control circuitry including a host channel adapter;

a first connector coupled to the control circuitry, to couple the cluster interconnect adapter to an external communication link for communication with the cluster partner; and

a second connector coupled to the control circuitry, to couple the cluster interconnect adapter to an internal expansion bus of the host device, through which to receive the log data.

2. A circuit module as recited in claim 1, wherein the second connector is configured to connect to a PCI bus.

3. A circuit module as recited in claim 1, wherein the circuit module is a circuit card.

4. A circuit module as recited in claim 1, wherein the non-volatile memory is a non-volatile random access memory.

5. A circuit module as recited in claim 1, wherein the host channel adapter comprises an InfiniBand host channel adapter.

6. A circuit module as recited in claim 1, wherein the circuit module is controllable to store data in the non-volatile memory during a local direct memory access (DMA) operation executed, and to store the data in a memory of the cluster partner via the cluster interconnect adapter during a remote DMA executed from the network storage system.

7. A circuit module as recited in claim 6, wherein the local DMA operation and the remote DMA operation are in response to a single transaction on the expansion bus.

8. A circuit module as recited in claim 6, wherein the local DMA operation comprises sending an InfiniBand message and using a loopback queue pair.

9. A circuit module as recited in claim 6, wherein the local DMA operation and the remote DMA operation are both in response to a data access request received from a client of the data storage system.

10. A circuit module as recited in claim 9, wherein the local DMA operation and the remote DMA operation each comprise sending data in an InfiniBand message.

11. A network storage system comprising:

a processor;

a storage adapter coupled to the processor;

a network adapter coupled to the processor; and

a circuit module coupled to the processor and separate from the storage adapter and the network adapter, the circuit module including

a circuit medium designed to be installed within a host device;

a non-volatile memory disposed on the circuit medium, to store the log data representing data access requests received by the network storage system, and

a cluster interconnect adapter disposed on the circuit medium, through which to send the log data to a cluster partner, the cluster interconnect adapter including

control circuitry to control access to the non-volatile memory and to control communication with the cluster partner, the control circuitry including a host channel adapter;

a first connector coupled to the control circuitry, to couple the cluster interconnect adapter to an external communication link for communication with the cluster partner; and

a second connector coupled to the control circuitry, to couple the cluster interconnect adapter to an internal expansion bus of the host device, through which to receive the log data.

12. A network storage system as recited in claim 11, wherein the circuit module is a circuit card.

13. A network storage system as recited in claim 11, wherein the non-volatile memory is a non-volatile random access memory.

14. A network storage system as recited in claim 11, wherein the host channel adapter comprises an InfiniBand host channel adapter.

15. A network storage system as recited in claim 11, wherein the network storage system comprises a filer.

16. A network storage system as recited in claim 11, wherein the circuit module is controllable to store data in the non-volatile memory during a local direct memory access (DMA) operation executed in the network storage system, and to store the data in a memory of the cluster partner via the cluster interconnect adapter during a remote DMA executed from the network storage system.

17. A network storage system as recited in claim 16, wherein the local DMA operation and the remote DMA operation are performed in response to a single transaction on the expansion bus.

18. A network storage system as recited in claim 17, wherein the local DMA operation comprises sending an InfiniBand message and using a loopback queue pair.

19. A network storage server comprising:

a processor;

a system memory coupled to the processor;

a network adapter coupled to the processor, through which to receive a write request from a host over a network;

a storage adapter coupled to the processor, through which to access data stored in a mass storage facility in response to the write request;

an expansion bus coupled to the processor; and

a peripheral unit coupled to the expansion bus and separate from the storage adapter and the network adapter, the peripheral unit including

a circuit medium designed to be installed within a host device;

a non-volatile random access memory (NVRAM) located on the circuit medium, to receive data representing the write request from the system memory during a local DMA operation and to store the data representing the write request as part of a log of write requests received by the network storage server, and

a cluster interconnect adapter located on the circuit medium, through which to send data representing the write request to a cluster partner during a remote DMA operation, wherein the cluster interconnect adapter includes;

control circuitry to control access to the NVRAM and to control communication with the cluster partner, the control circuitry including a host channel adapter;

a first connector coupled to the control circuitry, to couple the cluster interconnect adapter to an external communication link for communication with the cluster partner; and

a second connector coupled to the control circuitry, to couple the cluster interconnect adapter to an internal expansion bus of the host device, through which to receive the data representing the write request; and

an instruction memory storing a single software stack including instructions which when executed by the processor cause the processor to operate the NVRAM and the cluster interconnect adapter, wherein the network storage server is operable to store data in the NVRAM during a local direct memory access (DMA) operation executed in the network storage server, and is further operable to store the data in a memory of the cluster partner via the cluster interconnect adapter during a remote DMA executed from the network storage server and wherein the local DMA operation and the remote DMA operation are performed in response to a single transaction on the expansion bus, wherein the local DMA operation and the remote DMA operation each include sending data in an InfiniBand message, and wherein the local DMA operation further includes using a loopback queue pair.

20. A network storage server as recited in claim 19 wherein the circuit medium is a circuit card.

21. A network storage server as recited in claim 19, wherein the cluster interconnect adapter comprises an InfiniBand host channel adapter.

22. A network storage server as recited in claim 19, wherein the network storage server comprises a filer.