Method and apparatus for assigning a local identifier to a cluster interconnect port in a network storage system

• US 7,769,913 B1
• Filed: 04/30/2004
• Issued: 08/03/2010
• Est. Priority Date: 04/30/2004
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Budhia, Ravi K., Bali, Naveen
• Primary Examiner(s): Sorrell; Eron J
• Assistant Examiner(s): Phan; Dean
• Application Number:
  US10/837,139
• Time in Patent Office: 2,286 Days
• Field of Search: None
• US Class Current: 710/9
• CPC Class Codes: G06F 11/2033 switching over of hardware resources
• CPC Class Codes: G06F 11/2092 Techniques of failing over between control units
• CPC Class Codes: H04L 61/5038 for local use, e.g. in LAN or USB networks, or in a controller area network [CAN]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A network storage system includes a non-volatile memory to store data including a log of received data access requests, and a cluster interconnect adapter through which to send data to a cluster partner. The nonvolatile memory and the cluster interconnect adapter are implemented in a single device connected to an expansion bus in the network storage system. Communication with the nonvolatile memory is carried out using LDMA, and communication with the cluster partner is carried out using RDMA via the cluster interconnect adapter. LDMA and RDMA functionality are merged in a common software stack.

Claims

1. A method of initializing a network storage system, the method comprising:

reading a persistent serial number from an InfiniBand-compliant circuit card within the network storage system, the InfiniBand-compliant circuit card including a non-volatile memory (NVRAM) and a cluster interconnect adapter;

assigning at least a portion of the serial number as an InfiniBand local identifier (LID) of a port of the InfiniBand-compliant circuit card, the LID being assigned without requiring any communication between the InfiniBand-compliant circuit card and any entity external to the network storage system;

using the LID to communicate between the NVRAM and a system memory of the network storage system which is external to the InfiniBand-compliant circuit card; and

using the LID to access the NVRAM by local direct memory access (LDMA) and to access a memory of an external cluster partner of the network storage system by remote direct memory access (RDMA), in one Peripheral Connect Interface (PCI) transaction.

2. A method as recited in claim 1, further comprising

using the LID to communicate between the InfiniBand-compliant circuit card and another entity within the network storage system.

3. A method as recited in claim 1, further comprising

sending the LID to the external cluster partner in an InfiniBand Subnet Management Packet (SMP), to inform the external cluster partner of the LID.

4. A method as recited in claim 1, wherein the network storage system is a filer.

5. A method as recited in claim 4, wherein the external cluster partner is a filer.

6. A network storage system comprising:

a processor;

a system memory coupled to the processor;

a network adapter coupled to the processor, through which to receive a data access request from a host over a network;

a storage adapter coupled to the processor, through which to access data stored in a mass storage facility in response to the data access request; and

a cluster device to enable the network storage system to communicate with a cluster partner external to the network storage system, the cluster device including;

a non-volatile memory (NVRAM),

an Infiniband-compliant cluster interconnect adapter,

a port through which to communicate with the cluster partner, and

control circuitry to control the cluster interconnect adapter, the control circuitry further

to assign a local identifier to the port without requiring any communication between the cluster interconnect adapter and any entity external to the network storage system, wherein assigning the local identifier includes reading a persistent serial number from the cluster interconnect adapter and assigning at least a portion of the persistent serial number as the local identifier, and

to use the local identifier to access the NVRAM by local direct memory access (LDMA) and to access a memory of the cluster partner by remote direct memory access (RDMA), in one Peripheral Connect Interface (PCI) transaction.

7. A network storage system as recited in claim 6, wherein the cluster interconnect adapter is InfiniBand-compliant, and wherein the local identifier is an InfiniBand local identifier (LID).

8. A network storage system as recited in claim 6, wherein the local identifier further is used to internally communicate within the network storage system.

9. A network storage system as recited in claim 6, wherein the local identifier further is used to internally communicate between the NVRAM and a system memory of the network storage system.

10. A network storage system as recited in claim 7, wherein the control circuitry further is to send the LID to the cluster partner in an InfiniBand Subnet Management Packet (SMP), to inform the cluster partner of the LID.

11. A network storage system comprising:

a processor;

a system memory coupled to the processor;

a network adapter coupled to the processor, through which to receive a write request from a host over a network;

a storage adapter coupled to the processor, through which to access data stored in a mass storage facility in response to the write request;

an expansion bus coupled to the processor; and

an InfiniBand-compliant node coupled to the expansion bus, the InfiniBand-compliant node including;

a non-volatile random access memory (NVRAM) to receive data representing the write request from the system memory during a local DMA transaction,

a cluster interconnect adapter, through which to send data to a remote cluster partner;

means for reading a persistent serial number stored in the InfiniBand-compliant node;

means for assigning at least a portion of the serial number as an InfiniBand local identifier (LID) of a port of the InfiniBand-compliant node;

means for using the LID for communication between the system memory and the NVRAM during the local DMA transaction;

means for using the LID for direct memory access (DMA) to both the NVRAM and a memory of the remote cluster partner in one Peripheral Connect Interface (PCI) transaction; and

means for sending the LID to the remote cluster partner in an InfiniBand Subnet Management Packet (SMP), to inform the remote cluster partner of the LID.

12. A network storage system as recited in claim 11, said assigning requires without communication between the network storage system and any entity external to the network storage system.

13. A network storage system as recited in claim 11, wherein the NVRAM stores a log of write requests received by the network storage system from one or more hosts.

14. A method comprising:

connecting a network storage system with a remote cluster partner of the network storage system via a port of an Infiniband-compliant cluster interconnect adapter, wherein the network storage system contains the cluster interconnect adapter and a non-volatile memory (NVRAM), and the NVRAM is embedded in the cluster interconnect adapter;

assigning an InfiniBand local identifier (LID) to the port of the cluster interconnect adapter, wherein the LID is generated without requiring any communication between the cluster interconnect adapter and any entity external to the network storage system, and

wherein assigning the LID includes reading a persistent serial number from the cluster interconnect adapter and assigning at least a portion of the serial number as the LID;

sending the LID to the remote cluster partner in an Infiniband Subnet Management Packet (SMP) to inform the remote cluster partner of the LID; and

using the LID to access the NVRAM by local direct memory access (LDMA) and to access a memory of the remote cluster partner by remote direct memory access (RDMA).

15. The method as recited in claim 14, wherein a single driver performs Direct Memory Access to both the NVRAM and the memory of the remote cluster partner.

16. The method as recited in claim 14, wherein Direct Memory Access to both the NVRAM and the memory of the remote cluster partner is performed in one Peripheral Connect Interface (PCI) transaction.