Multicasting message in a network storage system to local NVRAM and remote cluster partner

• US 7,962,562 B1
• Filed: 04/30/2004
• Issued: 06/14/2011
• Est. Priority Date: 04/30/2004
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Bali, Naveen, Budhia, Ravi K.
• Primary Examiner(s): Dharia; Rupal D
• Assistant Examiner(s): McAdams; Robert B
• Application Number:
  US10/837,149
• Time in Patent Office: 2,601 Days
• Field of Search: 709/212, 710/22
• US Class Current: 709/212
• CPC Class Codes: G06F 3/0613 in relation to throughput
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

A network storage system includes a non-volatile memory to store data including a log of received data access requests, and a cluster interconnect adapter through which to send data to a cluster partner. The nonvolatile memory and the cluster interconnect adapter are implemented in a single device connected to an expansion bus in the network storage system. Communication with the nonvolatile memory is carried out using LDMA, and communication with the cluster partner is carried out using RDMA via the cluster interconnect adapter. LDMA and RDMA functionality are merged in a common software stack.

Claims

1. A method comprising:

generating a message in a first network storage server; and

multicasting the message by simultaneously transmitting, using a single PCI transaction, the message from a system memory of the first network storage server to a first cluster interconnect device in the first network storage server and a second cluster interconnect device in the second network storage server, wherein the first cluster interconnect device includes a first non-volatile memory, embedded in the first cluster interconnect device, for receiving and storing the message received by the first cluster interconnect device, wherein the second cluster interconnect device includes a second non-volatile memory, embedded in the second cluster interconnect device, for receiving and storing the message received by the second cluster interconnect device, wherein the second network storage server is configured as a cluster partner of the first network storage server, wherein the message is transmitted from the first network storage server to the second network storage server via a cluster interconnect coupling the first cluster interconnect device to the second cluster interconnect device, and wherein the first non-volatile memory and the second non-volatile memory are destinations of the multicasting.

2. A method as recited in claim 1, wherein the first non-volatile memory is a non-volatile random access memory (NVRAM), and the second non-volatile memory is an NVRAM.

3. A method as recited in claim 1, wherein multicasting the message includes communicating data stored in a system memory of the first network storage server to the first non-volatile memory and to the second non-volatile memory.

4. A method as recited in claim 1, wherein the message is an INFINIBAND message.

5. A method comprising:

executing a local direct memory access (DMA) operation in a first network storage server to transmit data from a system memory of the first network storage server to a first cluster interconnect device in the first network storage server to store data in a first non-volatile memory embedded in the first cluster interconnect device in the first network storage server;

wherein the first cluster interconnect device couples the first network storage server to a cluster interconnect; and

executing a remote DMA operation, concurrently with the local DMA operation in a single PCI transaction, from the first network storage server to transmit data from the system memory of the first network storage server to a second cluster interconnect device in the second network storage server to store data in a second non-volatile memory embedded in the second cluster interconnect device in the second network storage server, wherein the second network storage server is configured as a cluster partner of the first network storage server, wherein the remote DMA operation is executed via the cluster interconnect, wherein the data is transmitted to the second cluster interconnect device using the remote DMA operation without waiting for the data to be first copied into the first non-volatile memory, wherein the local DMA operation and the remote DMA operation are portions of a multicasting operation, and wherein the first non-volatile memory and the second non-volatile memory are destinations of the multicasting operation.

6. A method as recited in claim 5, wherein the local DMA operation and the remote DMA operation are executed in response to receiving, at the first network storage server, a data access request from a remote client of the first network storage server.

7. A method as recited in claim 6, wherein the data stored in the first non-volatile memory and the second non-volatile memory represents the request.

8. A method as recited in claim 7, wherein the request is a write request relating to data stored by the first network storage server.

9. A method as recited in claim 8, wherein the first non-volatile memory is a non-volatile random access memory (NVRAM).

10. A method as recited in claim 9, wherein the data is stored in the first non-volatile memory and the second non-volatile memory as part of a log of data access requests received by the first network storage server.

11. A method as recited in claim 5, wherein the local DMA operation and the remote DMA operation each includes sending the data in an INFINIBAND message.

12. A first network storage server comprising:

a network interface through which to receive data access requests from a plurality of remote hosts over a network;

a storage interface through which to access a set of mass storage devices according to the data access requests;

a first cluster interconnect device through which to communicate with a second network storage server configured as a cluster partner of the first network storage server over a cluster interconnect, wherein a first non-volatile memory is embedded in the cluster interconnect device; and

a processor to generate a message and to multicast the message by simultaneously transmitting, using a single PCI transaction, the message from a system memory of the first network storage server to the first cluster interconnect device in the first network storage device and a second cluster interconnect device in the second network storage device, wherein the first cluster interconnect device receives and stores the message in the first non-volatile memory in the first cluster interconnect device and the second cluster interconnect device stores and receives the message in a second non-volatile memory embedded in the second cluster interconnect device of the second network storage server, wherein the first non-volatile memory and the second non-volatile memory are multicast destinations of the message, wherein the message is transmitted to the second cluster interconnect device using without waiting for the message to be first stored in the first non-volatile memory.

13. A network storage server as recited in claim 12, wherein the message is an INFINIBAND message.

14. A first network storage server as recited in claim 12, further comprising a system memory, wherein multicasting the message includes concurrently communicating data stored in the system memory to the first non-volatile memory and to the second non-volatile memory.

15. A first network storage server as recited in claim 12, wherein the first non-volatile memory is a non-volatile random access memory (NVRAM).

16. A first network storage server as recited in claim 15, further comprising a system memory, wherein multicasting the message includes concurrently communicating data stored in the system memory to the NVRAM and to the second non-volatile memory.

17. A first network storage server as recited in claim 16, wherein the message is an INFINIBAND message.

18. A network storage system comprising:

means for generating a message in a first network storage server; and

means for multicasting the message by simultaneously transmitting, using a single PCI transaction, the message from a system memory of the first network storage server to a first cluster interconnect device in the first network storage server and a second cluster interconnect device in the second network storage server, wherein the first cluster interconnect device includes a first non-volatile memory, embedded in the first cluster interconnect device, for receiving and storing the message received by the first cluster interconnect device, wherein the second cluster interconnect device includes a second non-volatile memory, embedded in the second cluster interconnect device, for receiving and storing the message received by the second cluster interconnect device, wherein the second network storage server is configured as a cluster partner of the first network storage server in the network storage system, wherein the message is transmitted from the first network storage server to the second network storage server via a cluster interconnect coupling the first cluster interconnect device to the second cluster interconnect device, and wherein the first non-volatile memory and the second non-volatile memory are destinations of the multicasting.

19. A network storage system as recited in claim 18, wherein the first non-volatile memory is a non-volatile random access memory (NVRAM), and the second non-volatile memory is an NVRAM.

20. A network storage system as recited in claim 18, wherein multicasting the message includes communicating data stored in a system memory of the first network storage server to the first non-volatile memory and to the second non-volatile memory.

21. A network storage system as recited in claim 20, wherein the first non-volatile memory is a non-volatile random access memory (NVRAM), and the second non-volatile memory is an NVRAM.

22. A network storage system as recited in claim 18, wherein the message is an INFINIBAND message.