Live NV replay for enabling high performance and efficient takeover in multi-node storage cluster

• US 9,342,417 B2
• Filed: 05/16/2014
• Issued: 05/17/2016
• Est. Priority Date: 03/13/2014
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Usgaonkar, Ameya Prakash, Nandi, Siddhartha
• Primary Examiner(s): BUTLER, SARAI E
• Application Number:
  US14/280,139
• Publication Number:
  US 20150261633A1
• Time in Patent Office: 732 Days
• Field of Search: 714/4.1
• US Class Current: 1/1
• CPC Class Codes: G06F 11/2005 using redundant communication controllers
• CPC Class Codes: G06F 11/2007 using redundant communication media
• CPC Class Codes: G06F 11/2064 while ensuring consistency
• CPC Class Codes: G06F 11/2071 using a plurality of controllers
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A live non-volatile (NV) replay technique enables a partner node to efficiently takeover a failed node of a high-availability pair in a multi-node storage cluster by dynamically replaying operations synchronously logged in a non-volatile random access memory (NVRAM) of the partner node, while also providing high performance during normal operation. Dynamic live replay may be effected through interpretation of metadata describing the logged operations. The metadata may specify a location and type of each logged operation within a partner portion of the NVRAM, as well as any dependency among the logged operation and any other logged operations that would impose an ordering constraint. During normal operation, the partner node may consult the metadata to identify dependent logged operations and dynamically replay those operations to satisfy one or more requests. Upon failure of the node, the partner node may replay, in parallel, those logged operations having no imposed ordering constraint, thereby reducing time needed to complete takeover of the failed node.

Claims

1. A system comprising:

one or more processors of a partner node interconnected to a local node of a multi-node storage cluster;

a non-volatile random access memory (NVRAM) connected to the one or more processors, the NVRAM configured to maintain operations serviced and logged by the local node, but not committed to storage devices of the local node; and

a memory coupled to the one or more processors and configured to store metadata describing a type of each operation logged in the NVRAM, the metadata further describing a location of the logged operation in the NVRAM and any dependency among the logged operation and other logged operations in the NVRAM, the memory further configured to store a storage operating system implemented as one or processes executable by the one or more processors to, upon failure of the local node, interpret the metadata to dynamically replay in parallel any logged operations not constrained by an ordering requirement.

2. The system of claim 1 wherein the NVRAM is organized into a local portion and a partner portion, and wherein the logged operations serviced by the local node are maintained in the partner portion of the NVRAM.

3. The system of claim 1 wherein the metadata is stored in a dependency index table of the memory.

4. The system of claim 3 wherein the one or more processes comprises a semantic replay parser configured to interpret the metadata stored in the dependency index table.

5. The system of claim 4 wherein the semantic replay parser cooperates with a file system of the storage operating system to replay the logged operations not constrained by the ordering requirement.

6. A method comprising:

executing an operation at a local node of multi-node storage cluster, the local node coupled to a partner node of the cluster by a high availability (HA) interconnect;

logging the operation in a first non-volatile random access memory (NVRAM) of the local node;

generating metadata describing the logged operation at the local node;

mirroring the logged operation from the first NVRAM to a second NVRAM on the partner node;

transferring the metadata to a dependency index data structure of the partner node;

receiving a request issued by an application at the partner node to redirect servicing of the request to the local node;

interpreting the metadata at a semantic replay parser of the partner node to determine whether replay of the logged operation requires ordering due to a dependency among other operations logged in the second NVRAM; and

dynamically replaying one or more of the logged operations having the dependency to service the request at the partner node without redirecting service to the local node.

7. The method of claim 6 wherein logging the request comprises:

logging the operation in a local portion of the first NVRAM.

8. The method of claim 7 wherein generating the metadata comprises:

generating the metadata to include a type of the operation, a location of the operation logged in the local portion of the first NVRAM and any dependency among the operation and the other operations logged in the local portion of the first NVRAM.

9. The method of claim 8 wherein mirroring comprises:

mirroring the operation from the location in the local portion of the first NVRAM to a corresponding location in a partner portion of a second NVRAM on the partner node.

10. The method of claim 9 further comprising:

determining whether ordered replay of the request is required due to overlap among the operation and the other operations that would impose an ordering constraint.

11. The method of claim 10 further comprising:

if ordering is not required, arranging the operation for parallel replay with the other operations upon failure of the local node.

12. The method of claim 10 further comprising:

if ordering is required, arranging the overlapping requests for sequential replay upon failure of the local node.

13. A system comprising:

one or more processors of a local node interconnected by a cluster switching fabric to a plurality of nodes, wherein the local node is further coupled to a partner node by a high availability (HA) interconnect;

a first non-volatile random access memory (NVRAM) connected to the one or more processors, the first NVRAM having a local portion configured to maintain operations logged and serviced by the local node, but not yet committed to storage devices of the local node; and

a memory coupled to the one or more processors and configured to store a storage operating system implemented as one or processes executable by the one or more processors to generate metadata describing a type of each operation logged in the local portion of the first NVRAM, the metadata further describing a location of each logged operation in the local portion of the first NVRAM and any dependency among the logged operations in the local portion of the first NVRAM.

14. The system of claim 13 further comprising a direct memory access (DMA) controller configured to mirror each logged operation over the HA interconnect to a partner portion of a second NVRAM on the partner node.

15. The system of claim 14 wherein the DMA controller is further configured to transfer the metadata over the HA interconnect to the partner node to synchronize the local and partner nodes with respect to the logged operation.

16. The system of claim 15 wherein the DMA controller is configured to mirror the logged operation and transfer the metadata using a remote direct memory access (RDMA) protocol.

17. The system of claim 16 wherein the metadata is transferred over the HA interconnect and loaded into a data structure of the partner node using a RDMA message that provides an offset of the metadata into the data structure.

18. The system of claim 16 wherein the metadata is transferred over the HA interconnect as part of the mirror of the logged operation.

19. The system of claim 18 wherein the metadata is loaded into a data structure of the partner node in response to an interrupt generated by the RDMA protocol once the mirror completes.

20. The system of claim 19 wherein the interrupt provides an offset of the metadata in the data structure.