Redundant, fault-tolerant, distributed remote procedure call cache in a storage system

• US 9,021,297 B1
• Filed: 07/02/2014
• Issued: 04/28/2015
• Est. Priority Date: 07/02/2014
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Pure Storage, Inc.
• Inventors: Hayes, John, Lee, Robert, Vajgel, Peter, Robinson, Joshua
• Primary Examiner(s): Wilson, Yolanda L
• Application Number:
  US14/322,888
• Time in Patent Office: 300 Days
• Field of Search: 714/5.1, 714/6.3, 719/330
• US Class Current: 714/6.3
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 11/1666 where the redundant component is memory or memory area
• CPC Class Codes: G06F 11/2056 by mirroring
• CPC Class Codes: G06F 11/2058 using more than 2 mirrored copies
• CPC Class Codes: G06F 11/2069 Management of state, configuration or failover
• CPC Class Codes: G06F 11/2089 Redundant storage control functionality
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: G06F 12/0246 in block erasable memory, e.g. flash memory
• CPC Class Codes: G06F 12/0813 with a network or matrix configuration
• CPC Class Codes: G06F 12/0875 with dedicated cache, e.g. instruction or stack
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 2212/1032 Reliability improvement, data loss prevention, degraded operation etc
• CPC Class Codes: G06F 2212/154 Networked environment
• CPC Class Codes: G06F 2212/284 being distributed
• CPC Class Codes: G06F 2212/286 Mirrored cache memory
• CPC Class Codes: G06F 2212/452 Instruction code
• CPC Class Codes: G06F 2212/7208 Multiple device management, e.g. distributing data over multiple flash devices

Abstract

A method of operating a remote procedure call cache in a storage cluster is provided. The method includes receiving a remote procedure call at a first storage node having solid-state memory and writing information, relating to the remote procedure call, to a remote procedure call cache of the first storage node. The method includes mirroring the remote procedure call cache of the first storage node in a mirrored remote procedure call cache of a second storage node. The storage cluster includes the first storage node and the second storage node. The storage system includes the first storage blade and the second storage blade. A storage node and a storage cluster are also provided.

Claims

1. A plurality of storage nodes in a single chassis, comprising:

the plurality of storage nodes configured to communicate together as a storage cluster;

each of the plurality of storage nodes having non-volatile solid-state memory for user data storage, a remote procedure call cache, and a mirrored remote procedure call cache configured to mirror a remote procedure call cache of another storage node; and

the plurality of storage nodes configured to distribute the user data and metadata associated with the user data throughout the plurality of storage nodes such that the plurality of storage nodes maintain ability to read the user data, using erasure coding, despite a loss of two of the plurality of storage nodes.

2. The plurality of storage nodes in a single chassis of claim 1, further comprising:

each of the plurality of storage nodes having a table, configured to indicate a primary authority, a first backup authority, and a second backup authority, wherein the remote procedure call cache corresponds to the primary authority.

3. The plurality of storage nodes in a single chassis of claim 1, wherein each of the plurality of storage nodes is configured to send a copy of contents of the remote procedure call cache to a further storage node for storage in the mirrored remote procedure call cache of the further storage node.

4. The plurality of storage nodes in a single chassis of claim 1, further comprising:

each of the plurality of storage nodes configured to check the remote procedure call cache and to determine based on the check whether a result of a remote procedure call is already posted.

5. The plurality of storage nodes in a single chassis of claim 1, further comprising:

each of the plurality of storage nodes having a non-volatile random access memory (NVRAM) containing the remote procedure call cache and the mirrored remote procedure call cache.

6. The plurality of storage nodes in a single chassis of claim 1, further comprising:

the remote procedure call cache configured to store, as metadata relating to a remote procedure call, a transaction identifier, a client identifier, and a result.

7. The plurality of storage nodes in a single chassis of claim 1, further comprising:

each of the plurality of storage nodes configured to determine whether a mirrored remote procedure call cache is unreachable and to mirror the remote procedure call cache to a replacement mirrored remote procedure call cache responsive to the remote procedure call cache being unreachable.

8. A storage cluster, comprising:

a plurality of storage nodes within a single chassis;

each storage node of the plurality of storage nodes having non-volatile solid-state memory for storage of user data and configured to access data according to remote procedure calls, a remote procedure call cache, and a mirrored remote procedure call cache;

each storage node of the plurality of storage nodes configured to mirror the remote procedure call cache in the mirrored remote procedure call cache of a differing one of the plurality of storage nodes; and

the plurality of storage nodes configured to distribute the user data and metadata associated with the user data throughout the plurality of storage nodes such that the plurality of storage nodes can access the user data, via erasure coding, with a failure of two of the plurality of storage nodes.

9. The storage cluster of claim 8, further comprising:

each storage node of the plurality of storage nodes configured to mirror, in the mirrored remote procedure call cache of the each storage node, the remote procedure call cache of a further one of the plurality of storage nodes.

10. The storage cluster of claim 8, further comprising:

a switch configured to parse the remote procedure calls and to route the remote procedure calls to the plurality of storage nodes.

11. The storage cluster of claim 8, further comprising:

each of the plurality of storage nodes having a table configured to indicate which storage node of the plurality of storage nodes has the remote procedure call cache corresponding to a specified data; and

each storage node configured to forward a remote procedure call to one of the plurality of storage nodes in accordance with the table.

12. The storage cluster of claim 8, further comprising:

each of the plurality of storage nodes configured to determine whether a result of a remote procedure call is available in the remote procedure call cache, and to respond to the remote procedure call with the result in an event that the result is available in the remote procedure call cache.

13. The storage cluster of claim 8, further comprising:

the plurality of storage nodes configured to determine a replacement mirrored remote procedure call cache responsive to the remote procedure call cache of a one of the plurality of storage nodes being unreachable, wherein the plurality of storage nodes obtains a level of redundancy of the remote procedure call across the plurality of storage nodes.

14. A method of operating a remote procedure call cache in a storage cluster having a plurality of storage nodes each having non-volatile solid-state storage, comprising:

distributing user data throughout the plurality of storage nodes through erasure coding, wherein the plurality of storage nodes are housed within a single chassis that couples the storage nodes as the storage cluster;

receiving a remote procedure call at a first one of the plurality of storage nodes;

writing information, relating to the remote procedure call, to a remote procedure call cache of the first one of the plurality of storage nodes;

mirroring the remote procedure call cache of the first one of the plurality of storage nodes in a mirrored remote procedure call cache of a second one of the plurality of storage nodes,

wherein a processor performs at least one method operation.

15. The method of claim 14, further comprising:

mirroring the remote procedure call cache of the first one of the plurality of storage nodes in a mirrored remote procedure call cache of a third one of the plurality of storage nodes.

16. The method of claim 14, further comprising:

mirroring remote procedure call caches of two further storage nodes in two mirrored remote procedure call caches of the first one of the plurality of storage nodes.

17. The method of claim 14, further comprising:

determining a replacement mirrored remote procedure call cache, in accordance with a table, responsive to the mirrored remote procedure call cache of the second one of the plurality of storage nodes being unreachable;

mirroring the remote procedure call cache of the first one of the plurality of storage nodes to the replacement mirrored remote procedure call cache.

18. The method of claim 14, further comprising:

routing the remote procedure call to the first one of the plurality of storage nodes in accordance with the first one of the plurality of storage nodes having authority for files in a first filesystem, wherein the second one of the plurality of storage nodes has authority for files in a second filesystem, the first filesystem differing from the second filesystem.

19. The method of claim 14, further comprising:

routing the remote procedure call from a switch to the first one of the plurality of storage nodes, based on a first inode number; and

routing a second remote procedure call from the switch to the second one of the plurality of storage nodes, based on a second inode number.

20. The method of claim 14, further comprising:

receiving a further remote procedure call at the first one of the plurality of storage nodes;

checking the remote procedure call cache of the first one of the plurality of storage nodes as to whether a result of the remote procedure call is already available; and

responding to the remote procedure call with the result, responsive to a determination that the result of the remote procedure call is already available.