Error recovery in a storage cluster

• US 10,324,812 B2
• Filed: 11/14/2016
• Issued: 06/18/2019
• Est. Priority Date: 08/07/2014
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Pure Storage, Inc.
• Inventors: Davis, John D., Hayes, John, Kannan, Hari, Miladinovic, Nenad, Tan, Zhangxi
• Primary Examiner(s): Ko, Chae M
• Application Number:
  US15/351,303
• Publication Number:
  US 20170060711A1
• Time in Patent Office: 946 Days
• Field of Search: 714 62, 714 624
• CPC Class Codes: G06F 11/1068 in sector programmable memories, e.g. flash disk G06F11/1072 takes precedence
• CPC Class Codes: G06F 11/1072 in multilevel memories
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1612 where the redundant component is persistent storage
• CPC Class Codes: G06F 11/2069 Management of state, configuration or failover
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: G06F 2201/82 Solving problems relating to consistency ensuring consistency in mirrored systems G06F11/2064
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0685 Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD

Abstract

A plurality of storage nodes within a single chassis is provided. The plurality of storage nodes is configured to communicate together as a storage cluster. The plurality of storage nodes has a non-volatile solid-state storage for user data storage. The plurality of storage nodes is configured to distribute the user data and metadata associated with the user data throughout the plurality of storage nodes, with erasure coding of the user data. The plurality of storage nodes is configured to recover from failure of two of the plurality of storage nodes by applying the erasure coding to the user data from a remainder of the plurality of storage nodes. The plurality of storage nodes is configured to detect an error and engage in an error recovery via one of a processor of one of the plurality of storage nodes, a processor of the non-volatile solid state storage, or the flash memory.

Claims

1. A method for error recovery in a storage system, performed by a plurality of storage nodes of the storage system, comprising:

detecting a failure of a first one of the plurality of storage nodes having a remote procedure call cache; and

accessing a mirrored remote procedure call cache in a second one of the plurality of storage nodes, with the mirrored remote procedure call cache mirroring the remote procedure call cache of the first one of the plurality of storage nodes.

2. The method of claim 1, further comprising:

determining, from the mirrored remote procedure call cache, whether a result of servicing a remote procedure call is already posted.

3. The method of claim 1, further comprising:

returning a result of servicing a remote procedure call, responsive to determining from the mirrored remote procedure call cache that the result for the remote procedure call is posted.

4. The method of claim 1, further comprising:

servicing a remote procedure call, responsive to determining from the mirrored remote procedure call cache that a result for the remote procedure call is not posted.

5. The method of claim 1, further comprising:

detecting a failure of a third one of the plurality of storage nodes having a further mirrored remote procedure call cache mirroring the remote procedure call cache of the first one of the plurality of storage nodes, wherein the accessing the mirrored remote procedure call cache in the second one of the plurality of storage nodes is responsive to the detecting the failure of the first and third ones of the plurality of storage nodes.

6. The method of claim 1, further comprising:

determining, from a table in the second one or a third one of the plurality of storage nodes, that the second one of the plurality of storage nodes has the mirrored remote procedure call cache.

7. The method of claim 1, wherein the failure of the first one of the plurality of storage nodes having the remote procedure call cache is one of a failure of a memory holding the remote procedure call cache, a failure of a non-volatile solid-state storage, or a failure of a storage node.

8. A storage system, comprising:

a plurality of storage nodes, each having storage memory and a remote procedure call cache; and

each of the plurality of storage nodes configured to determine, responsive to a failure of one of the plurality of storage nodes or the remote procedure call cache therein, which other one of the plurality of storage nodes has a mirrored remote procedure call cache that mirrors the remote procedure call cache of the failed one of the plurality of storage nodes, and to access the mirrored remote procedure call cache.

9. The storage system of claim 8, further comprising:

each of the plurality of storage nodes configured to determine, responsive to a failure of the one of the plurality of storage nodes having the remote procedure call cache and failure of the other one of the plurality of storage nodes having the mirrored remote procedure call cache, which further one of the plurality of storage nodes has a replacement mirrored remote procedure call cache that mirrors the remote procedure call cache, and to access the replacement mirrored remote procedure call cache.

10. The storage system of claim 8, further comprising:

at least one of the plurality of storage nodes configured to return error corrected data from a data read as a response to a remote procedure call, responsive to the failure of the one of the plurality of storage nodes or the remote procedure call cache therein, and a determination that the error corrected data is available as a result from the mirrored remote procedure call cache.

11. The storage system of claim 8, further comprising:

at least one of the plurality of storage nodes configured to service a remote procedure call and return error corrected data, responsive to the failure of the one of the plurality of storage nodes or the remote procedure call cache therein, and a determination that the error corrected data is not available as a result from the mirrored remote procedure call cache.

12. The storage system of claim 8, further comprising:

each of the plurality of storage nodes configured to have a table for determining which of the plurality of storage nodes has the mirrored remote procedure call cache and which of the plurality of storage nodes has a further mirrored remote procedure call cache.

13. The storage system of claim 8, further comprising:

the remote procedure call cache and the mirrored remote procedure call cache each configured to contain a transaction identifier identifying a transaction of a remote procedure call message, and a result of completing an action for the remote procedure call message.

14. The storage system of claim 8, further comprising:

at least one of the plurality of storage nodes configured to answer a repeated remote procedure call and return a result available from the mirrored remote procedure call cache, responsive to the failure of the one of the plurality of storage nodes or the remote procedure call cache therein.