Die-level monitoring in a storage cluster

• US 9,082,512 B1
• Filed: 08/07/2014
• Issued: 07/14/2015
• Est. Priority Date: 08/07/2014
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Pure Storage, Inc.
• Inventors: Hayes, John, Tan, Zhangxi, Davis, John D., Kannan, Hari, Miladinovic, Nenad
• Primary Examiner(s): Trimmings, John P
• Application Number:
  US14/454,522
• Time in Patent Office: 341 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/076 by exceeding a count or rate limit, e.g. word- or bit count limit
• CPC Class Codes: G06F 11/1048 using arrangements adapted for a specific error detection or correction feature
• CPC Class Codes: G06F 11/1068 in sector programmable memories, e.g. flash disk G06F11/1072 takes precedence
• CPC Class Codes: G06F 12/0246 in block erasable memory, e.g. flash memory
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/1827 Management specifically adapted to NAS management of storage area networks [SAN] G06F3/067
• CPC Class Codes: G06F 2212/7207 management of metadata or control data
• CPC Class Codes: G06F 2212/7208 Multiple device management, e.g. distributing data over multiple flash devices
• CPC Class Codes: G06F 2212/7211 Wear leveling
• CPC Class Codes: G11C 16/349 Arrangements for evaluating degradation, retention or wearout, e.g. by counting erase cycles
• CPC Class Codes: G11C 2029/0409 Online test
• CPC Class Codes: G11C 2029/0411 Online error correction
• CPC Class Codes: G11C 29/44 Indication or identification of errors, e.g. for repair
• CPC Class Codes: G11C 29/52 Protection of memory contents; Detection of errors in memory contents

Abstract

In some embodiments, a method for die-level monitoring is provided. The method includes distributing user data throughout a plurality of storage nodes through erasure coding, wherein the plurality of storage nodes are housed within a chassis that couples the storage nodes. Each of the storage nodes has a non-volatile solid-state storage with non-volatile memory and the user data is accessible via the erasure coding from a remainder of the storage nodes in event of two of the storage nodes being unreachable. The method includes producing diagnostic information that diagnoses the non-volatile memory on a basis of per package, per die, per plane, per block, or per page, the producing performed by each of the plurality of storage nodes. The method includes writing the diagnostic information to a memory in the storage cluster.

Claims

1. A method for die-level monitoring in a storage cluster, comprising:

distributing user data throughout a plurality of storage nodes of the storage cluster through erasure coding;

performing, in a controller in a non-volatile solid-state storage, error correction of reads of a non-volatile memory; and

producing diagnostic information that diagnoses the non-volatile memory of the non-volatile solid-state storage of each of the plurality of storage nodes, based on the performing.

2. The method of claim 1, further comprising:

characterizing the non-volatile memory over time, based on the diagnostic information.

3. The method of claim 1, further comprising:

sending at least a portion of the diagnostic information from the storage cluster to a network.

4. The method of claim 1, further comprising:

biasing at least one of a read from or a write to the non-volatile memory, responsive to the producing the diagnostic information.

5. The method of claim 1, further comprising:

tracking wear of a plurality of blocks of the non-volatile memory, based on the diagnostic information.

6. The method of claim 1, wherein producing the diagnostic information further comprises:

tracking bit errors per page or per codeword, each page having a plurality of codewords; and

forwarding information pertaining to the bit errors from the non-volatile solid-state storage to a processor of a storage node, for each of the plurality of storage nodes, wherein the memory in the storage cluster includes a memory coupled to the processor of the storage node.

7. The method of claim 1, further comprising:

reporting incidents of the error correction, wherein the diagnostic information is based on the incidents of the error correction.

8. The method of claim 1, further comprising:

determining to which of a plurality of blocks to write a portion of the user data, based on the diagnostic information.

9. A plurality of storage nodes in a chassis, comprising:

the plurality of storage nodes configured to communicate together as a storage cluster, each of the plurality of storage nodes having non-volatile memory, the plurality of storage nodes configured to distribute user data and metadata associated with the user data throughout the plurality of storage nodes using erasure coding;

each of the plurality of storage nodes configured to determine diagnostic information from the non-volatile memory at a level of one of a package, a die, a plane, a block, or a page; and

each non-volatile solid-state storage having a controller configured to track bit errors, wherein the diagnostic information is based at least in part on the bit errors.

10. The plurality of storage nodes in a chassis of claim 9, further comprising:

the plurality of storage nodes configured to derive characterization information from the diagnostic information, the characterization information indicating at least one change, prediction or trend in the non-volatile memory.

11. The plurality of storage nodes in a chassis of claim 9, further comprising:

the plurality of storage nodes configured to report the diagnostic information via a network.

12. The plurality of storage nodes in a chassis of claim 9, further comprising:

each of the plurality of storage nodes having a non-volatile solid-state storage, wherein the non-volatile memory is included in the non-volatile solid-state storage;

each of the plurality of storage nodes having a processor;

the controller of each non-volatile solid-state storage configured to forward information regarding the bit errors to the processor; and

the plurality of storage nodes configured to collect and update the diagnostic information.

13. The plurality of storage nodes in a chassis of claim 9, further comprising:

each of the plurality of storage nodes configured to direct a portion of the non-volatile memory to change one of;

a variable write time, a variable write voltage, a variable read time, a variable reference voltage, a variable reference current, or a variable number of reads, based on the diagnostic information.

14. The plurality of storage nodes in a chassis of claim 9, further comprising:

a wear list, coupled to the plurality of storage nodes and configured to categorize blocks of the non-volatile memory by wear level;

a retention list, coupled to the plurality of storage nodes and configured to track data retention in the non-volatile memory; and

the plurality of storage nodes configured to update the wear list responsive to the diagnostic information.

15. A storage cluster, comprising:

a plurality of storage nodes within a chassis, each of the plurality of storage nodes having non-volatile memory, the plurality of storage nodes configured to distribute user data and metadata associated with the user data throughout the plurality of storage nodes;

each of the plurality of storage nodes having a controller with an error correction block configured to perform error correction on a data stripe, wherein diagnostic information generated by the storage nodes is based at least in part on a result of the error correction block.

16. The storage cluster of claim 15, further comprising:

the plurality of storage nodes configured to track the diagnostic information over time, wherein the diagnostic information is based on feedback, regarding bit errors, from the error correction block of each of the plurality of storage nodes, and wherein the error correction performed on the data stripe includes reads of the non-volatile memory across the plurality of storage nodes.

17. The storage cluster of claim 15, further comprising:

each of the plurality of storage nodes configured to determine relative wear of blocks; and

each of the plurality of storage nodes configured to direct a write to a second block, based on a determination that the second block has a lower wear than a first block according to the diagnostic information.

18. The storage cluster of claim 15, further comprising:

each of the plurality of storage nodes configured to direct a portion of the non-volatile memory to change a read reference voltage or a read reference current, responsive to a determination that the portion of the non-volatile memory has one of a greater wear, a greater retention time, a greater read time, or a greater write time, than a further portion of the non-volatile memory according to the diagnostic information.

19. The storage cluster of claim 15, further comprising:

each of the plurality of storage nodes configured to direct a portion of the non-volatile memory to change from multilevel cell (MLC) operation to single level cell (SLC) operation, responsive to a determination that the portion of the non-volatile memory has a greater wear than a further portion of the non-volatile memory according to the diagnostic information.

20. The storage cluster of claim 15, further comprising:

the plurality of storage nodes configured to track trends or changes of the non-volatile memory based on the diagnostic information; and

the plurality of storage nodes configured to derive characterization information from the diagnostic information pertaining to the non-volatile memory, and from one of application data or meta data, pertaining to the user data.