Using reason codes to determine how to handle memory device error conditions

• US 10,282,118 B2
• Filed: 11/03/2016
• Issued: 05/07/2019
• Est. Priority Date: 12/29/2009
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Baptist, Andrew D., Iljazi, Ilir, McShane, Niall J., Resch, Jason K.
• Primary Examiner(s): Parry, Chris
• Assistant Examiner(s): Stiltner, Weiwei Y
• Application Number:
  US15/342,633
• Publication Number:
  US 20170052842A1
• Time in Patent Office: 915 Days
• Field of Search: 709213, 714 42, 714 52, 714 54
• CPC Class Codes: G06F 1/3296 by lowering the supply or operating voltage
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0772 Means for error signaling, e.g. using interrupts, exception flags, dedicated error registers
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/22 Detection or location of defective computer hardware by testing during standby operation or during idle time, e.g. start-up testing
• CPC Class Codes: G06F 11/2221 to test input/output devices or peripheral units
• CPC Class Codes: G06F 11/28 by checking the correct order of processing G06F11/08 - G06F11/26 take precedence; monitoring patterns of pulse trains H03K5/19
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/184 implemented as replicated file system
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 21/10 Protecting distributed programs or content, e.g. vending or licensing of copyrighted material protection in video systems or pay television H04N7/16 ; Digital rights management [DRM]
• CPC Class Codes: G06F 21/554 involving event detection and direct action
• CPC Class Codes: G06F 21/80 in storage media based on magnetic or optical technology, e.g. disks with sectors preventing unauthorised reproduction or copying of disc-type recordable media G11B20/00
• CPC Class Codes: G06F 2221/033 Test or assess software
• CPC Class Codes: G06F 2221/034 Test or assess a computer or a system
• CPC Class Codes: G06F 2221/2129 : Authenticate client device independently of the user
• CPC Class Codes: G06F 2221/2151 : Time stamp
• CPC Class Codes: G06F 3/0614 : Improving the reliability of storage systems
• CPC Class Codes: G06F 3/0619 : in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0625 : Power saving in storage systems
• CPC Class Codes: G06F 3/064 : Management of blocks
• CPC Class Codes: G06F 3/0647 : Migration mechanisms
• CPC Class Codes: G06F 3/065 : Replication mechanisms
• CPC Class Codes: G06F 3/0653 : Monitoring storage devices or systems
• CPC Class Codes: G06F 3/067 : Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0683 : Plurality of storage devices
• CPC Class Codes: H03M 13/1515 : Reed-Solomon codes
• CPC Class Codes: H04L 1/16 : in which the return channel carries supervisory signals, e.g. repetition request signals
• CPC Class Codes: H04L 2209/34 : Encoding or coding, e.g. Huffman coding or error correction
• CPC Class Codes: H04L 41/0661 : by reconfiguring faulty entities
• CPC Class Codes: H04L 43/16 : Threshold monitoring
• CPC Class Codes: H04L 63/101 : Access control lists [ACL]
• CPC Class Codes: H04L 63/102 : Entity profiles
• CPC Class Codes: H04L 63/1416 : Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 67/06 : specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 9/085 : Secret sharing or secret splitting, e.g. threshold schemes
• CPC Class Codes: H04L 9/0894 : Escrow, recovery or storing of secret information, e.g. secret key escrow or cryptographic key storage
• CPC Class Codes: H04N 21/2181 : comprising remotely distributed storage units, e.g. when movies are replicated over a plurality of video servers distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: H04N 21/23116 : involving data replication, e.g. over plural servers synchronization of replicated data G06F11/1658; error detection or correction by means of data replication G06F11/2053; replication in distributed file systems G06F16/10; replication in distributed file systems G06F16/27; replication or mirroring of data in data networks H04L67/1095
• CPC Class Codes: H04N 21/232 : Content retrieval operation locally within server, e.g. reading video streams from disk arrays storage management G06F3/0604; details of querying and searching of video data from a database G06F16/739
• CPC Class Codes: H04N 21/274 : Storing end-user multimedia data in response to end-user request , e.g. network recorder

Abstract

A method for dispersed storage network (DSN) begins by detecting a memory error associated with a memory device of a storage unit. The method continues by identifying an error descriptor code based on the detected memory error. The method continues by determining whether to perform an intermediate action based on the error descriptor code. The method continues, when not performing the intermediate action, by issuing memory status information to the DSTN managing unit. The method continues, when performing the intermediate action, by performing the intermediate action to produce an action result. The method continues by determining whether the memory device is to remain in service based on one or more of the action result and the error descriptor code and, when the memory device is not to remain in service, issuing further status information to the DSTN managing unit to indicate the failed status indicator.

Claims

1. A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN), the method comprises:

detecting, while a storage unit retaining dispersed storage error encoded data is servicing encoded data slice access messages from a DSN processing unit, a memory error associated with a memory device of the storage unit, wherein the detecting occurs while attempting to access one or more of;

a read threshold number (R) of encoded data slices to be read from one or more of the storage units for decoding of a segment of the data, a decode threshold number (D) of encoded data slices needed to reconstruct the data, or a write threshold number (W) indicating a number of encoded data slices that must be accurately stored in the one or more storage units before the encoded data segment is deemed to have been properly stored;

identifying an error descriptor code based on the detected memory error;

determining whether to perform an intermediate action based on the error descriptor code;

when not performing the intermediate action, issuing memory status information to a distributed storage and task network (DSTN) managing unit;

when performing the intermediate action, executing a lookup in an intermediate action table using the error descriptor code to identify the intermediate action and executing the identified intermediate action to produce an action result;

determining whether the memory device is to remain in service based on one or more of the action result and the error descriptor code; and

when the memory device is not to remain in service, issuing further status information to the DSTN managing unit to indicate a failed status.

2. The method of claim 1, wherein the detecting includes any of:

interpreting an error message, interpreting a test result, detecting a timing issue, detecting a data error, detecting a naming error, or detecting a data age error.

3. The method of claim 1, wherein the identifying includes one or more of:

performing a look up, interpreting a query response, interpreting system registry information, or receiving the error descriptor code.

4. The method of claim 1, wherein the determining whether to perform an intermediate action based on the error descriptor code includes performing a look up in an intermediate action table and using the error descriptor code to identify whether an intermediate action is associated with the error descriptor code.

5. The method of claim 1, wherein the memory status information includes one or more of:

an identifier of the memory device, an identifier of the storage unit, the error descriptor code, or a failed status indicator.

6. The method of claim 1, wherein the issuing includes one or more of:

generating the memory status information and sending the memory status information to at least the DSTN managing unit or changing the memory status information to indicate unavailable based on the error descriptor code.

7. The method of claim 6, wherein the changing the memory status information to unavailable includes immediately failing the memory device and quarantining from further utilization for a particular error descriptor.

8. The method of claim 1, wherein the intermediate action includes one or more of:

power cycling a memory device, resuming operations, resetting and resuming operations, or performing a test.

9. The method of claim 1, wherein the determining whether the memory device is to remain in service includes indicating to remain in service when the action result compares favorably to an expected action result based on the error descriptor code.

10. The method of claim 1, wherein the issuing further status information includes generating the memory status information to indicate the failed status.

11. A computing device of a group of computing devices of a dispersed storage network (DSN), the computing device comprises:

an interface;

a local memory; and

a processing module operably coupled to the interface and the local memory, wherein the processing module functions to;

detect, while a storage unit retaining dispersed storage error encoded data within the DSN is servicing encoded data slice access messages from a DSN processing unit, a memory error associated with the memory device of a storage unit, wherein the detecting occurs while attempting to access one or more of;

a read threshold number (R) of encoded data slices to be read from one or more of the storage units for decoding of a segment of the data, a decode threshold number (D) of encoded data slices needed to reconstruct the data, or a write threshold number (W) indicating a number of encoded data slices that must be accurately stored in the one or more storage units before the encoded data segment is deemed to have been properly stored;

identify an error descriptor code based on the detected memory error;

determine whether to perform an intermediate action based on the error descriptor code;

when not performing the intermediate action, issue memory status information to a distributed storage and task network (DSTN) managing unit;

when performing the intermediate action, executing a lookup in an intermediate action table using the error descriptor code to identify the intermediate action and executing the identified intermediate action to produce an action result;

determine whether the memory device is to remain in service based on one or more of the action result and the error descriptor code; and

when the memory device is not to remain in service, issue further status information to the DSTN managing unit to indicate a failed status.

12. The computing device of claim 11, wherein the detecting includes any of:

interpreting an error message, interpreting a test result, detecting a timing issue, detecting a data error, detecting a naming error, or detecting a data age error.

13. The computing device of claim 11, wherein the identifying includes one or more of:

performing a look up, interpreting a query response, interpreting system registry information, receiving the error descriptor code.

14. The computing device of claim 11, wherein the determining whether to perform an intermediate action based on the error descriptor code includes performing a look up in an intermediate action table using the error descriptor code to identify whether an intermediate action is associated with the error descriptor code.

15. The computing device of claim 11, wherein the status information includes one or more of:

an identifier of the memory device, an identifier of the storage unit, the error descriptor code, or a failed status.

16. The computing device of claim 11, wherein the issuing further status information includes one or more of:

generating the status information and sending the status information to at least the DSTN managing unit or changing the status information to indicate unavailable based on the error descriptor code.

17. The computing device of claim 16, wherein the changing the status information to unavailable includes immediately failing the memory device and quarantining from further utilization for a particular one of the error descriptor code.

18. The computing device of claim 11, wherein the intermediate action includes one or more of:

power cycling a memory device, resuming operations, resetting and resuming operations, or performing a test.