Creating environmental snapshots of storage device failure events

• US 8,949,863 B1
• Filed: 04/30/2008
• Issued: 02/03/2015
• Est. Priority Date: 04/30/2008
• Status: Expired due to Fees

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Coatney, Doug, Gavarre, Sharon
• Primary Examiner(s): Sough, H S
• Assistant Examiner(s): Roni, Syed
• Application Number:
  US12/112,427
• Time in Patent Office: 2,470 Days
• US Class Current: 719/327
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/073 in a memory management context, e.g. virtual memory or cache management memory management G06F12/00; testing of static memory units G11C29/00
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/076 by exceeding a count or rate limit, e.g. word- or bit count limit
• CPC Class Codes: G06F 11/0766 Error or fault reporting or storing
• CPC Class Codes: G06F 11/0787 Storage of error reports, e.g. persistent data storage, storage using memory protection
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3058 Monitoring arrangements for monitoring environmental properties or parameters of the computing system or of the computing system component, e.g. monitoring of power, currents, temperature, humidity, position, vibrations thermal management in cooling arrangements of a computing system G06F1/206
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 2201/86 Event-based monitoring

Abstract

A storage device failure in a computer storage system can be analyzed by the storage system by examining relevant information about the storage device and its environment. Information about the storage device is collected in real-time and stored; this is an on-going process such that some information is continuously available. The information can include information relating to the storage device, such as input/output related information, and information relating to a storage shelf where the storage device is located, such as a status of adjacent storage devices on the shelf. All of the relevant information is analyzed to determine a reason for the storage device failure. Optionally, additional information may be collected and analyzed by the storage system to help determine the reason for the storage device failure. The analysis and supporting information can be stored in a log and/or presented to a storage system administrator to view.

Claims

1. A method for analyzing a storage device failure in a computer storage system, comprising the steps of:

continuously collecting information about the storage device including at least one of an average input/output (I/O) time of the storage device or a maximum I/O time of the storage device;

storing the collected information;

polling shelf log data from a shelf controller of a storage shelf containing the storage device when an I/O error occurs, wherein the I/O error is time-correlated with one or more errors of the shelf log data;

determining whether the storage device has failed;

analyzing the stored collected information and the shelf log data by the computer storage system to determine a reason for the storage device failure;

wherein if the computer storage system cannot determine the reason for the storage device failure based on the stored collected information, the method further comprising;

gathering additional information about the storage device failure; and

said analyzing step analyzing the stored collected information and the gathered additional information by the computer storage system to determine the reason for the storage device failure, the gathered additional information including information about the storage shelf on which the storage device is located and information about adjacent storage devices on the storage shelf.

2. The method according to claim 1, wherein the collected information further includes at least one of:

a number of errors encountered by the storage device, a number of I/O operations the storage device was performing, a number of I/O operations the storage device had errors with, Self-Monitoring, Analysis and Reporting Technology data of the storage device, or information about another component of the computer storage system.

3. The method according to claim 2, wherein the other component of the computer storage system includes at least one of:

a storage shelf on which the storage device is located, a storage device driver, or an adapter driver.

4. The method according to claim 1, wherein said storing step includes storing the collected information on the storage device.

5. The method according to claim 1, wherein the gathered additional information includes at least one of:

a number of errors encountered by the storage device, a number of I/O operations the storage device was performing, a number of I/O operations the storage device had errors with, or Self-Monitoring, Analysis and Reporting Technology data.

6. The method according to claim 1, further comprising the step of:

logging the storage device failure to a general system log file.

7. The method according to claim 1, further comprising the step of:

presenting the reason for the storage device failure and information used to determine the reason to a storage system administrator.

8. The method according to claim 1, further comprising the step of:

logging the reason for the storage device failure and information used to determine the reason to a general system log file.

9. The method according to claim 1, wherein the storage device is a disk drive.

10. The method according to claim 1, wherein the storage device is a solid state device.

11. A system for analyzing a storage device failure in a storage system, comprising:

a storage shelf including a shelf controller and a storage device; and

a storage device driver in communication with said storage device, said storage device driver configured to;

continuously collect information regarding said storage device and a storage shelf on which said storage device is located, wherein the information regarding said storage device includes at least one of an average input/output (I/O) time of the storage device or a maximum I/O time of the storage device;

poll shelf log data from the shelf controller of the storage shelf when an I/O error occurs, wherein the I/O error is time-correlated with one or more errors of the shelf log data;

analyze the collected information and the shelf log data to determine a reason for the storage device failure;

wherein if the system cannot determine the reason for the storage device failure based on the collected information, said storage device driver is configured to;

gather additional information about the storage device failure; and

analyze the collected information and the gathered additional information by the system to determine the reason for the storage device failure, the gathered additional information including information about the storage shelf on which the storage device is located and information about adjacent storage devices on the storage shelf.

12. The system according to claim 11, further comprising:

a log file entry generator in communication with said storage device driver, said log file entry generator configured to;

receive the collected information and the reason for the storage device failure; and

generate a log file entry including the collected information and the reason for the storage device failure.

13. The system according to claim 11, wherein said storage shelf includes a shelf controller in communication with said storage device driver and said collected information regarding said storage shelf includes information collected by said shelf controller regarding other storage devices on said storage shelf.

14. The system according to claim 13, further comprising:

a log file entry generator in communication with said storage device driver, said log file entry generator configured to;

receive the collected information and the reason for the storage device failure; and

generate a log file entry including the collected information and the reason for the storage device failure.

15. The system according to claim 11, wherein said storage device is a disk drive.

16. The system according to claim 11, wherein said storage device is a solid state device.

17. A non-transitory computer-readable storage medium storing a set of instructions for execution by a general purpose computer to analyze failure of a storage device in a computer storage system, the set of instructions comprising:

a collecting code segment for collecting information about the storage device and at least one other component of the computer storage system, wherein;

the information about the storage device includes an average input/output (I/O) time of the storage device or a maximum I/O time of the storage device; and

the at least one other component is not another storage device;

a storing code segment for storing the collected information;

a polling code segment for polling shelf log data from a shelf controller of a storage shelf containing the storage device when an I/O error occurs, wherein the I/O error is time-correlated with one or more errors of the shelf log data;

a determining code segment for determining whether the storage device has failed;

an analyzing code segment for analyzing the stored collected information and the shelf log data to determine a reason for the storage device failure;

a code segment for if the computer cannot determine the reason for the storage device failure based on the stored collected information;

gathering additional information about the storage device failure; and

analyzing the stored collected information and the gathered additional information by the computer to determine the reason for the storage device failure, the gathered additional information including information about the storage shelf on which the storage device is located and information about adjacent storage devices on the storage shelf.

18. The storage medium according to claim 17, the set of instructions further comprising:

a logging code segment for logging the storage device failure to a log file.

19. The storage medium according to claim 17, the set of instructions further comprising:

a logging code segment for logging the reason for the storage device failure and information used to determine the reason to a log file.

20. The storage medium according to claim 17, the set of instructions further comprising:

a presenting code segment for presenting the reason for the storage device failure and information used to determine the reason to a storage system administrator.