Determining failure location in a storage system

• US 10,346,238 B2
• Filed: 09/28/2015
• Issued: 07/09/2019
• Est. Priority Date: 09/30/2014
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Lv, Man, Liu, Chris Zirui, Zou, Colin Yong
• Primary Examiner(s): Duncan, Marc
• Assistant Examiner(s): Gibson, Jonathan D
• Application Number:
  US14/867,153
• Publication Number:
  US 20160092292A1
• Time in Patent Office: 1,380 Days
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0754 by exceeding limits
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3409 for performance assessment

Abstract

Embodiments of the present disclosure provide a method, a computer program product and an apparatus for determining a failure location in a storage system by obtaining performance information of a disk; in response to the performance information indicating that at least one or more performance indices exceed a corresponding predetermined threshold, determining whether a health condition of the disk is normal based on at least one or more performance indices; and in response to determining that the health condition of the disk is normal, determining a failure location based on the health condition information of at least one or more elements in a communication path of the disk, and it may be diagnosed whether a failure occurs to the disk per se or in its communication path, and before a potential failure occurs to the disk, it may also predict the failure thereby preventing data loss.

Claims

1. A method for determining a failure location in a storage system, the method comprising:

obtaining by an obtaining unit, from a disk management system, performance information of a disk, wherein the performance information indicates an abnormal disk performance;

in response to the performance information indicating that at least one or more performance indices exceed a corresponding predetermined threshold;

retrieving, by the failure diagnostic unit, health condition information of the disk from the disk,

determining, by the failure diagnostic unit, whether a health condition of the disk is normal based on both

i) the at least one or more performance indices, and

ii) health condition information of the disk retrieved in response to the one or more performance indices exceeding the corresponding predetermined threshold,

wherein the failure diagnostic unit determines that the health condition of the disk is normal in response to detecting that both

i) the performance indices exceeding the predetermined threshold are irrelevant to the health condition of the disk, and

ii) the retrieved health condition information of the disk indicates nothing abnormal; and

in response to determining, by the failure diagnostic unit, that the health condition of the disk is normal despite the performance information indicating the abnormal disk performance, determining, by the failure diagnostic unit, a failure location of at least one or more elements in a communication path of the disk, wherein a storage system backend connection structure comprises the communication path, and wherein failing the disk unnecessarily is avoided, by the failure diagnostic unit, by the determining the failure location.

2. The method according to claim 1, further comprises:

obtaining health condition information of the at least one or more elements in the communication path of the disk; and

in response to the health condition information of at least one element in the communication path being abnormal, determining that the at least one element failed.

3. The method according to claim 1, further comprising:

determining that the disk fails, when the at least one or more performance indices associated with the health condition of the disk, and/or the health condition information of the disk is detected to be abnormal.

4. The method according to claim 1, further comprises periodically executing, manually triggering, triggering in response to detecting an I/O performance abnormality with the disk, or triggering in response to a reception of an information reporting of a disk I/O failure.

5. The method according to claim 4, wherein the disk I/O failure information is reported by a disk array RAID.

6. The method according to claim 1, wherein the at least one or more performance indices include at least one or more of the following from the group consisting of:

an I/O error count, an average I/O delay, a data error, a reallocated sector count, a SAS G-list count, and a SAS background media scan information.

7. The method according to claim 1, wherein the health condition information comprises at least one of a historical health condition information and a current health condition information.

8. The method according to claim 1, wherein the health condition information is reported using Self-Monitoring Analysis and Reporting Technology (S.M.A.R.T).

9. The method of claim 1, wherein the failure diagnostic unit determines that the health condition of the disk is normal in response to detecting that both:

i) the performance indices exceeding the performance indices comprise an I/O error count, and

ii) retrieved Self-Monitoring Analysis and Reporting Technology (S.M.A.R.T) health condition information of the disk indicates nothing abnormal.

10. An apparatus for determining a failure location in a storage system, the apparatus configured to:

obtain by an obtaining unit, from a disk management system, performance information of a disk, wherein the performance information indicates an abnormal disk performance;

in response to the performance information indicating that at least one or more performance indices exceed a corresponding predetermined threshold;

retrieve, by the failure diagnostic unit, health condition information of the disk from the disk,

determine, by the failure diagnostic unit, whether a health condition of the disk is normal based on both

i) the at least one or more performance indices, and

ii) health condition information of the disk retrieved in response to the one or more performance indices exceeding the corresponding predetermined threshold, wherein the failure diagnostic unit determines that the health condition of the disk is normal in response to detecting that both

i) the performance indices exceeding the predetermined threshold are irrelevant to the health condition of the disk, and

ii) the retrieved health condition information of the disk indicates nothing abnormal; and

in response to a determination, by the failure diagnostic unit, that the health condition of the disk is normal despite the performance information indicating the abnormal disk performance, determine, by the failure diagnostic unit, a failure location of at least one or more elements in a communication path of the disk, wherein a storage system backend connection structure comprises the communication path, and wherein failing the disk unnecessarily is avoided, by the failure diagnostic unit, by the determining the failure location.

11. The apparatus according to claim 10, further configured to:

obtain health condition information of the at least one or more elements in the communication path in sequence of the disk; and

in response to the health condition information of the at least one element In the communication path being abnormal, determine that the at least one element failed.

12. The apparatus according to claim 10, further configured to:

determine that the disk fails, when the at least one or more performance indices associated with the health condition of the disk, and/or the health condition information of the disk is detected to be abnormal.

13. The apparatus according to claim 10, further configured to periodically executed, manually trigger, trigger in response to detection of an I/O performance abnormality with the disk, or trigger in response to reception of an information reporting of a disk I/O failure.

14. The apparatus according to claim 13, wherein the disk I/O failure information is reported by a disk array RAID.

15. The apparatus according to claim 10, wherein the at least one or more performance indices include at least one or more of the following from a group consisting of:

an I/O error count, an average I/O delay, a data error, a reallocated sector count, a SAS G-list count, and a SAS background media scan information.

16. The apparatus according to claim 10, wherein the health condition information comprises at least one of a historical health condition information and a current health condition information.

17. The apparatus according to claim 10, wherein the health condition information is reported using Self-Monitoring Analysis and Reporting Technology (S.M.A.R.T).

18. A computer program product for determining a failure location in a storage system, the computer program product being tangibly stored on a non-transient computer-readable medium and comprising machine executable instructions which, when executed, cause the machine to perform steps of:

obtaining by an obtaining unit, from a disk management system, performance information of a disk, wherein the performance information indicates an abnormal disk performance;

in response to the performance information indicating that at least one or more performance indices exceed a corresponding predetermined threshold;

retrieving, by the failure diagnostic unit, health condition information of the disk from the disk,

determining, by the failure diagnostic unit, whether a health condition of the disk is normal based on both

i) the at least one or more performance indices, and

ii) health condition information of the disk retrieved in response to the one or more performance indices exceeding the corresponding predetermined threshold, wherein the failure diagnostic unit determines that the health condition of the disk is normal in response to detecting that both

i) the performance indices exceeding the predetermined threshold are irrelevant to the health condition of the disk, and

ii) the retrieved health condition information of the disk indicates nothing abnormal further comprising, wherein the health condition information comprises at least one of a historical health condition information and a current health condition information; and

in response to determining, by the failure diagnostic unit, that the health condition of the disk is normal despite the performance information indicating the abnormal disk performance, determining, by the failure diagnostic unit, a failure location of at least one or more elements in a communication path of the disk, wherein a storage system backend connection structure comprises the communication path, and wherein failing the disk unnecessarily is avoided, by the failure diagnostic unit, by the determining the failure location.

19. The computer program product according to claim 18, the steps further comprising:

determining that the disk fails, when the at least one or more performance indices associated with the health condition of the disk, and/or the health condition information of the disk is detected to be abnormal;

periodically executing, manually triggering, triggering in response to detecting an I/O performance abnormality with the disk, or triggering in response to a reception of an information reporting of a disk I/O failure, the disk I/O failure information reported by a disk array RAID.