Method for dynamically modeling medium error evolution to predict disk failure

• US 10,191,668 B1
• Filed: 06/27/2016
• Issued: 01/29/2019
• Est. Priority Date: 06/27/2016
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Traylor, Rachel, Chamness, Mark
• Primary Examiner(s): Thammavong, Prasith
• Application Number:
  US15/194,189
• Time in Patent Office: 946 Days
• CPC Class Codes: G06F 11/008 Reliability or availability analysis
• CPC Class Codes: G06F 11/30 Monitoring
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3055 Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available error or fault processing without redundancy G06F11/0703; error detection or correction by redundancy in data representation G06F11/08; error detection or correction by redundancy in operation G06F11/14; error detection or correction by redundancy in hardware G06F11/16
• CPC Class Codes: G06F 2201/88 Monitoring involving counting
• CPC Class Codes: G06F 3/0616 in relation to life time, e.g. increasing Mean Time Between Failures [MTBF]
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0653 Monitoring storage devices or systems
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks

Abstract

A method for predicting disk failures in a RAID environment is provided. A data collection center receives hard disk status information from one or more sets of hard disks in a storage system. For each of the sets of hard disks, the data collection center calculates a transitional probability that a hard disk will fail within a predetermined period of time based on the hard disk status information, and generates a first risk profile for the hard disk based on the calculated transitional probability. The data collection center then generates a second risk profile for a set of hard disks based on two or more of the first risk profiles, and compares the second risk profiles of the sets of hard disks to determine which of the sets of hard disks has a highest probability of failing within the predetermined period of time.

Claims

1. A computer-implemented method for predicting disk failures in a redundant array of independent disks (RAID) environment, the method comprising:

respectively receiving hard disk status information from each set of a plurality of sets of hard disks in a storage system, wherein the hard disk status information comprises a plurality of states within the set, a number of hard disks within the set, and an indicator of how many hard disks have failed within the set;

for each set of the plurality of sets of hard disks,

calculating a transitional probability that a hard disk will fail within a predetermined period of time based on the respective hard disk status information, wherein calculating the transitional probability comprises;

querying a medium error history of hard disks within the set,

using the medium error history to identify which of the hard disks has experienced at least one medium error,

counting a number of transitions of each of the hard disks having the at least one medium error according to different transition types, each of the transition types representing a specific transition from one of the plurality of states to another of the plurality of states, wherein the one state and the other state are of same state or different states, and

identifying a number of transition types based on the counting, and

generating a first risk profile for the hard disk based on the calculated transitional probability;

generating a second risk profile for a set of hard disks based on two or more of the first risk profiles; and

determining which of the plurality of sets of hard disks has a highest probability of failing within the predetermined period of time based on the second risk profile.

2. The method of claim 1, wherein the plurality of states within the sets of hard disks further comprises at least one of an initial state of the hard disk, a second state of the hard disk that represents, a third state of the hard disk, and an absorbing state of the hard disk, wherein the initial state of the hard disk represents a non-zero error count starting point for the transitional probability, the second state of the hard disk represents a second error count for the transitional probability that falls between the initial state and the third state, the third state of the hard disk represents a third error count for the transitional probability that falls between the second state and the absorbing state, and the absorbing state of the hard disk represents a status of FAILED for the hard disk.

3. The method of claim 1, wherein the hard disk information further comprises a non-zero error indicator, including at least one of a medium error, a remote access service error, a hardware error, a protocol error, a controller error, and a driver state error.

4. The method of claim 1, wherein the hard disk status information further comprises a manufacturer and a model of a storage system that houses the hard disk.

5. The method of claim 1, wherein the predetermined period of time further comprises at least one of:

1 day, 7 days, 30 days, and 60 days.

6. The method of claim 2, wherein calculating the transitional probability further comprises:

further querying a hard disk failure record for each set of the sets of hard disks, the hard disk failure record including a date that each hard disk within the set failed, wherein the date is used as a timestamp to classify the hard disk within one of the plurality of states; and

determining a second count number of transitions from the initial state of the hard disk to the second state of the hard disk, and a third count number of transitions from the initial state of the hard disk to at least one of the third state of the hard disk and the absorbing state of the hard disk.

7. The method of claim 6, wherein the transitional probability further comprises:

a single state transition probability that is used to determine an imminent transition from one state to another state based on the plurality of states within the sets of hard disks;

a predetermined state transition probability that is used to determine the probability of one or more state transitions in one or more predetermined number of days;

an average period of time in a specified state that is used to determine an expected number of days that the hard disk will be within the specified state; and

an expected period of time to failure of the sets of hard disks that is used to determine an expected number of days that the hard disk has to reach the absorption state.

8. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations for predicting disk failures in RAID environment, the operations comprising:

respectively receiving hard disk status information from each set of a plurality of sets of hard disks in a storage system, wherein the hard disk status information comprises a plurality of states within the set, a number of hard disks within the set, and an indicator of how many hard disks have failed within the set;

for each set of the plurality of sets of hard disks,

calculating a transitional probability that a hard disk will fail within a predetermined period of time based on the respective hard disk status information, wherein calculating the transitional probability comprises;

querying a medium error history of hard disks within the set,

using the medium error history to identify which of the hard disks has experienced at least one medium error,

counting a number of transitions of each of the hard disks having the at least one medium error according to different transition types, each of the transition types representing a specific transition from one of the plurality of states to another of the plurality of states, wherein the one state and the other state are of same state or different states, and

identifying a number of transition types based on the counting, and

generating a first risk profile for the hard disk based on the calculated transitional probability;

generating a second risk profile for a set of hard disks based on two or more of the first risk profiles; and

determining which of the sets of hard disks has a highest probability of failing within the predetermined period of time based on the second risk profile.

9. The medium of claim 8, wherein the plurality of states within the sets of hard disks further comprises at least one of an initial state of the hard disk, a second state of the hard disk that represents, a third state of the hard disk, and an absorbing state of the hard disk, wherein the initial state of the hard disk represents a non-zero error count starting point for the transitional probability, the second state of the hard disk represents a second error count for the transitional probability that falls between the initial state and the third state, the third state of the hard disk represents a third error count for the transitional probability that falls between the second state and the absorbing state, and the absorbing state of the hard disk represents a status of FAILED for the hard disk.

10. The medium of claim 8, wherein the hard disk information further comprises a non-zero error indicator, including at least one of a medium error, a remote access service error, a hardware error, a protocol error, a controller error, and a driver state error.

11. The medium of claim 8, wherein the hard disk status information further comprises a manufacturer and a model of a storage system that houses the hard disk.

12. The medium of claim 8, wherein the predetermined period of time further comprises at least one of:

1 day, 7 days, 30 days, and 60 days.

13. The medium of claim 9, wherein calculating the transitional probability further comprises:

further querying a hard disk failure record for each set of the sets of hard disks, the hard disk failure record including a date that the each hard disk within the set failed, wherein the date is used as a timestamp to classify the hard disk within one of the plurality of states; and

determining a second count number of transitions from the initial state of the hard disk to the second state of the hard disk, and a third count number of transitions from the initial state of the hard disk to at least one of the third state of the hard disk and the absorbing state of the hard disk.

14. The medium of claim 13, wherein the transitional probability further comprises:

a single state transition probability that is used to determine an imminent transition from one state to another state based on the plurality of states within the sets of hard disks;

a predetermined state transition probability that is used to determine the probability of one or more state transitions in one or more predetermined number of days;

an average period of time in a specified state that is used to determine an expected number of days that the hard disk will be within the specified state; and

an expected period of time to failure of the sets of hard disks that is used to determine an expected number of days that the hard disk has to reach the absorption state.

15. A system, comprising:

a processor; and

a memory coupled to the processor for storing instructions, which when executed from the memory, cause the processor to perform operations for predicting disk failures in RAID environment, the operations including;

respectively receiving hard disk status information from each set of a plurality of sets of hard disks in a storage system, wherein the hard disk status information comprises a plurality of states within the set, a number of hard disks within the set, and an indicator of how many hard disks have failed within the set;

for each set of the plurality of sets of hard disks,

calculating a transitional probability that a hard disk will fail within a predetermined period of time based on the respective hard disk status information, wherein calculating the transitional probability comprises;

querying a medium error history of hard disks within the set,

using the medium error history to identify which of the hard disks has experienced at least one medium error,

counting a number of transitions of each of the hard disks having the at least one medium error according to different transition types, each of the transition types representing a specific transition from one of the plurality of states to another of the plurality of states, wherein the one state and the other state are of same state or different states, and

identifying a number of transition types based on the counting, and

generating a first risk profile for the hard disk based on the calculated transitional probability;

generating a second risk profile for a set of hard disks based on two or more of the first risk profiles; and

determining which of the sets of hard disks has a highest probability of failing within the predetermined period of time based on the second risk profile.

16. The system of claim 15, wherein the plurality of states within the sets of hard disks further comprises at least one of an initial state of the hard disk, a second state of the hard disk that represents, a third state of the hard disk, and an absorbing state of the hard disk, wherein the initial state of the hard disk represents a non-zero error count starting point for the transitional probability, the second state of the hard disk represents a second error count for the transitional probability that falls between the initial state and the third state, the third state of the hard disk represents a third error count for the transitional probability that falls between the second state and the absorbing state, and the absorbing state of the hard disk represents a status of FAILED for the hard disk.

17. The system of claim 15, wherein the hard disk information comprises a non-zero error indicator, including at least one of a medium error, a remote access service error, a hardware error, a protocol error, a controller error, and a driver state error, and wherein the hard disk status information further comprises a manufacturer and a model of a storage system that houses the hard disk.

18. The system of claim 15, wherein the predetermined period of time further comprises at least one of:

1 day, 7 days, 30 days, and 60 days.

19. The system of claim 16, wherein calculating the transitional probability further comprises:

further querying a hard disk failure record for each set of the sets of hard disks, the hard disk failure record including a date that each hard disk failed, wherein the date is used as a timestamp to classify the hard disk within one of the plurality of states; and

determining a second count number of transitions from the initial state of the hard disk to the second state of the hard disk, and a third count number of transitions from the initial state of the hard disk to at least one of the third state of the hard disk and the absorbing state of the hard disk.

20. The system of claim 19, wherein the transitional probability further comprises:

a single state transition probability that is used to determine an imminent transition from one state to another state based on the plurality of states within the sets of hard disks;

a predetermined state transition probability that is used to determine the probability of one or more state transitions in one or more predetermined number of days;

an average period of time in a specified state that is used to determine an expected number of days that the hard disk will be within the specified state; and

an expected period of time to failure of the sets of hard disks that is used to determine an expected number of days that the hard disk has to reach the absorption state.