Proactively resilvering a striped disk array in advance of a predicted disk drive failure

• US 10,599,343 B2
• Filed: 04/06/2018
• Issued: 03/24/2020
• Est. Priority Date: 04/06/2018
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Gross, Kenny C., Gawlick, Dieter
• Primary Examiner(s): Yi, David
• Assistant Examiner(s): Goldschmidt, Craig S
• Application Number:
  US15/947,446
• Publication Number:
  US 20190310781A1
• Time in Patent Office: 718 Days
• Field of Search: None
• CPC Class Codes: G06F 11/008 Reliability or availability analysis
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3058 Monitoring arrangements for monitoring environmental properties or parameters of the computing system or of the computing system component, e.g. monitoring of power, currents, temperature, humidity, position, vibrations thermal management in cooling arrangements of a computing system G06F1/206
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3419 by assessing time
• CPC Class Codes: G06F 3/0617 in relation to availability
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0629 Configuration or reconfiguration of storage systems
• CPC Class Codes: G06F 3/0638 Organizing or formatting or addressing of data
• CPC Class Codes: G06F 3/0653 Monitoring storage devices or systems
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD

Abstract

The disclosed embodiments provide a system that proactively resilvers a disk array when a disk drive in the array is determined to have an elevated risk of failure. The system receives time-series signals associated with the disk array during operation of the disk array. Next, the system analyzes the time-series signals to identify at-risk disk drives that have an elevated risk of failure. If one or more disk drives are identified as being at-risk, the system performs a proactive resilvering operation on the disk array using a background process while the disk array continues to operate using the at-risk disk drives.

Claims

1. A method for proactively resilvering a disk array when a disk drive in the array is determined to have an elevated risk of failure, comprising:

receiving time-series signals associated with the disk array during operation of the disk array;

analyzing the received time-series signals to identify at-risk disk drives that have an elevated risk of failure by;

using an inferential model trained on previously received time-series signals associated with the disk array to generate estimated values for the time-series signals based on correlations among the time-series signals,

performing a pairwise differencing operation between actual values and the estimated values for the time-series signals to produce residuals, and

performing a sequential probability ratio test (SPRT) on the residuals to identify one or more at-risk disk drives that have an elevated risk of failure; and

when one or more disk drives are identified as being at-risk, performing a proactive resilvering operation on the disk array using a background process while the disk array continues to operate using the at-risk disk drives.

2. The method of claim 1, further comprising estimating a remaining-useful life (RUL) for an at-risk disk drive.

3. The method of claim 2, wherein estimating the RUL for an at-risk disk drive comprises:

determining a rate at which the SPRT is generating alarms associated with the at-risk disk drive; and

applying a linear logistic regression model to the determined rate to estimate an RUL for the at-risk disk drive.

4. The method of claim 2, further comprising adjusting a priority of the background process based on estimated RULs for at-risk disk drives to ensure that the proactive resilvering operation completes before an at-risk disk drive fails.

5. The method of claim 1, wherein when the resilvering process completes and successfully produces a copy of an at-risk disk drive from redundant data stored on other disk drives in the disk array, the method further comprises replacing the at-risk disk drive with the copy of the at-risk disk drive while the disk array continues to operate.

6. The method of claim 1, wherein the received time-series signals include information specifying one or more of the following for each disk drive in the disk array and/or a server associated with the disk array:

a load;

a throughput;

a utilization factor;

a queue length;

a transaction latency;

an I/O rate;

an idle time;

read errors;

write errors;

read retries;

write retries;

temperatures;

vibrations; and

voltages.

7. The method of claim 1, wherein the disk array comprises a RAID array, which uses disk striping to distribute and redundantly store data across multiple physical disk drives in a manner that facilitates data recovery after a disk drive failure.

8. A non-transitory, computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for proactively resilvering a disk array when a disk drive in the array is determined to have an elevated risk of failure, the method comprising:

receiving time-series signals associated with the disk array during operation of the disk array;

analyzing the received time-series signals to identify at-risk disk drives that have an elevated risk of failure by;

using an inferential model trained on previously received time-series signals associated with the disk array to generate estimated values for the time-series signals based on correlations among the time-series signals,

performing a pairwise differencing operation between actual values and the estimated values for the time-series signals to produce residuals, and

performing a sequential probability ratio test (SPRT) on the residuals to identify one or more at-risk disk drives that have an elevated risk of failure; and

when one or more disk drives are identified as being at-risk, performing a proactive resilvering operation on the disk array using a background process while the disk array continues to operate using the at-risk disk drives.

9. The non-transitory, computer-readable storage medium of claim 8, wherein the method further comprises estimating a remaining-useful life (RUL) for an at-risk disk drive.

10. The non-transitory, computer-readable storage medium of claim 9, wherein estimating the RUL for an at-risk disk drive comprises:

determining a rate at which the SPRT is generating alarms associated with the at-risk disk drive; and

applying a linear logistic regression model to the determined rate to estimate an RUL for the at-risk disk drive.

11. The non-transitory, computer-readable storage medium of claim 9, wherein the method further comprises adjusting a priority of the background process based on estimated RULs for at-risk disk drives to ensure that the proactive resilvering operation completes before an at-risk disk drive fails.

12. The non-transitory, computer-readable storage medium of claim 8, wherein when the resilvering process completes and successfully produces a copy of an at-risk disk drive from redundant data stored on other disk drives in the disk array, the method further comprises replacing the at-risk disk drive with the copy of the at-risk disk drive while the disk array continues to operate.

13. The non-transitory, computer-readable storage medium of claim 8, wherein the received time-series signals include information specifying one or more of the following for each disk drive in the disk array and/or a server associated with the disk array:

a load;

a throughput;

a utilization factor;

a queue length;

a transaction latency;

an I/O rate;

an idle time;

read errors;

write errors;

read retries;

write retries;

temperatures;

vibrations; and

voltages.

14. The non-transitory, computer-readable storage medium of claim 8, wherein the disk array comprises a RAID array, which uses disk striping to distribute and redundantly store data across multiple physical disk drives in a manner that facilitates data recovery after a disk drive failure.

15. A system that proactively resilvers a disk array when a disk drive in the array is determined to have an elevated risk of failure, comprising:

at least one processor and at least one associated memory; and

a proactive resilvering mechanism that executes on the at least one processor, wherein during operation, the proactive resilvering mechanism;

receives time-series signals associated with the disk array during operation of the disk array;

analyzes the received time-series signals to identify at-risk disk drives that have an elevated risk of failure by;

using an inferential model trained on previously received time-series signals associated with the disk array to generate estimated values for the time-series signals based on correlations among the time-series signals,

performing a pairwise differencing operation between actual values and the estimated values for the time-series signals to produce residuals, and

performing a sequential probability ratio test (SPRT) on the residuals to identify one or more at-risk disk drives that have an elevated risk of failure; and

when one or more disk drives are identified as being at-risk, performs a proactive resilvering operation on the disk array using a background process while the disk array continues to operate using the at-risk disk drives.

16. The system of claim 15, wherein the proactive resilvering mechanism additionally estimates a remaining-useful life (RUL) for an at-risk disk drive.

17. The system of claim 16, wherein the proactive resilvering mechanism adjusts a priority of the background process based on estimated RULs for at-risk disk drives to ensure that the proactive resilvering operation completes before an at-risk disk drive fails.