Systems and methods for data storage remediation

• US 9,354,971 B2
• Filed: 04/23/2014
• Issued: 05/31/2016
• Est. Priority Date: 04/23/2014
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Brauer, Jonathan Charles, Niewczas, Mateusz Marek, Tal, Eran
• Primary Examiner(s): Wilson, Yolanda L
• Application Number:
  US14/260,109
• Publication Number:
  US 20150309871A1
• Time in Patent Office: 769 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/006 Identification G06F11/2289 takes precedence
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1084 Degraded mode, e.g. caused by single or multiple storage removals or disk failures
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 2211/109 Sector level checksum or ECC, i.e. sector or stripe level checksum or ECC in addition to the RAID parity calculation
• CPC Class Codes: G06F 3/06 Digital input from, or digital output to, record carriers , e.g. RAID, emulated record carriers or networked record carriers

Abstract

Systems, methods, and non-transitory computer-readable media can receive an indication that a drive included in a set of drives has experienced an error. Acquiring first data associated with the drive can be attempted. The drive can be prompted to undergo a power-cycle. Acquiring second data associated with the drive can be attempted. At least one of the first data or the second data can be analyzed to determine that the drive is remediable. At least the drive included in the set of drives can be remediated, such as by being rebuilt or reconfigured.

Claims

1. A computer-implemented method comprising:

receiving, by a computing system, an indication that a drive included in a set of drives has experienced an error;

attempting, by the computing system, to acquire first data associated with the drive;

prompting, by the computing system, the drive to undergo a power-cycle;

attempting, by the computing system, to acquire second data associated with the drive, wherein attempting to acquire the second data occurs subsequent to the power-cycle, and wherein attempting to acquire the first data occurs prior to the power-cycle;

analyzing, by the computing system, at least one of the first data or the second data to determine that the drive is remediable; and

remediating, by the computing system, at least the drive included in the set of drives.

2. The computer-implemented method of claim 1, wherein the remediating at least the drive included in the set of drives further comprises:

attempting to acquire third data associated with the drive;

analyzing the third data to verify that the drive is operational; and

performing at least one of a rebuild or a reconfiguration with respect to one or more file systems associated with at least the drive included in the set of drives.

3. The computer-implemented method of claim 1, wherein the analyzing the at least one of the first data or the second data to determine that the drive is remediable further comprises comparing the first data and the second data to determine one or more differences, the one or more differences being indicative that the drive has at least a threshold likelihood of being remediable.

4. The computer-implemented method of claim 1, wherein each of the first data and the second data includes Self-Monitoring, Analysis and Reporting Technology (SMART) data.

5. The computer-implemented method of claim 4, wherein the SMART data includes information associated with at least one of a grown defects list (GLIST), an error-correcting code (ECC) memory, or a quantity of uncorrectable errors.

6. The computer-implemented method of claim 1, wherein the set of drives corresponds to a Redundant Array of Independent Disks (RAID) system.

7. The computer-implemented method of claim 1, further comprising:

removing at least a portion of historical data associated with the drive prior to the remediating at least the drive included in the set of drives.

8. The computer-implemented method of claim 7, wherein the removing at least the portion of the historical data associated with the drive comprises at least one of clearing a cache associated with the drive, clearing configuration data associated with the drive, or removing metadata associated with the drive.

9. The computer-implemented method of claim 8, wherein the set of drives is associated with a RAID 6 system.

10. The computer-implemented method of claim 1, further comprising:

performing a scan on the drive to verify that the drive is remediable, wherein the scan is performed prior to the remediating at least the drive included in the set of drives.

11. The computer-implemented method of claim 10, wherein the set of drives is associated with a RAID 0 system.

12. The computer-implemented method of claim 10, wherein the scan is associated with a bad block scan.

13. The computer-implemented method of claim 12, wherein the bad block scan is associated with a destructive scan.

14. The computer-implemented method of claim 1, wherein the analyzing the at least one of the first data or the second data to determine that the drive is remediable further comprises:

determining an identifier associated with the drive;

acquiring information associated with the identifier; and

determining that the drive had not previously experienced the error.

15. The computer-implemented method of claim 1, further comprising:

recording, over time, at least one of the first data or the second data;

analyzing the at least one of the first data or the second data to determine a first set of properties associated with a first category of drives; and

categorizing another drive having the first set of properties as being irremediable.

16. The computer-implemented method of claim 1, further comprising:

recording, over time, at least one of the first data or the second data;

analyzing the at least one of the first data or the second data to determine a second set of properties associated with a second category of drives; and

categorizing another drive having the second set of properties as being remediable.

17. The computer-implemented method of claim 1, wherein the error is associated with at least one of a firmware fail, an Input/Output (IO) error, or a file system error.

18. The computer-implemented method of claim 1, wherein the indication is received via at least one of a RAID controller, a system health checker, or a monitoring component associated with a social networking service.

19. A system comprising:

at least one processor; and

a memory storing instructions that, when executed by the at least one processor, cause the system to perform;

receiving an indication that a drive included in a set of drives has experienced an error;

attempting to acquire first data associated with the drive;

prompting the drive to undergo a power-cycle;

attempting to acquire second data associated with the drive, wherein attempting to acquire the second data occurs subsequent to the power-cycle, and wherein attempting to acquire the first data occurs prior to the power-cycle;

analyzing at least one of the first data or the second data to determine that the drive is remediable; and

remediating at least the drive included in the set of drives.

20. A non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing system, cause the computing system to perform:

receiving an indication that a drive included in a set of drives has experienced an error;

attempting to acquire first data associated with the drive;

prompting the drive to undergo a power-cycle;

attempting to acquire second data associated with the drive, wherein attempting to acquire the second data occurs subsequent to the power-cycle, and wherein attempting to acquire the first data occurs prior to the power-cycle;

analyzing at least one of the first data or the second data to determine that the drive is remediable; and

remediating at least the drive included in the set of drives.