Mechanism for correcting errors beyond the fault tolerant level of a raid array in a storage system

• US 8,417,987 B1
• Filed: 12/01/2009
• Issued: 04/09/2013
• Est. Priority Date: 12/01/2009
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Goel, Atul, Sankar, Sunitha
• Primary Examiner(s): LE, DIEU MINH T
• Application Number:
  US12/629,032
• Time in Patent Office: 1,225 Days
• Field of Search: 714/2, 714/6.1, 714/6.21, 714/6.22, 714/6.23, 714/6.24, 714/6.3, 714/42, 714/48, 714/54
• US Class Current: 714/6.1
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems

Abstract

Embodiments of the present invention provide novel, reliable and efficient technique for tracking, tolerating and correcting unrecoverable errors (i.e., errors that cannot be recovered by the existing RAID protection schemes) in a RAID array by reducing the need to perform drastic recovery actions, such as a file system consistency check, which typically disrupts client access to the storage system. Advantageously, ability to tolerate and correct errors in the RAID array beyond the fault tolerance level of the underlying RAID technique increases resiliency and availability of the storage system.

Claims

1. A method for correcting unrecoverable errors on storage devices connected to a storage system, the method comprising:

identifying, by the storage system during a data access request to the storage devices, a data block having an unrecoverable error, the unrecoverable error cannot be corrected by an underlying RAID protection technique at the storage system;

providing an indicator within a close proximity of the data block having the unrecoverable error that the data block is invalid, the indicator stored at a location so that when an I/O is issued to the data block having the unrecoverable error, the indicator is read as part of the same I/O;

protecting the indicator by the underlying RAID protection technique; and

recovering the data block having the unrecoverable error without initiating a consistency check operation in the storage system by obtaining a good copy of the data block having the unrecoverable error asynchronously to the data access request to the storage devices during which the data block having the unrecoverable error was identified.

2. The method of claim 1, wherein obtaining a good copy of the data block having the unrecoverable error asynchronously to the data access request further comprises obtaining the good copy of the data block after the data access request to the storage devices during which the unrecoverable error was identified.

3. The method of claim 1, wherein the good copy of the data block having the unrecoverable error is provided at a time after the data access request to the storage devices during which the unrecoverable errors was identified.

4. The method of claim 1, wherein obtaining a good copy of the data blocks having the unrecoverable error further comprises obtaining the good copy stored in a memory at the storage system.

5. The method of claim 1, wherein obtaining a good copy of the data block having the unrecoverable error further comprises obtaining the good copy of the data block remotely at a destination storage system storing mirror copies of the data blocks residing at the storage system, the destination system connected to the storage system over a network.

6. The method of claim 1, wherein recovering the data block having the unrecoverable error without initiating a consistency check operation in the storage system further comprises continuing servicing client data access requests by the storage system.

7. The method of claim 1, wherein recovering the data block having the unrecoverable errors further comprises writing the good copy of the data block in place of the data block having the unrecoverable error.

8. The method of claim 1, wherein recovering the data block having the unrecoverable error further comprises writing the good copy of the data block to a different location other than the location where the data block having the unrecoverable error is located and further comprising marking the data block as free at the storage system.

9. The method of claim 1, wherein providing an indicator within close proximity of the data block having the unrecoverable error further comprises setting a pseudo-bad indicator in a checksum area adjacent to the data block.

10. The method of claim 1, further comprising removing the indicator that the data block is invalid once the good copy of the data block is obtained.

11. The method of claim 1, the good copy comprising a mirrored version of the data block.

12. The method of claim 1, further comprising obtaining the good copy of the data block having the unrecoverable error synchronously to the data access request.

13. The method of claim 1, wherein obtaining the good copy of the data block having the unrecoverable error further comprises obtaining a snapshot copy of the data block.

14. The method of claim 1, wherein protecting the indicator by the underlying RAID protection technique further comprises computing a parity for the indicator.

15. The method of claim 1, wherein protecting the indicator by the underlying RAID protection technique further comprises mirroring the indicator and storing the indicator within a close proximity of a mirrored copy of the data block.

16. The method of claim 1, further comprising determining if a copy of the data block having an unrecoverable error is a good copy by:

comparing a file system write signature stored within the copy of the data block and a received file system write signature from a file system upon reading the copy of the data block;

if a match between the file system write signature and the received file system write signature is detected, using the copy of the data block to correct the unrecoverable error; and

if a match between the file system write signature and the received file system write signature is not detected, not using the copy of the data block to correct the unrecoverable error.

17. The method of claim 1, further comprising:

performing a checksum value verification to determine if a checksum value computed upon reading the good copy of the data block and a checksum value that has previously been computed for the data block match; and

responsive to the two values not matching, determining that the good copy of the data block is bad and cannot be used to perform error recovery.

18. The method of claim 14, wherein computing the parity for the indicator further comprises:

performing an exclusive OR (XOR) operation on the indicator to create a computed value; and

writing the computed value within a close proximity of the data block that stores parity for data blocks in a stripe in the storage system.

19. The method of claim 1, further comprising generating a new value for each data block having an unrecoverable error by:

responsive to an error being a media error or a missing block error, writing an identifiable pattern to the data blocks having an indicator set and updating data integrity information to indicate that the data integrity information cannot be used to verify data in the data block;

responsive to an error being a checksum error, filling the data block with original data read from a storage device where the data block resides and updating data integrity information to indicate that the data integrity information cannot be used to verify data in the data block; and

responsive to an error being a lost write error, filling the data block with original data.

20. The method of claim 1, wherein the good copy of the data block includes data identical to original data that was written to the data block prior to encountering the unrecoverable error.

21. The method of claim 1, wherein the good copy of the data block is not marked with an indicator indicating that the data block is invalid.

22. A method for correcting unrecoverable errors in a storage array connected to a storage system for storing data, the method comprising:

performing a data access request to the storage array;

identifying, by a storage module, a data block having an error;

initiating, by the storage module, a recovery operation to recover the data block that has the error;

determining, by the storage module, that the storage array has an unrecoverable error, wherein unrecoverable errors are errors that cannot be corrected by an underlying RAID protection technique of the storage module;

obtaining a good copy of the data block having the unrecoverable error asynchronously to the data access request to the storage array during which the data block having the unrecoverable error was identified, the obtaining comprising;

comparing a file system write signature stored within a copy of the data block and a received file system write signature from a file system upon reading the copy of the data block;

if a match is detected, then using the copy of the data block as the good copy to correct the unrecoverable error; and

if a match is not detected, then not using the copy of the data block as the good copy to correct the unrecoverable error; and

writing the good copy of the data block to correct the error in the storage array.

23. The method of claim 22, wherein the good copy of the data block includes data identical to original data that was written to the data block prior to encountering the unrecoverable error.

24. The method of claim 22, wherein obtaining the good copy of the data block having the unrecoverable error asynchronously to the data access request to the storage array further comprises obtaining the good copy of the data block after the data access request to the storage array, during which the data block having the unrecoverable error was identified.

25. A method for correcting unrecoverable errors in a storage array connected to a storage system for storing data, the method comprising:

identifying, by a RAID system during a data access request, a data block having an unrecoverable error that cannot be corrected by an underlying RAID technique at the storage system;

providing an indicator within close proximity of the data block having the unrecoverable error that the data block has an error;

protecting the indicator by the underlying RAID technique, thereby reliably maintaining information about data blocks having unrecoverable errors;

obtaining a good copy of the data block having the unrecoverable error; and

generating a new value for the data block, comprising;

if the error is a media error or a missing block error, then writing an identifiable pattern to the data block and updating data integrity information to indicate that the data integrity information cannot be used to verify data in the data block;

if the error is a checksum error, then filling the data block with original data read from a storage device where the data block resides and updating data integrity information to indicate that the data integrity information cannot be used to verify data in the data block; and

if the error is a lost write error, then filling the data block with original data.

26. The method of claim 25, wherein protecting the indicator by the underlying RAID technique further comprises computing a parity value for the indicator.

27. The method of claim 25, wherein protecting the indicator by the underlying RAID technique further comprises mirroring the indicator.

28. A storage system for correcting unrecoverable errors in a storage array, the system comprising:

a storage module that maintains a data protection mechanism, the storage module configured to identify during a data access request, a data block having an unrecoverable error condition that cannot be corrected by the data protection mechanism;

a pseudo-bad block management module configured to set an indicator within a close proximity to the data block having the unrecoverable error condition and to protect the indicator by the data protection mechanism;

a module for opportunistic error recovery configured to perform error recovery to identify a good copy of the data block having the unrecoverable error asynchronously to the data access request during which the unrecoverable error was encountered, and to recover the data block having the unrecoverable error;

a first mirroring module executed at the storage system, the first mirroring module configured to send copies of data blocks to a destination storage system connected to the storage system via a network; and

a second mirroring module executed at the destination storage system configured to store copies of the data maintained at the storage system, the second mirroring module configured to provide a copy of the data blocks upon request by the first mirroring module for opportunistic error recovery, at least some of at least one of the storage module, the pseudo-bad block management module, the module for opportunistic error recovery, the first mirroring module, and the second mirroring module implemented at least in part via a processor of the storage system.

29. The system of claim 28, further comprising a data structure for storing mappings between data blocks at the storage system and copies of the data blocks maintained at the destination storage system connected to the storage system over a network.

30. The system of claim 28, further comprising a buffer cache memory for storing copies of the data blocks locally at the storage system.

31. The system of claim 28, the data protection mechanism comprising an underlying RAID technique.