Systems and Methods for Detection, Isolation, and Recovery of Faults in a Fail-in-Place Storage Array

• US 20100077252A1
• Filed: 02/27/2009
• Published: 03/25/2010
• Est. Priority Date: 02/29/2008
• Status: Active Grant

• Current Assignee: Assurance Software and Hardware Solution LLC
• Original Assignee: Atrato, Inc. (Assurance Software & Hardware Solution LLC)
• Inventors: Clark, Phillip, Boehnke, Lars E., Siewert, Samuel Burk
• Granted Patent: US 8,473,779 B2
• Time in Patent Office: Days
• US Class Current: 714/6
• CPC Class Codes: G06F 11/008 Reliability or availability analysis
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence

Abstract

Systems and methods for recovering from a fault in an array of data storage devices are provided. Fault recovery includes determining that a first data storage device of the array of data storage devices is more likely to fail that other storage devices of the array of data storage devices. A second data storage device in the array of data storage devices is selected to be used in recovering from a failure of the first data storage device. Data from the first data storage device is stored at the second storage device. In the event of a failure at the first data storage device, data storage operations are performed using the second storage device.

Claims

1. A method for recovering from a fault in an array of data storage devices, comprising:

determining that a first data storage device of the array of data storage devices is more likely to fail that other storage devices of the array of data storage devices;

selecting at least a second data storage device in the array of data storage devices to be used in recovering from a failure of the first data storage device;

storing data from the first data storage device at the second storage device; and

in the event of a failure at the first data storage device, continuing data storage operations at the array of data storage devices using the second storage device.

2. The method, as claimed in claim 1, further comprising:

determining that a third data storage device of the array of data storage devices is more likely to fail than the first data storage device;

selecting at least a fourth data storage device in the array of data storage devices to be used in recovering from a failure of the third data storage device;

storing data from the third data storage device at the selected fourth data storage device; and

in the event of a failure at the third data storage device, continuing data storage operations at the array of data storage devices using the selected fourth data storage device.

3. The method, as claimed in claim 1, wherein data stored at the first data storage device is stored in a redundant manner on at least one other data storage device prior to said determining.

4. The method, as claimed in claim 1, further comprising:

providing the second data storage device as available for data storage operations prior to the first data storage device failing.

5. The method, as claimed in claim 1, further comprising:

storing data, prior to said selecting, at said second data storage device in data storage operations of the array of data storage devices.

6. The method, as claimed in claim 1, wherein in the event of a failure of a data storage device of said array of data storage devices, the failed device is not replaced.

7. The method, as claimed in claim 1, wherein said determining comprises:

collecting data from data storage devices of the array of data storage devices; and

analyzing said data to determine that said first data storage device is more likely to fail than other data storage devices of the array of data storage devices.

8. The method, as claimed in claim 7, wherein said data comprises information related to the operational conditions for the respective data storage device.

9. The method, as claimed in claim 7, wherein said data comprises at least one of power-on hours, temperature, reallocated sectors, ECC errors, and ICRC errors for the respective data storage device.

10. The method, as claimed in claim 1, wherein said selecting comprises:

determining a group of data storage devices associated with data that is stored at the first data storage device;

determining a second group of data storage devices, each of the storage devices in the second group storing data that is unrelated to the data stored at the first group; and

selecting said second group as said at least a second data storage device.

11. The method, as claimed in claim 1, wherein said second data storage device is selected from a plurality of other data storage devices according to at least one of:

a physical orientation of said second data storage device, a storage controller associated with said second data storage device, and operational conditions for said second data storage device.

13. The method, as claimed in claim 11, wherein said storing comprises:

storing said data and said digest at said array of data storage devices; and

storing a mirror of said data and said digest at said array of data storage devices.

14. The method, as claimed in claim 13, further comprising:

when an error is detected, reading said data and said digest from said mirror;

performing a digest on said read mirror data;

comparing said digest of said read mirror data to said stored mirror digest; and

determining that there is an error when said comparing indicates a difference in said digests.

12. A method for detecting errors in data stored at an array of data storage devices, comprising:

receiving data to be stored at an array of data storage devices;

performing a digest on said received data;

storing said data and said digest at said array of data storage devices;

reading said data and said digest from said array of data storage devices;

performing a digest on said read data;

comparing said digest of said read data to said stored digest; and

determining that there is an error when said comparing indicates a difference in said digests.

15. The method, as claimed in claim 12, wherein said data is stored at said array of data storage devices in LBAs, and said digest is a 128 bit digest of the data of each LBA.

16. The method, as claimed in claim 12, wherein said reading is not in response to a request of a user to access said data.

17. The method, as claimed in claim 12, wherein said reading comprises:

determining that said data has not been accessed for at least a predetermined time period; and

reading said data and said digest from said array of data storage devices when said data has not been accessed for at least the predetermined time period.

18. The method, as claimed in claim 17, wherein when it is determined that said data has not been accessed for at least the predetermined time period, said reading further comprises:

monitoring a queue of storage related requests related to the array of storage devices; and

inserting a read request for said data into said queue when fewer than a predetermined number of storage related requests are in said queue.

19. The method, as claimed in claim 12, wherein storage requests related to said array of storage devices are processed at a first processor, and said digests are processed at a second processor.