Method for detecting problematic disk drives and disk channels in a RAID memory system based on command processing latency

• US 20090106602A1
• Filed: 10/17/2007
• Published: 04/23/2009
• Est. Priority Date: 10/17/2007
• Status: Active Grant

• Current Assignee: DataDirect Networks Incorporated
• Original Assignee: DataDirect Networks Incorporated
• Inventors: Pontos, Thomas, Piszczek, Michael, Manning, J. Gordon
• Granted Patent: US 7,917,810 B2
• Time in Patent Office: Days
• US Class Current: 714/42
• CPC Class Codes: G06F 11/008 Reliability or availability analysis
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 2211/1057 Parity-multiple bits-RAID6, i.e. RAID 6 implementations

Abstract

In order to detect problematic drives in random arrays of independent disks, the system measures the latency of executing command sets which are broadcast to all disks in the data storage system and the results are compared to identify which disks take substantially longer to complete the requests. Disks that take longer to complete requests are likely to be problematic and are candidates for further examination and replacement. The disks in each tier group are compared to determine if any disk in that group exhibits problems. Also, counters for each tier group are compared to determine if the problem is with the disk or with the channel of the tier group. The latency of each disk in the tier group is saved in a table to provide a histogram of the latency of the disks in the tier group. Histograms of the disks in a single tier group are compared to determine if a specific disk is problematic. Histograms of each tier group are compared to determine if a specific disk is problematic or all the disks on the same channel exhibit problems.

Claims

1. ) A method for detecting problematic disk storage devices in an array of independent disk storage devices, comprising the steps of:

broadcasting a command set substantially simultaneously to a plurality of independent disk storage devices under study in the array thereof, acquiring a latency count of executing said command set by each of said plurality of independent disk storage devices, and identifying a respective one of said plurality of independent disk storage devices as a problematic disk storage device if said latency count thereof exceeds a predetermined latency value.

2. ) The method of claim 1, further comprising the steps of:

broadcasting said command set to said array of independent disk storage devices repeatedly, and

monitoring said latency counts for each said independent disk storage device.

3. ) The method of claim 1, wherein said array of independent disk storage devices includes a plurality of disk channels, each disk channel corresponding to a respective one of said plurality of disk storage devices, the method further comprising the steps of:

producing a map of respective disk channels to which said command set is broadcast; and

, simultaneously broadcasting said command set through said respective disk channels.

4. ) The method of claim 1, further comprising the steps of:

indicating the issuance of said command set with an issuance time indicia,

indicating the completion of a first command in said command set with a first time indicia, and

calculating a duration of execution of said first command as the difference between said first time indicia and said issuance time indicia.

5. ) The method of claim 4, further comprising the steps of:

indicating the completion of each current command in said command set, subsequent to said first command with a completion time indicia,

calculating a duration of completion of each said command in said command set as a difference between the completion time indicia of the currently executed command and the previously executed command.

6. ) The method of claim 5, further comprising the steps of:

calculating said latency count of executing said command set by said each disk storage device by accumulating durations of completion of said first command and said each command in said command set when a final command in said command set is completed by said each disk storage device.

7. ) The method of claim 3, wherein said array of independent disk storage devices includes at least a set of first and second tier groups of the disk storage devices, wherein corresponding disk storage devices in said set of at least first and second tier groups share a common one of said plurality of disk channels, the method further comprising the steps of:

acquiring said latency count of said corresponding disk storage devices in said set of at least first and second tier groups,

identifying said common disk channel as problematic if said corresponding disk storage devices in said set of at least first and second tier groups exhibit said latency count exceeding said predetermined latency value.

8. ) The method of claim 3, further comprising the steps of:

measuring a duration of the execution of said command set at each of said plurality of disk channels.

9. ) The method of claim 1, further comprising the step of:

upon acquiring said latency count, building a latency table reflecting said acquired latency counts for said each disk storage device.

10. ) The method of claim 7, further comprising the step of:

subsequent to acquiring said latency counts, developing a latency table reflecting said acquired latency count for each of said set of at least first and second tier groups.

11. ) The method of claim 9, further comprising the step of:

grouping said acquired latency counts into predetermined time increments.

12. ) The method of claim 1, further comprising the step of:

upon identifying said problematic independent disk storage device, identifying hardware or software errors thereof.

13. ) The method of claim 12, further comprising the step of:

subsequent to identifying said problematic independent disk storage device, identifying Self-Monitoring, Analysis and Reporting Technology (SMART) related errors of said identified disk storage device.

14. ) The method of claim 13, further comprising the step of:

subsequent to identifying said problematic independent disk storage device, identifying medium error thereof.

15. ) The method of claim 14, further comprising the step of:

upon identifying said problematic disk storage device, identifying a recovered error thereof.

16. ) An array of independent disk storage devices with enhanced capability of problematic disk storage devices detection, comprising:

a plurality of independent disk storage devices distributed in at least a first tier group and a second tier group thereof, corresponding disk storage devices of said at least first and second tier group being coupled to a respective one of plurality of disk channels of said array;

a processor unit issuing a command set and broadcasting said command set to said plurality of independent disk storage devices simultaneously through said plurality of disk channels;

a counter unit coupled to said processor unit and calculating a latency count of executing said command set by each of said plurality of independent disk storage devices and a cumulative latency count of executing said command set by each of said at least first and second tier groups;

a first latency table built by said processor unit and reflecting said latency counts for said each disk storage device; and

a second latency table built by said processor unit and reflecting said cumulative latency count for each of said at least first and second tier groups;

wherein said processor unit analyzes said first latency table and identifies said each disk storage device as a problematic disk storage device if said latency count thereof exceeds a predetermined latency value, and

wherein said processor unit analyzes said second latency table and identifies said respective disk channel as a problematic one if said corresponding disk storage devices of said at least first and second tier groups exhibit said latency count exceeding said predetermined latency value.