Method for detecting problematic disk drives and disk channels in a RAID memory system based on command processing latency

• US 7,917,810 B2
• Filed: 10/17/2007
• Issued: 03/29/2011
• Est. Priority Date: 10/17/2007
• Status: Active Grant

• Current Assignee: DataDirect Networks Incorporated
• Original Assignee: DataDirect Networks Incorporated
• Inventors: Pontos, Thomas, Manning, J. Gordon, Piszczek, Michael
• Primary Examiner(s): Lohn; Joshua A
• Application Number:
  US11/907,722
• Publication Number:
  US 20090106602A1
• Time in Patent Office: 1,259 Days
• Field of Search: 714/27, 714/33, 714/42, 714/43
• US Class Current: 714/43
• CPC Class Codes: G06F 11/008 Reliability or availability analysis
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 2211/1057 Parity-multiple bits-RAID6, i.e. RAID 6 implementations

Abstract

In order to detect problematic drives in random arrays of independent disks, the system measures the latency of executing command sets which are broadcast to all disks in the data storage system and the results are compared to identify which disks take substantially longer to complete the requests. Disks that take longer to complete requests are likely to be problematic and are candidates for further examination and replacement. The disks in each tier group are compared to determine if any disk in that group exhibits problems. Also, counters for each tier group are compared to determine if the problem is with the disk or with the channel of the tier group. The latency of each disk in the tier group is saved in a table to provide a histogram of the latency of the disks in the tier group. Histograms of the disks in a single tier group are compared to determine if a specific disk is problematic. Histograms of each tier group are compared to determine if a specific disk is problematic or all the disks on the same channel exhibit problems.

Claims

1. A method for detecting problematic disk storage devices in an array of independent disk storage devices, comprising the steps of:

broadcasting a command set of any of read or write commands substantially simultaneously to a plurality of independent disk storage devices in the array thereof,

acquiring a latency count for each of said plurality of independent disk storage devices with respect to execution of said command set executed in common by each of said plurality of independent disk storage devices, and

identifying a respective one of said plurality of independent disk storage devices as a problematic disk storage device if said latency count thereof exceeds a predetermined latency value established as a function of latency values of said plurality of independent disk storage devices.

2. The method of claim 1, further comprising the steps of:

broadcasting successive command sets to said array of independent disk storage devices, and

monitoring said latency counts for each said independent disk storage device for each successive command set.

3. The method of claim 1, wherein said array of independent disk storage devices includes a plurality of disk channels, each disk channel corresponding to a respective one of said plurality of disk storage devices, the method further comprising the steps of:

producing a map of respective disk channels to which said command set is broadcast; and

,

simultaneously broadcasting said command set through said respective disk channels.

4. The method of claim 3, further comprising the steps of:

measuring a duration of the execution of said command set at each of said plurality of disk channels.

5. The method of claim 1, further comprising the steps of:

indicating the issuance of said command set with an issuance time indicia,

indicating the completion of a first command in said command set with a first time indicia, and

calculating a duration of execution of said first command as the difference between said first time indicia and said issuance time indicia.

6. The method of claim 5, further comprising the steps of:

indicating the completion of each current command in said command set, subsequent to said first command with a completion time indicia,

calculating a duration of completion of each said command in said command set as a difference between the completion time indicia of the currently executed command and the previously executed command.

7. The method of claim 6, further comprising the steps of:

calculating said latency count of executing said command set for each said disk storage device by accumulating durations of completion of said first command and each said command in said command set when a final command in said command set is completed by each said disk storage device.

8. The method of claim 1, further comprising the step of:

upon acquiring said latency count, building a latency table reflecting said acquired latency counts for said each disk storage device.

9. The method of claim 8, further comprising the step of:

grouping said acquired latency counts into predetermined time increments.

10. The method of claim 1, further comprising the step of:

identifying said respective one of said plurality of independent disk storage devices as a problematic independent disk storage device if hardware or software errors thereof have been identified.

11. The method of claim 1, further comprising the step of:

identifying said respective one of said plurality of independent disk storage devices as a problematic independent disk storage device if Self-Monitoring, Analysis and Reporting Technology (SMART) related errors of said identified disk storage device have been identified.

12. The method of claim 1, further comprising the step of:

identifying said respective one of said plurality of independent disk storage devices as a problematic independent disk storage device if a storage medium error count above a limit value has been identified.

13. The method of claim 1, further comprising the step of:

identifying said respective one of said plurality of independent disk storage devices as a problematic disk storage device if a recovered error count above a limit value has been identified.

14. The method of claim 1, wherein said step of identifying a respective one of said plurality of independent disk storage devices as a problematic disk storage device includes the step of comparing said latency count of each of said plurality of independent disk storage devices with respect to execution of said command set to an average latency count for all of said plurality of independent disk storage devices executing said command set in common.

15. A method for detecting problematic disk storage devices in an array of independent disk storage devices, said array of independent disk storage devices including a plurality of disk channels, said array of independent disk storage devices further including at least a set of first and second tier groups of the disk storage devices, wherein corresponding disk storage devices in said set of at least first and second tier groups share a common one of said plurality of disk channels, the method further comprising the steps of:

producing a map of respective disk channels to which a command set is broadcast;

broadcasting said command set substantially simultaneously to a plurality of independent disk storage devices in the array through said respective disk channels

acquiring a latency count for said command set of said corresponding disk storage devices in said set of at least first and second tier groups,

identifying a respective one of said plurality of independent disk storage devices as a problematic disk storage device if said latency count thereof exceeds a predetermined latency value; and

identifying said common disk channel as problematic if said corresponding disk storage devices in said set of at least first and second tier groups exhibit said latency count exceeding said predetermined latency value.

16. The method of claim 15, further comprising the step of:

subsequent to acquiring said latency counts, developing a latency table reflecting said acquired latency count for each of said set of at least first and second tier groups.

17. An array of independent disk storage devices with enhanced capability of problematic disk storage devices detection, comprising:

a plurality of independent disk storage devices distributed in at least a first tier group and a second tier group thereof, corresponding disk storage devices of said at least first and second tier group being coupled to a respective one of plurality of disk channels of said array;

a processor unit issuing a command set and broadcasting said command set to said plurality of independent disk storage devices simultaneously through said plurality of disk channels;

a counter unit coupled to said processor unit and calculating a latency count of executing said command set by each of said plurality of independent disk storage devices and a cumulative latency count of executing said command set by each of said at least first and second tier groups;

a first latency table built by said processor unit and reflecting said latency counts for said each disk storage device; and

a second latency table built by said processor unit and reflecting said cumulative latency count for each of said at least first and second tier groups;

wherein said processor unit analyzes said first latency table and identifies said each disk storage device as a problematic disk storage device if said latency count thereof exceeds a predetermined latency value, and

wherein said processor unit analyzes said second latency table and identifies said respective disk channel as a problematic one if said corresponding disk storage devices of said at least first and second tier groups exhibit said latency count exceeding said predetermined latency value.

18. A method for detecting problematic disk storage devices in an array of independent disk storage devices, comprising the steps of:

broadcasting a command set in common to a plurality of independent disk storage devices in the array;

acquiring a latency count for each of said plurality of independent disk storage devices executing said command set; and

identifying a respective one of said plurality of independent disk storage devices as a problematic disk storage device responsive to said latency count thereof exceeding any one of a plurality of latency performance values, at least one of said plurality of latency performance values being a value established relative to latency values of other of said plurality of independent disk storage devices.

19. The method of claim 18, wherein said step of identifying a respective one of said plurality of independent disk storage devices as a problematic disk storage device includes the step of comparing said latency count of each of said plurality of independent disk storage devices with respect to execution of said command set to a latency performance value based on an average latency count for all of said plurality of independent disk storage devices executing said command set in common.