Method and apparatus of a profiling algorithm to quickly detect faulty disks/HBA to avoid application disruptions and higher latencies

• US 10,216,591 B1
• Filed: 06/30/2016
• Issued: 02/26/2019
• Est. Priority Date: 06/30/2016
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Ravindranath, Anil, Gudipati, Krishna
• Primary Examiner(s): Iqbal, Nadeem
• Application Number:
  US15/198,216
• Time in Patent Office: 971 Days
• CPC Class Codes: G06F 11/1616 where the redundant component is an I/O device or an adapter therefor
• CPC Class Codes: G06F 11/2092 Techniques of failing over between control units
• CPC Class Codes: G06F 2201/85 Active fault masking without idle spares active fault masking without idle spare hardware where processing functionality is redundant G06F11/2035

Abstract

One embodiment is related to a method for determining a faulty hardware component within a data storage system, comprising: collecting data relating to a plurality of input/output (IO) errors associated with a first storage processor within the data storage system; compiling IO error statistics based on the data relating to the plurality of IO errors; and determining a faulty hardware component based on the IO error statistics, wherein the determining of the faulty hardware component comprises utilizing a second storage processor of the data storage system independent from the first storage processor.

Claims

1. A method for determining a faulty hardware component within a data storage system, comprising:

collecting, by a processor, data relating to a plurality of input/output (IO) errors associated with a first storage processor within the data storage system, wherein the data storage system includes a plurality of disk array enclosures (DAEs), each DAE having one or more disk drives;

compiling, by the processor, IO error statistics based on the data relating to the plurality of IO errors, the IO error statistics being related to a first one of the DAEs of the data storage system; and

determining, by the processor, a faulty hardware component based on the IO error statistics, wherein the determining of the faulty hardware component comprises utilizing a second storage processor of the data storage system independent from the first storage processor, including examining IO access statistics of the second storage processor for accessing the first DAE through a different path, and

wherein the plurality of DAEs are connected to the first storage processor and the second storage processor through an independent first path and an independent second path, and each of the one or more disk drives has a first port connected to the first storage processor through the first path and a second port connected to the second storage processor through the second path.

2. The method of claim 1, wherein the data relating to the plurality of IO errors associated with the first storage processor comprise timestamps of the plurality of IO errors and drives of the data storage system associated with each of the plurality of IO errors.

3. The method of claim 1, wherein a count of the plurality of IO errors is above a third threshold, the plurality of IO errors take place over a period of time longer than a second time period threshold, and a time difference between any two consecutive IO errors within the plurality of IO errors is below a first time period threshold.

4. The method of claim 1, further comprising:

determining drives implicated in the plurality of IO errors and a count of IO errors associated with each implicated drive, wherein each implicated drive is associated with more IO errors than a fourth threshold.

5. The method of claim 4, further comprising:

determining the drives implicated in the plurality of IO errors as faulty based on a fact that

     1) IO operations cannot be successfully performed on the drives implicated in the plurality of IO errors with the second storage processor,

     2) a count of the drives implicated in the plurality of IO errors is below a fifth threshold, or a combination of both.

6. The method of claim 4, further comprising:

determining the first storage processor or its associated hardware data path as faulty based on a fact that

     1) a count of the drives implicated in the plurality of IO errors is above a fifth threshold,

     2) the first storage processor or at least one connected entity within the hardware data path is associated with an error count above a sixth threshold, or a combination of both, wherein the error count is one of Invalid DWord Count, Running Disparity Error Count, or Loss of DWord Sync Count.

7. The method of claim 6, further comprising:

switching all future IO operations to the second storage processor in response to the first storage processor or its associated hardware data path having been determined as faulty.

8. A non-transitory machine-readable medium having instructions stored therein which, when executed by a processor, cause the processor to perform troubleshooting operations, the operations comprising:

collecting data relating to a plurality of input/output (IO) errors associated with a first storage processor within a data storage system, wherein the data storage system includes a plurality of disk array enclosures (DAEs), each DAE having one or more disk drives;

compiling IO error statistics based on the data relating to the plurality of IO errors, the IO error statistics being related to a first one of the DAEs of the data storage system; and

determining a faulty hardware component based on the IO error statistics, wherein the determining of the faulty hardware component comprises utilizing a second storage processor of the data storage system independent from the first storage processor, including examining IO access statistics of the second storage processor for accessing the first DAE through a different path, and

wherein the plurality of DAEs are connected to the first storage processor and the second storage processor through an independent first path and an independent second path, and each of the one or more disk drives has a first port connected to the first storage processor through the first path and a second port connected to the second storage processor through the second path.

9. The non-transitory machine-readable medium of claim 8, wherein the data relating to the plurality of IO errors associated with the first storage processor comprise timestamps of the plurality of IO errors and drives of the data storage system associated with each of the plurality of IO errors.

10. The non-transitory machine-readable medium of claim 8, wherein a count of the plurality of IO errors is above a third threshold, the plurality of IO errors take place over a period of time longer than a second time period threshold, and a time difference between any two consecutive IO errors within the plurality of IO errors is below a first time period threshold.

11. The non-transitory machine-readable medium of claim 8, the operations further comprising:

determining drives implicated in the plurality of IO errors and a count of IO errors associated with each implicated drive, wherein each implicated drive is associated with more IO errors than a fourth threshold.

12. The non-transitory machine-readable medium of claim 11, the operations further comprising:

determining the drives implicated in the plurality of IO errors as faulty based on a fact that

     1) IO operations cannot be successfully performed on the drives implicated in the plurality of IO errors with the second storage processor,

     2) a count of the drives implicated in the plurality of IO errors is below a fifth threshold, or a combination of both.

13. The non-transitory machine-readable medium of claim 11, the operations further comprising:

determining the first storage processor or its associated hardware data path as faulty based on a fact that

     1) a count of the drives implicated in the plurality of IO errors is above a fifth threshold,

     2) the first storage processor or at least one connected entity within the hardware data path is associated with an error count above a sixth threshold, or a combination of both, wherein the error count is one of Invalid DWord Count, Running Disparity Error Count, or Loss of DWord Sync Count.

14. The non-transitory machine-readable medium of claim 13, the operations further comprising:

switching all future IO operations to the second storage processor in response to the first storage processor or its associated hardware data path having been determined as faulty.

15. A data processing system, comprising:

a processor; and

a memory coupled to the processor storing instructions which, when executed by the processor, cause the processor to perform troubleshooting operations, the operations including

collecting data relating to a plurality of input/output (IO) errors associated with a first storage processor within the data processing system, wherein the data storage system includes a plurality of disk array enclosures (DAEs), each DAE having one or more disk drives;

compiling IO error statistics based on the data relating to the plurality of IO errors, the IO error statistics being related to a first one of the DAEs of the data storage system; and

determining a faulty hardware component based on the IO error statistics, wherein the determining of the faulty hardware component comprises utilizing a second storage processor of the data processing system independent from the first storage processor, including examining IO access statistics of the second storage processor for accessing the first DAE through a path, and

wherein the plurality of DAEs are connected to the first storage processor and the second storage processor through an independent first path and an independent second path, and each of the one or more disk drives has a first port connected to the first storage processor through the first path and a second port connected to the second storage processor through the second path.

16. The system of claim 15, wherein the data relating to the plurality of IO errors associated with the first storage processor comprise timestamps of the plurality of IO errors and drives of the data processing system associated with each of the plurality of IO errors.

17. The system of claim 15, wherein a count of the plurality of IO errors is above a third threshold, the plurality of IO errors take place over a period of time longer than a second time period threshold, and a time difference between any two consecutive IO errors within the plurality of IO errors is below a first time period threshold.

18. The system of claim 15, the operations further comprising:

determining drives implicated in the plurality of IO errors and a count of IO errors associated with each implicated drive, wherein each implicated drive is associated with more IO errors than a fourth threshold.

19. The system of claim 18, the operations further comprising:

determining the drives implicated in the plurality of IO errors as faulty based on a fact that

     1) IO operations cannot be successfully performed on the drives implicated in the plurality of IO errors with the second storage processor,

     2) a count of the drives implicated in the plurality of IO errors is below a fifth threshold, or a combination of both.

20. The system of claim 18, the operations further comprising:

determining the first storage processor or its associated hardware data path as faulty based on a fact that

     1) a count of the drives implicated in the plurality of IO errors is above a fifth threshold,

     2) the first storage processor or at least one connected entity within the hardware data path is associated with an error count above a sixth threshold, or a combination of both, wherein the error count is one of Invalid DWord Count, Running Disparity Error Count, or Loss of DWord Sync Count.

21. The system of claim 20, the operations further comprising:

switching all future IO operations to the second storage processor in response to the first storage processor or its associated hardware data path having been determined as faulty.