Recovery from transitory storage area network component failures

• US 8,051,335 B1
• Filed: 06/02/2008
• Issued: 11/01/2011
• Est. Priority Date: 06/02/2008
• Status: Active Grant

• Current Assignee: Network Appliance Incorporated (NetApp, Inc.)
• Original Assignee: Network Appliance Incorporated (NetApp, Inc.)
• Inventors: Busick, Chris, Reimers, Jan
• Primary Examiner(s): Wilson; Yolanda L
• Application Number:
  US12/131,713
• Time in Patent Office: 1,247 Days
• Field of Search: 714/43, 714/4, 714/47, 714/48, 714/26
• US Class Current: 714/43
• CPC Class Codes: G06F 11/201 between storage system components
• CPC Class Codes: G06F 11/2053 where persistent mass storage functionality or persistent mass storage control functionality is redundant error detection or correction in information storage based on relative movement between record carrier and transducer G11B20/18
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0617 in relation to availability
• CPC Class Codes: G06F 3/0635 by changing the path, e.g. traffic rerouting, path reconfiguration
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

Lun communications between a storage server and a storage subsystem for a particular lun are assigned both a current path and an alternate path. Lun communications use the current path unless the current path is determined to be faulty. Path errors may result in the storage server determining a path to be faulty. If the current path for a lun communication is determined to be faulty, then the lun communications will be sent through the alternate path so long as the alternate path is determined to be reliable. Over time, a path previously determined to be faulty may recover and be used again for lun communications.

Claims

1. A mass data storage system comprising a storage server computer, a data storage subsystem, and a plurality of communication paths connecting the storage server computer to the data storage subsystem over which communications are conducted between the storage server computer and the data storage subsystem, each communication path subject to multiple different types of errors each having a specific severity which negatively affects the communications over that communication path;

and wherein the storage server computer is operative to;

detect each type of error which occurs on each communication path;

count each detected error for each communication path on a continuous ongoing basis to establish an accumulated count value equal to the number of each type of detected error for each communication path;

decrement the accumulated count value for each type of detected error for each communication path by a predetermined amount at periodic intervals to establish a decremented accumulated count value until the decremented accumulated count value for each type of error for each communication path reaches a zero value;

attribute a weight value for each type of detected error related to the severity of the type of detected error;

calculate a weighted error value for each communication path by multiplying the weight value for each type of error by the decremented accumulated count value for each type of error and adding the results of such multiplications for each communication path;

establish a first fault threshold of weighted error values indicative of unreliable communications over each communication path;

establish a second fault threshold of weighted error values indicative of reliable communications over each communication path, the second fault threshold being less than the first fault threshold;

cease use of any one communication path for communications when the weighted error value for that one communication path exceeds or is equal to the first fault threshold; and

resume use of the one communication path when the weighted error value for that one communication path is less than or equal to the second fault threshold.

2. A mass data storage system as defined in claim 1, wherein:

the data storage subsystem has a plurality of luns for storing data;

a communication between the storage server computer and the data storage subsystem pertaining to each lun constitutes a lun communication;

a first communication path and a second communication path are associated with each lun;

the storage server computer initially uses the first communication path for lun communications to each lun; and

the storage server computer stops using the first communication path for lun communications to each lun and starts using the second communication path for lun communications to that lun when the weighted error value for the first communication path exceeds or is equal to the first fault threshold and the weighted error value for the second communication path is less than or equal to the second fault threshold.

3. A mass data storage system as defined in claim 2, wherein:

each lun communication comprises an input/output (“

I/O”

) command sent from the storage server computer to the data storage subsystem and a response to the I/O command sent from the data storage subsystem to the storage server computer; and

the storage server computer resends each I/O command through the second communication path for any lun when the storage server computer has not yet received a response to any I/O command previously sent through the first communication path associated with that lun after the storage server computer starts using the second communication path.

4. A mass data storage system as defined in claim 1, wherein:

the type of at least one of the errors detected on one communication path is different from the type of errors detected on a different communication path.

5. A mass data storage system as defined in claim 1, wherein:

at least two different weight values are established for at least two different types of detected errors on each communication path.

6. A mass data storage system as defined in claim 1, wherein:

the accumulated count values are decremented for each type of error for each communication path at the same periodic intervals.

7. A mass data storage system as defined in claim 1, wherein:

the amount by which the accumulated count value for one type of detected error is decremented is different from the amount by which the accumulated count value for a different type of error is decremented for at least one communication path.

8. A mass data storage system as defined in claim 1, wherein:

the weight value attributed to each type of detected error is predetermined.

9. A mass data storage system as defined in claim 1, wherein:

the second fault threshold is a predetermined percentage of the first fault threshold.

10. A method of determining the use of a plurality of communication paths connecting a storage server computer to a data storage subsystem for communications between the storage server computer and the data storage subsystem, each communication path subject to multiple different types of errors each having a specific severity which negatively affects the communications over that communication path, comprising:

detecting each type of error which occurs on each communication path;

attributing a weight value for each type of detected error related to the severity of the type of detected error;

counting each detected error for each communication path on a continuous ongoing basis to establish an accumulated count value equal to the number of each type of detected error for each communication path;

decrementing the accumulated count value for each type of detected error for each communication path by a predetermined amount at periodic intervals to establish a decremented accumulated count value until the decremented accumulated count value for each type of error for each communication path reaches a zero decremented accumulated count value;

calculating a weighted error value for each communication path by multiplying the weight value for each type of error by the decremented accumulated count value for each type of error and adding the results of such multiplications for each communication path;

establishing a first fault threshold of weighted error values indicative of unreliable communications over each communication path;

establishing a second fault threshold of weighted error values indicative of reliable communications over each communication path, the second fault threshold being less than the first fault threshold;

ceasing use of any one communication path for communications when the weighted error value for that one communication path equals or exceeds the first fault threshold; and

resuming use of the one communication path when the weighted error value for that one communication path is less than or equal to the second fault threshold.

11. A method as defined in claim 10, wherein the data storage subsystem has a plurality of virtual disks (“

luns”

) for the storage and retrieval of data by the storage server computer, communications between the storage server computer and the data storage subsystem related to one of the luns constituting lun communications, the method further comprising;

associating a first and a second communication path with each lun,

initially using the first communication path for lun communications to each lun; and

stopping use of the first communication path for lun communications to any lun and starting use of the second communication path for lun communications to that lun when the weighted error value of the first communication path is greater than or equal to the first fault threshold and the weighted error value of the second communication path is less than or equal to the second fault threshold.

12. A method as defined in claim 11, wherein each lun communication comprises an I/O command sent from the storage server computer to the data storage subsystem and a response to the I/O command from the data storage subsystem to the storage server computer, the method further comprising:

resending each I/O command through the second communication path for any lun when the storage server computer has not received a response to any I/O command previously sent through the first communication path associated with that lun after the storage server computer starts using the second communication path.

13. A method as defined in claim 10, further comprising:

detecting at least one error on one communication path which is different from the type of errors detected on a different communication path.

14. A method as defined in claim 10, further comprising:

attributing different weight values for at least two different types of detected errors on each communication path.

15. A method as defined in claim 10, further comprising:

decrementing the accumulated count value for each type of error for each communication path at the same periodic interval.

16. A method as defined in claim 10, further comprising:

decrementing the accumulated count values for at least two different types of errors for each communication path by different predetermined amounts.

17. A method as defined in claim 10, further comprising:

attributing predetermined weight values for each type of detected error.

18. A method as defined in claim 10, further comprising:

establishing the second fault threshold as a predetermined percentage of the first fault threshold.

19. A mass data storage system comprising a storage server computer, a data storage subsystem, and a plurality of communication paths connecting the storage server computer and the data storage subsystem over which communications are conducted between the storage server computer and the data storage subsystem, each communication path subject to multiple different types of errors each having a specific severity which negatively affects the communications over that communication path, and wherein the storage server computer is operative to:

calculate a weighted error value for each communication path by multiplying a weight value for each type of error that occurs on each communication path by a count value for each error type and adding the results of all such multiplications for each communication path;

establish first and second fault thresholds of weighted error values indicative of unreliable and reliable communications over each communication path, respectively, the second fault threshold being less than the first fault threshold;

cease use of any one communication path for communications when the weighted error value for the one communication path exceeds or is equal to the first fault threshold; and

resume use of the one communication path when the weighted error value for the one communication path is less than or equal to the second fault threshold.

20. A mass data storage system as defined in claim 19, wherein each count value relates to the number of occurrences of each type of error per communication path.

21. A mass data storage system as defined in claim 19, wherein the storage server computer is further operative to:

decrement the count values for each error type and for each communication path which are greater than zero at a periodic interval.

22. A method of determining the use of a plurality of communication paths connecting a storage server computer to a data storage subsystem for communications between the storage server computer and the data storage subsystem, each communication path subject to multiple different types of errors each having a specific severity which negatively affects the communications over that communication path, comprising:

calculating a weighted error value for each communication path by multiplying a weight value for each type of error that occurs on each communication path by a count value for each error type and adding the results of all such multiplications for each communication path;

establishing first and second fault thresholds of weighted error values indicative of unreliable and reliable communications over each communication path, respectively, the second fault threshold being less than the first fault threshold;

ceasing use of any one communication path for communications when the weighted error value for the one communication path exceeds or is equal to the first fault threshold; and

resuming use of the one communication path when the weighted error value for the one communication path is less than or equal to the second fault threshold.

23. A method as defined in claim 22, wherein each count value relates to the number of occurrences of each type of error per communication path.

24. A method as defined in claim 23, further comprising:

decrementing the count values which are greater than zero by a predetermined amount at a periodic interval.