Error identification and handling in storage area networks

• US 9,619,311 B2
• Filed: 11/26/2013
• Issued: 04/11/2017
• Est. Priority Date: 11/26/2013
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Coronado, Juan A., Martinez, Lisa R., Olson, James E., Peterson, Beth A.
• Primary Examiner(s): Mehrmanesh, Elmira
• Application Number:
  US14/089,869
• Publication Number:
  US 20150149822A1
• Time in Patent Office: 1,232 Days
• Field of Search: 714 25, 714 26
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/34 Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
• CPC Class Codes: H04L 41/065 involving logical or physical relationship, e.g. grouping and hierarchies
• CPC Class Codes: H04L 41/0661 by reconfiguring faulty entities

Abstract

Storage area network (SAN) components contain a processor configured to provide a first event handling module that can receive an error event at a first hardware component of the storage area network. A database is accessed that contains associations between error event data and potential sources of errors. A plurality of hardware components are identified using the database and error event data. The hardware components are ranked and one is selected based on the ranking. An error notification is sent to a second event handling module of the hardware component. Based upon the response of the second event handling module, an error handling procedure is carried out.

Claims

1. A computer implemented method for handling error events relating to a storage area network, the method comprising:

receiving an error event at a first hardware component of the storage area network;

using, in response to the error event, a first event handling module operating on the first hardware component to;

access a database containing associations between error event data and potential sources of errors;

identify a plurality of hardware components based on similarities between the associations in the database and error event data contained in the error event;

generate a ranking for the plurality of hardware components;

select, based on the ranking, a particular hardware component from the plurality of hardware components;

transmit an error notification to a second event handling module of the particular hardware component of the plurality of hardware components;

monitor a response, of the second event handling module to the error notification;

carry out an error handling procedure based on the response;

perform a self test of the particular hardware component using the second event handling module

comparing a first version identifier of the database to a second version identifier in order to identify an update to the database; and

applying, in response to a mismatch between the first and second version identifiers, the update to the database.

2. The method of claim 1, further comprising:

determining that the response indicates that the particular hardware component is not the source of the error; and

transmitting, in response to determining, another error notification to another hardware component from the plurality of hardware components.

3. The method of claim 1, further comprising:

comparing a first software version identifier of the first event handling module to a second software version identifier to identify an update to the first event handling module; and

applying, in response to a mismatch between the first and second version identifiers, the update to the first event handling module.

4. The method of claim 1, wherein ranking the plurality of hardware components includes determining a probability that the hardware components are the source of the error event and ordering the hardware components based upon the probability.

5. The method of claim 1, wherein transmitting an error notification to a second event handling module of the particular hardware component of the plurality of hardware components includes transmission of at least one from a group consisting of an interrupt message and a diagnostic command.

6. The method of claim 1, further comprising using the first event handling module to select another hardware component from the plurality of hardware components by comparing a likelihood of the hardware component being the source of the error event to a threshold level.

7. A system comprising:

a first storage area network (SAN) hardware component, the first SAN hardware component being of a type selected from the group consisting of a SAN controller, a storage controller, a storage device, and a network fabric switch, the first SAN hardware component having a processor configured to provide a first event handling module that is configured to;

receive an error event related to an error at one or more hardware components of the SAN;

access a database containing associations between error event data and potential sources of errors;

identify a plurality of additional SAN hardware components of the SAN based on similarities between the associations in the database and error event data contained in the error event, each additional SAN hardware component being of a type selected from the group consisting of a SAN controller, a storage controller, a storage device, and a network fabric switch;

generate a ranking for the plurality of additional SAN hardware components;

select, based on the ranking, a particular additional SAN hardware component from the plurality of additional SAN hardware components;

transmit an error notification to a second event handling module of the particular additional SAN hardware component of the plurality of additional SAN hardware components, the error notification instructing the second event handling module to access a second database containing associations between error event data and potential sources of errors, the error notification further instructing the second event handling module to perform testing and analysis of the SAN based on the second database, and the error notification further instructing the second event handling module to respond with a response that contains the result of the performed testing and analysis;

receive the response of the second event handling module of the particular additional SAN hardware component, to the error notification; and

carry out an error handling procedure based on the response.

8. The system of claim 7, further comprising the plurality of the hardware components and wherein each hardware component is configured to provide a respective event handling module.

9. The system of claim 7, wherein the first event handling module is further configured to:

determine that the response indicates that the particular additional SAN hardware component is not the source of the error; and

transmit, in response to determining, another error notification to another SAN hardware component of the plurality of additional SAN hardware components.

10. The system of claim 7, further including a synchronization module that is configured to:

compare a first software version identifier of the first event handling module to a second software version identifier;

identify, based on the comparison, an update to the first event handling module; and

apply, in response to the identified update, the identified update to the first event handling module.

11. The system of claim 7, further including a synchronization module that is configured to:

compare a first version identifier of the database to a second version identifier of another database;

identify, based on the comparison, an update to the database; and

apply, based on the comparison and based on the another database, the update to the database.

12. The system of claim 7, wherein ranking the plurality of additional SAN hardware components includes determining a probability that the each of the plurality of the additional SAN hardware components are the source of the error event and ordering the plurality of additional SAN hardware components based upon the probability as determined.

13. A computer program product for handling events relating to a storage area network (SAN), the computer program product comprising a non-transitory computer readable storage medium having program code embodied therewith, the program code readable/executable by a computer to perform a method comprising:

receiving, by a first event handling module of a first SAN hardware component of the SAN, a request to investigate an event, the event related to a potential problem with the SAN;

analyzing, by the first event handling module and based on the received request, whether the first event handling module can directly correct the potential problem;

determining that the first event handling module is unable to directly correct the potential problem;

accessing, by the first event handling module and in response to the determination, a first database containing associations between event data and potential sources of errors;

selecting, by the first event handling module and based on the first database and based on the received request, a second event handling module of a second SAN hardware component of the SAN potentially capable of correcting the potential problem;

transmitting, by the first event handling module and to the second event handling module, a message that instructs the second event handling module to attempt to correct the potential problem;

receiving, by the first event handling module, the results of the attempt of the second event handling module; and

carry out, by the first event handling module, an error handling procedure based on the results.

14. The computer program product of claim 13, wherein the second event handling module was unsuccessful in correcting the potential problem, and wherein the method further comprises:

selecting, by the first event handling module and based on the first database and based on the received request, a third event handling module of a third SAN hardware component of the SAN potentially capable of correcting the potential problem;

transmitting, by the first event handling module and to the third event handling module, a message that instructs the third event handling module to attempt to correct the potential problem;

receiving, by the first event handling module, second results of the attempt of the third event handling module; and

carry out, by the first event handling module, a second error handling procedure based on the results and the second results.

15. The computer program product of claim 13, wherein the second event handling module was unsuccessful in correcting the potential problem, and wherein the error handling procedure is responding to the request to investigate with the determination and with the results of the attempt of the second event handling module.

16. The computer program product of claim 13, wherein the second event handling module was successful in correcting the potential problem, and wherein the error handling procedure is responding to the request to investigate with the results of the second event handling module.

17. The computer program product of claim 13, wherein the selecting is based on a ranking, and wherein the method further comprises:

identifying a plurality of SAN hardware components based on similarity between the associations in the database and error data contained in the received event; and

generating, based on the identified plurality of SAN hardware components, the ranking.

18. The computer program product of claim 13, wherein the determining is attempting to correct the problem by the first event handling module.

19. The computer program product of claim 13, wherein the selecting the second event handling module is done without user interaction, and wherein the transmitting the message to the second event handling module is done without user interaction, and wherein the receiving the results of the attempt of the second event handling module is done without human interaction.