Error Processing Across Multiple Initiator Network

• US 20080320332A1
• Filed: 06/21/2007
• Published: 12/25/2008
• Est. Priority Date: 06/21/2007
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Kahler, David Ray, Jones, Michael John, Venturi, Ronald J., Leskovec, David Lawrence, Brown, Joanna Katharine, Mai Nguyen, Tram Thi, Settle, Jonathan Ian, Veen, Thomas van der, Charles Moore, Roderick Guy
• Granted Patent: US 7,779,308 B2
• Time in Patent Office: Days
• US Class Current: 714/38.14
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0766 Error or fault reporting or storing
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: H04L 41/0659 by isolating or reconfiguring faulty entities

Abstract

An architecture for error log processing is provided. Each error log is given a defined priority and mapped to an error recovery procedure (ERP) to be run if the log is seen. The system has a plurality of software layers to process the errors. Each software layer processes the error independently. Errors are reported to a higher software stack when error recovery fails from the lower stack ERPs and recovery is non-transparent. If the system host identified for error processing fails, the control of the ERP is transferred during the failover process. Non-obvious failed component isolating ERPs are grouped to be run together to assist in isolating the failed component. Prioritization of the error systems may be based on a plurality of criteria. ERPs are assigned to run within a particular software stack.

Claims

1. A computer program product comprising a computer useable medium having a computer readable program, wherein the computer readable program, when executed on a computing device, causes the computing device to:

receive an error event message indicating an error event in one of a plurality of software stacks in a network;

prioritize the error event;

assign an error recovery procedure for the error event to a software stack in the network; and

run the error recovery procedure in the software stack.

2. The computer program product of claim 1, wherein assigning the error event comprises:

identifying a host to be in control of error processing for the received error event.

3. The computer program product of claim 2, wherein the computer readable program, when executed on the computing device, further causes the computing device to:

transfer control of the error processing to another host in the network responsive to the host failing.

4. The computer program product of claim 2, wherein identifying the host to be in control of error processing for the received error event comprises:

determining in a local host whether the local host already has a lock for a current error event;

if the local host does not already have the lock for the current error event, obtaining the lock for a new error event and running the error recovery procedure for the new error event in the local host;

if the local host already has the lock for the current error event, determining whether the received error event has a higher priority than the current error event; and

if the received error event does not have a higher priority than the current error event, continuing with the error recovery procedure for the current error event.

5. The computer program product of claim 4, wherein running the error recovery procedure in the software stack further comprises:

if the received error event has a higher priority than the current error event, stopping the error recovery procedure for the current error event and running an error recovery procedure for the received error recovery procedure.

6. The computer program product of claim 1, wherein the computer readable program causes the computing device to prioritize the error event based on at least one of isolation capability, impact of the error event, logical configuration, operating environment, or workload.

7. The computer program product of claim 1, wherein the network comprises a drive software stack, a switch software stack, a controller software stack, an initiator software stack, and a system software stack.

8. The computer program product of claim 1, wherein the computer readable program, when executed on the computing device, further causes the computing device to:

report the error event to a higher software stack when error recovery fails from the error recovery procedure.

9. The computer program product of claim 1, wherein the computer readable program, when executed on the computing device, further causes the computing device to:

group non-obvious failed component isolating error recovery processes to be run together to assist in isolating the failed component.

10. A data processing system in a storage area network, comprising:

a processor; and

a memory, wherein the memory contains instructions which, when executed by the processor, cause the processor to;

receive an error event message indicating an error event in one of a plurality of software stacks in the network;

prioritize the error event;

assign an error recovery procedure for the error event to a software stack in the network; and

run the error recovery procedure in the software stack.

11. The data processing system of claim 10, wherein the storage area network comprises:

a plurality of drives running a drive software stack;

one or more switches connected to the plurality of drives, wherein the one or more switches run a switch software stack;

one or more controllers connected to the one or more switches, wherein the one or more controllers run a controller software stack;

one or more initiators connected to the one or more controllers, wherein the one or more initiators run an initiator software stack; and

one or more hosts connected to the one or more initiators, wherein the one or more hosts run a system software stack.

12. The data processing system of claim 11, wherein the data processing system is a local host within the one or more hosts and wherein the memory contains instructions which, when executed by the processor, cause the processor to:

determine whether the local host already has a lock for a current error event;

if the local host does not already have a lock for a current error event, obtain a lock for the new error event and running the error recovery procedure for the new error event in the local host;

if the local host already has a lock for a current error event, determine whether the received error event has a higher priority than the current error event; and

if the received error event has a higher priority than the current error event, stop the error recovery procedure for the current error event and running an error recovery procedure for the received error recovery procedure.

13. The data processing system of claim 10, wherein the instructions cause the processor to prioritize the error event based on at least one of isolation capability, impact of the error event, logical configuration, operating environment, or workload.

14. A method for error processing across a multiple initiator network, the method comprising:

receiving an error event message indicating an error event in one of a plurality of software stacks in the network;

prioritizing the error event;

assigning an error recovery procedure for the error event to a software stack in the network; and

running the error recovery procedure in the software stack.

15. The method of claim 14, wherein assigning the error event comprises:

identifying a host to be in control of error processing for the received error event.

16. The method of claim 15, further comprising:

responsive to the host failing, transferring control of the error processing to another host in the network.

17. The method of claim 15, wherein identifying a host to be in control of error processing for the received error event comprises:

determining in a local host whether the local host already has a lock for a current error event;

if the local host does not already have a lock for a current error event, obtaining a lock for the new error event and running the error recovery procedure for the new error event in the local host;

if the local host already has a lock for a current error event, determining whether the received error event has a higher priority than the current error event; and

if the received error event has a higher priority than the current error event, stopping the error recovery procedure for the current error event and running an error recovery procedure for the received error recovery procedure.

18. The method of claim 14, wherein prioritizing the error event comprises prioritizing the error event based on at least one of isolation capability, impact of the error event, logical configuration, operating environment, or workload.

19. The method of claim 14, further comprising:

reporting the error event to a higher software stack when error recovery fails from the error recovery procedure.

20. The method of claim 14, further comprising:

grouping non-obvious failed component isolating error recovery processes to be run together to assist in isolating the failed component.