Error processing across multiple initiator network

• US 7,779,308 B2
• Filed: 06/21/2007
• Issued: 08/17/2010
• Est. Priority Date: 06/21/2007
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Settle, Jonathan Ian, van der Veen, Thomas, Moore, Roderick Guy Charles, Leskovec, David Lawrence, Brown, Joanna Katharine, Nguyen, Tram Thi Mai, Kahler, David Ray, Jones, Michael John, Venturi, Ronald J.
• Primary Examiner(s): Baderman; Scott T
• Assistant Examiner(s): PATEL, KAMINI B
• Application Number:
  US11/766,492
• Publication Number:
  US 20080320332A1
• Time in Patent Office: 1,153 Days
• Field of Search: 714/48
• US Class Current: 714/48
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0766 Error or fault reporting or storing
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: H04L 41/0659 by isolating or reconfiguring faulty entities

Abstract

An architecture for error log processing is provided. Each error log is given a defined priority and mapped to an error recovery procedure (ERP) to be run if the log is seen. The system has a plurality of software layers to process the errors. Each software layer processes the error independently. Errors are reported to a higher software stack when error recovery fails from the lower stack ERPs and recovery is non-transparent. If the system host identified for error processing fails, the control of the ERP is transferred during the failover process. Non-obvious failed component isolating ERPs are grouped to be run together to assist in isolating the failed component. Prioritization of the error systems may be based on a plurality of criteria. ERPs are assigned to run within a particular software stack.

Claims

1. A computer program product comprising a computer recordable storage medium having a computer readable program recorded thereon, wherein the computer readable program, when executed on a computing device, causes the computing device to:

receive an error event message indicating an error event in a first software stack within a plurality of software stacks in a network;

determine a priority of the error event;

assign an error recovery procedure for the error event to a second software stack within the plurality of software stacks in the network based on the priority of the error event; and

run the error recovery procedure in the second software stack,

wherein assigning the error recovery procedure comprises identifying a host to be in control of error processing for the received error event message; and

wherein identifying the host to be in control of error processing for the received error event message comprises;

determining in a local host whether the local host already has a lock for a current error event;

if the local host does not already have the lock for the current error event, obtaining the lock for a new error event and running the error recovery procedure for the new error event in the local host;

if the local host already has the lock for the current error event, determining whether the received error event has a higher priority than the current error event; and

if the received error event does not have a higher priority than the current error event, continuing with the error recovery procedure for the current error event.

2. The computer program product of claim 1, wherein the computer readable program, when executed on the computing device, further causes the computing device to:

transfer control of the error processing to another host in the network responsive to the host failing.

3. The computer program product of claim 1, wherein running the error recovery procedure in the second software stack further comprises:

if the received error event has a higher priority than the current error event, stopping the error recovery procedure for the current error event and running an error recovery procedure for the received error recovery procedure.

4. The computer program product of claim 1, wherein the computer readable program causes the computing device to determine a priority of the error event based on at least one of isolation capability, impact of the error event, logical configuration, operating environment, or workload.

5. The computer program product of claim 1, wherein the plurality of software stacks comprises a drive software stack, a switch software stack, a controller software stack, an initiator software stack, and a system software stack.

6. The computer program product of claim 1, wherein the computer readable program, when executed on the computing device, further causes the computing device to:

report the error event to a higher software stack when error recovery fails from the error recovery procedure.

7. The computer program product of claim 1, wherein the computer readable program, when executed on the computing device, further causes the computing device to:

group non-obvious failed component isolating error recovery processes to be run together to assist in isolating the failed component.

8. A data processing system in a storage area network, comprising:

a processor; and

a memory, wherein the memory contains instruction which, when executed by the processor, cause the processor to;

receive an error event message indicating an error event in a first software stack within a plurality of software stacks in the network;

determine a priority of the error event;

assign an error recovery procedure for the error event to a second software stack within the plurality of software stacks in the network based on the priority of the error event; and

run the error recovery procedure in the second software stack,

wherein the storage area network comprises;

a plurality of drives running a drive software stack;

one or more switches connected to the plurality of drives, wherein the one or more switches run a switch software stack;

one or more controllers connected to the one or more switches, wherein the one or more controllers run a controller software stack;

one or more initiators connected to the one or more controllers, wherein the one or more initiators run an initiator software stack; and

one or more hosts connected to the one or more initiators, wherein the one or more hosts run a system software stack; and

wherein the data processing system is a local host within the one or more hosts and wherein the memory contains instructions which, when executed by the processor, cause the processor to;

determine whether the local host already has a lock for a current error event;

if the local host does not already have a lock for a current error event, obtain a lock for the new error event and running the error recovery procedure for the new error event in the local host;

if the local host already has a lock for a current error event, determine whether the received error event has a higher priority than the current error event; and

if the received error has a higher priority than the current error event, stop the error recovery procedure for the current error event and running an error recovery procedure for the received error recovery procedure.

9. The data processing system of claim 8, wherein the instructions cause the processor to prioritize the error event based on at least one of isolation capability, impact of the error event, logical configuration, operating environment, or workload.

10. A method for error processing across a multiple initiator network, the method comprising:

receiving an error event message indicating an error event in a first software stack within a plurality of software stacks in the network;

determining a priority of the error event;

assigning an error recovery procedure for the error event to a second software stack within the plurality of software stacks in the network based on the priority of the error event; and

running the error recovery procedure in the second software stack,

wherein assigning the error recovery procedure comprises identifying a host to be in control of error processing for the received error event message; and

wherein identifying a host to be in control of error processing for the received error event comprises;

determining in a local host whether the local host already has a lock for a current error event;

if the local host does not already have a lock for a current error event, obtaining a lock for the new error event and running the error recovery procedure for the new error event in the local host;

if the local host already has a lock for a current error event, determining whether the received error event has a higher priority than the current error event; and

if the received error event has a higher priority than the current error event, stopping the error recovery procedure for the current error event and running an error recovery procedure for the received error recovery procedure.

11. The method of claim 10, further comprising:

responsive to the host failing, transferring control of the error processing to another host in the network.

12. The method of claim 10, wherein determining a priority of the error event comprises determining the priority of the error event based on at least one of isolation capability, impact of the error event, logical configuration, operating environment, or workload.

13. The method of claim 10, further comprising:

reporting the error event to a higher software stack when error recovery fails from the error recovery procedure.

14. The method of claim 10, further comprising:

grouping non-obvious failed component isolating error recovery processes to be run together to assist in isolating the failed component.