In-band problem log data collection between a host system and a storage system

• US 7,882,393 B2
• Filed: 03/28/2007
• Issued: 02/01/2011
• Est. Priority Date: 03/28/2007
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Grimes, Andrew W., Khandelwal, Nikhil
• Primary Examiner(s): McCarthy; Christopher S
• Application Number:
  US11/692,223
• Publication Number:
  US 20080244331A1
• Time in Patent Office: 1,406 Days
• Field of Search: 714/15, 714/25, 714/45
• US Class Current: 714/25
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0778 Dumping, i.e. gathering error/state information after a fault for later diagnosis
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A mechanism for in-band problem log data collection is provided. Facilities are provided for a host system, host application, or server system to instigate a state save operation in a storage system utilizing direct commands in response to an error or failure. The host system may include an application program interface (API) to force the storage device to collect a set of state save data for debug purposes at a specific time interlocked with a host system log. The API of the illustrative embodiments may be provided in a failover driver and/or host bus adapter (HBA) driver in the prime code path such that first time data capture following an error is maximized. Since the host system is instigating the state save operation with direct commands, a larger amount of transient data may be collected to provide more comprehensive state information for debugging purposes.

Claims

1. A computer program product comprising a computer useable storage medium having a computer readable program stored thereon, wherein the computer readable program, when executed on a computing device, causes the computing device to:

receive a failure notification of a failure condition of a component of a data processing system in a network comprising a plurality of data processing systems;

receive state save data from the plurality of data processing systems in the network, wherein the state save data is generated in the plurality of data processing systems in response to an in-band state save command; and

output the state save data for use in resolving the failure condition of the component,

wherein the in-band state save command is a command issued across a data channel between a first data processing system and a plurality of second data processing systems in the plurality of data processing systems, wherein the first data processing system is a storage system and the plurality of second data processing systems comprise a plurality of host systems, wherein the in-band state save command is sent from a storage controller of the storage system to the plurality of host systems in response to the failure condition being a failure of a storage device in the storage system;

wherein each of the plurality of host systems runs one or more host applications, a host bus adapter driver, and a failover driver;

wherein the in-band state save command is issued to the plurality of host systems via a multi-host system interface of the storage system which establishes a separate data channel with each of the host systems in the plurality of host systems;

wherein responsive to each of the plurality of host systems receiving the in-band state save command at the respective host bus adapter driver and providing the in-band state save command to the respective failover driver, each of the plurality of host systems collects state save data and provides the collected state save data to the storage controller via the data channels established by the multi-host system interface; and

wherein the state save data collected by each of the plurality of host systems is packaged together with state save data from the storage system into a single data package associated with the detected failure condition.

2. The computer program product of claim 1, wherein the computer readable program further causes the computing device to:

request a first state save operation in the plurality of host systems to thereby generate a plurality of host state save log data structures; and

request a second state save operation in the storage system to thereby generate a storage system state save log data structure.

3. The computer program product of claim 2, wherein the computer readable program further causes the computing device to:

package the plurality of host state save log data structures with the storage system state save log data structure into a single state save package data structure that is stored in the storage system and is associated with the failure condition, wherein outputting the state save data for use in resolving the failure condition of the component comprises outputting the state save package data structure to a third data processing system.

4. The computer program product of claim 2, wherein the state save data for the plurality of host systems and the storage system are collected substantially immediately upon detection of the failure condition in the component.

5. The computer program product of claim 2, wherein, in response to receiving the request for the first state save operation in the plurality of host systems, the plurality of host systems initiate the first state save operation by pausing an operation of each given host system, retaining a state of components of the given host system, and storing state information of the components of the given host system.

6. The computer program product of claim 5, wherein the components of the given host system include host bus adapter buffers that have short term trace values stored therein, and wherein these short term trace values are collected substantially immediately upon detection of the failure condition.

7. The computer program product of claim 1, wherein the failure condition of the component is detected by a storage controller of the storage system, and wherein the computer readable program causes the computing device to request the state save operation in the plurality of host systems by initiating a state save operation in each of the plurality of host systems, each state save operation generating a separate host state save log data structure that is packaged with the storage system state save log data structure.

8. The computer program product of claim 1, wherein the failover driver a given host system receives the in-band state save command from the first data processing system, and wherein the host bus adapter driver of the given host system initiates a state save operation in the given host system in response to receipt of the in-band state save command.

9. The computer program product of claim 1, wherein the computer readable program further causes the computing device to:

determine if a state save command has been previously received in response to a detected failure condition within a predetermined period of time of the currently detected failure condition, wherein an in-band state save command is issued, state save data is collected, and the state save data is output only if a state save command has not already been previously received within the predetermined period of time.

10. A method, in a computing device, for collecting data corresponding to a failure in a data processing system, comprising:

receiving a failure notification of a failure condition of a component of a data processing system in a network comprising a plurality of data processing systems;

receiving state save data from the plurality of data processing systems in the network, wherein the state save data is generated in the plurality of data processing systems in response to an in-band state save command; and

outputting the state save data for use in resolving the failure condition of the component,

wherein the in-band state save command is a command issued across a data channel between a first data processing system and a plurality of second data processing systems in the plurality of data processing systems, wherein the first data processing system is a storage system and the plurality of second data processing comprise a plurality of host systems, wherein the in-band state save command is sent from a storage controller of the storage system to the plurality of host systems in response to the failure condition being a failure of a storage device in the storage system;

wherein each of the plurality of host systems runs one or more host applications, a host bus adapter driver, and a failover driver;

wherein the in-band state save command is issued to the plurality of host systems via a multi-host system interface of the storage system which establishes a separate data channel with each of the host systems in the plurality of host systems;

wherein responsive to each of the plurality of host systems receiving the in-band state save command at the respective host bus adapter driver and providing the in-band state save command to the respective failover driver, each of the plurality of host systems collects state save data and provides the collected state save data to the storage controller via the data channels established by the multi-host system interface; and

wherein the state save data collected by each of the plurality of host systems is packaged together with state save data from the storage system into a single data package associated with the detected failure condition.

11. The method of claim 10, further comprising:

requesting a first state save operation in the plurality of host systems to thereby generate a plurality of host state save log data structures; and

requesting a second state save operation in the storage system to thereby generate a storage system state save log data structure.

12. The method of claim 11, wherein the state save data for the at least one host system and the storage system are collected substantially immediately upon detection of the failure condition in the component.

13. The method of claim 11, wherein, in response to receiving the request for the first state save operation in the plurality of host systems, the plurality of host systems initiates the first state save operation by pausing an operation of each given host system within the plurality of host systems, retaining a state of components of the given host system, and storing state information of the components of the given host system.

14. The method of claim 13, wherein the components of the given host system include host bus adapter buffers that have short term trace values stored therein, and wherein these short term trace values are collected substantially immediately upon detection of the failure condition.

15. The method of claim 11, further comprising:

packaging the plurality of host state save log data structures with the storage system state save log data structure into a single state save package data structure that is stored in the storage system and is associated with the failure condition, wherein outputting the state save data for use in resolving the failure condition of the component comprises outputting the state save package data structure to a third data processing system.

16. The method of claim 10, wherein the failure condition of the component is detected by a storage controller of the storage system, and wherein requesting the state save operation in the plurality of host systems comprises initiating a state save operation in each of the plurality of host systems, each state save operation generating a separate host state save log data structure that is packaged with the storage system state save log data structure.

17. The method of claim 10, wherein the failover driver of a given host system receives the in-band state save command from the first data processing system, and wherein the host bus adapter driver of the given host system initiates a state save operation in the given host system in response to receipt of the in-band state save command.

18. The method of claim 10, further comprising:

determining if a state save command has been previously received in response to a detected failure condition within a predetermined period of time of the currently detected failure condition, wherein an in-band state save command is issued, state save data is collected, and the state save data is output only if a state save command has not already been previously received within the predetermined period of time.

19. A data processing system, comprising:

a plurality of host systems; and

a storage system coupled to the plurality of host systems, wherein;

one of the plurality of host systems or the storage system receives a failure notification of a failure condition of a component of the data processing system;

the storage system receives first state save data from the plurality of host systems, and collects second state save data from one or more storage devices of the storage system, wherein at least one of the first state save data or the second state save data is generated in response to an in-band state save command;

the storage system outputs the first state save data and second state save data for use in resolving the failure condition of the component, wherein the in-band state save command is a command issued across a data channel between the storage system and the plurality of host systems, wherein the in-band state save command is sent from a storage controller of the storage system to the plurality of host systems in response to the failure condition being a failure of a storage device in the storage system;

each of the plurality of host systems runs one or more host applications, a host bus adapter driver, and a failover driver;

the in-band state save command is issued to the plurality of host systems via a multi-host system interface of the storage system which establishes a separate data channel with each of the host systems in the plurality of host systems;

responsive to each of the plurality of host systems receiving the in-band state save command at the respective host bus adapter and providing the in-band state save command to the respective failover driver, each of the plurality of host systems collects state save data and provides the collected state save data to the storage controller via the data channels established by the multi-host system interface; and

the state save data collected by each of the plurality of host systems is packaged together with state save data from the storage system into a single data package associated with the detected failure condition.

20. The system of claim 19, wherein a storage controller of the storage system:

requests a first state save operation in the plurality of host systems to thereby generate a plurality of host state save log data structures; and

requests a second state save operation in the storage system to thereby generate a storage system state save log data structure.

21. The system of claim 20, wherein the storage controller of the storage system:

packages the plurality of host state save log data structures with the storage system state save log data structure into a single state save package data structure that is stored in the storage system and is associated with the failure condition, wherein the storage system outputs the state save data for use in resolving the failure condition of the component by outputting the state save package data structure to a data processing system.

22. The system of claim 19, wherein the failover driver of a given host system receives the in-band state save command from the first data processing system, and wherein the host bus adapter driver of the given host system initiates a state save operation in the given host system in response to receipt of the in-band state save command.