Computer system input/output management

• US 8,635,376 B2
• Filed: 06/17/2009
• Issued: 01/21/2014
• Est. Priority Date: 02/22/2006
• Status: Active Grant

• Current Assignee: Avago Technologies International Sales Pte Limited (Broadcom, Inc.)
• Original Assignee: Emulex Design & Manufacturing Corp. (Broadcom, Inc.)
• Inventors: Barrett, David Michael, Crowell, Erick, Sebastian, Bino Joseph, Waszak, John Peter
• Primary Examiner(s): Tsai, Henry
• Assistant Examiner(s): BORROMEO, JUANITO C
• Application Number:
  US12/486,670
• Publication Number:
  US 20090259749A1
• Time in Patent Office: 1,679 Days
• Field of Search: 710/58
• US Class Current: 710/5
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 11/3495 for systems
• CPC Class Codes: G06F 2201/86 Event-based monitoring
• CPC Class Codes: G06F 2201/88 Monitoring involving counting
• CPC Class Codes: H04L 41/0604 using filtering, e.g. reduction of information by using priority, element types, position or time
• CPC Class Codes: H04L 41/0631 using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
• CPC Class Codes: H04L 43/065 related to network devices
• CPC Class Codes: H04L 43/0823 Errors, e.g. transmission errors
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/133 Protocols for remote procedure calls [RPC]
• CPC Class Codes: H04L 69/28 Timers or timing mechanisms used in protocols

Abstract

The collection of performance data at multiple servers in a SAN and forwarding that data to a centralized server for analysis is disclosed. Remote agents and a central server application collect specific interesting negative event data to enable a picture of the operational health of the SAN to be determined. The agents are placed in servers having HBAs acting as initiators. The agents interact with the HBAs through a driver stack to collect event data. Because of the initiator function they perform, HBAs have visibility to parts of the network that other entities do not have access to, and thus are ideal locations for gathering event data. A SAN diagnostics manager then pulls the collected data from each agent so that a “picture” of the SAN can be developed. In addition to collecting initiator data, the agents also collect errors and performance data from the OS of the servers.

Claims

1. A system for collecting information related to I/O performance, comprising:

a plurality of servers, each server including an input/output (I/O) controller configured as an initiator in the network, an agent configured for collecting initiator event data related to the performance of the network, and an initiator driver stack tailored for enabling communications between the agent and the I/O controller; and

a network diagnostics manager configured for periodically polling the agents and collecting the initiator event data from the plurality of servers to assist in determining root causes of network performance issues,

wherein the network diagnostics manager comprises;

a service locations protocol (SLP) interface operable to communicate with the agents to determine an agent type;

a collector operable to communicate with the SLP interface to identify one or more agents of a particular type and operable to periodically poll one or more agents to collect the initiator event data;

a database operable to store the initiator event data; and

a base application operable to retrieve the initiator event data.

2. The system of claim 1, the agents further configured for collecting different types of event data at different times in response to commands received from the network diagnostics manager.

3. The system of claim 1, the agents further configured for receiving commands from the network diagnostics manager for one or more of activating, deactivating, configuring, and upgrading the agent.

4. The system of claim 1, further comprising an agent application programming interface (API) in each server configured for enabling communications between the initiator driver stack and the agent in the server.

5. The system of claim 1, wherein the initiator event data is protocol-based negative event data.

6. The system of claim 5, wherein the protocol-based negative event data includes one or more of error messages and observational performance information received from a target.

7. The system of claim 5, wherein the protocol-based negative event data includes one or more of error messages and observational performance information received from a switch.

8. The system of claim 1, the network diagnostics manager located in a centralized management server.

9. The system of claim 1, the agents further configured for requesting servicing from the network diagnostics manager if an amount of collected initiator event data exceeds a threshold.

10. The system of claim 1, the agents further configured for communicating with the initiator driver stacks using one or more of vendor-unique, open standard, and operating system API commands.

11. The system of claim 1, the agents further configured for collecting performance information seen at the servers from operating systems of the servers.

12. The system of claim 1, the network diagnostics manager configured for integrating performance data from an operating system (OS), the initiator driver stack, the network, and storage devices to determine which errors to ignore and which errors to address.

13. The system of claim 1, the agent configured for collecting initiator event data free of a need for access rights to the network or storage components.

14. A storage area network comprising the system of claim 1 and a plurality of storage arrays.

15. The system of claim 1, the network diagnostics manager further configured for collecting and managing I/O completion times for at least some of the plurality of servers to enable trending, comparisons, and identification of servers having unacceptable latency.

16. The system of claim 15, the network diagnostics manager further configured for integrating protocol error data, capacity data and latency data to enable the selective disregarding of protocol errors that leave latency or I/O volume unaffected.

17. The system of claim 15, the network diagnostics manager further configured for utilizing latency data including information about an amount of I/Os being completed by a target and logical units (LUNs) to determine whether load balancing software is operating properly.

18. The system of claim 16, the network diagnostics manager further configured for utilizing the integrated data in a predictive manner to prevent problems before they occur.

19. The system of claim 18, wherein the integrated data includes various network paths through which a server can access a target, and wherein the network diagnostics manager is further configured for determining when one or more of the various network paths is lost from the latency data and the protocol error data.

20. The system of claim 19, the network diagnostics manager further configured for generating an alert when a number of operational paths is reduced to a particular threshold.

21. The system of claim 17, the network diagnostics manager further configured for generating an alert if a Queue Full event came back a predetermined number of times in predetermined time interval.

22. The system of claim 17, the network diagnostics manager further configured for generating an alert if the latency for a particular initiator/target port pair exceeds a predetermined average over a given time interval.

23. The system of claim 1, wherein:

each agent is further configured for collecting a maximum configured queue depth of queues in the servers associated with logical units (LUNs) being serviced by a particular array port, and a maximum array port handling capacity; and

the network diagnostics manager is further configured for computing a configured oversubscription value as the configured queue depths of the queues in the servers associated with a LUN being serviced by a particular array port divided by the maximum array port handling capacity.

24. The system of claim 1, wherein:

each agent is further configured for collecting an actual queue depth of queues in the servers associated with logical units (LUNs) being serviced by a particular array port, and a maximum array port handling capacity; and

the network diagnostics manager is further configured for computing an actual oversubscription value as the actual queue depths of the queues in the servers associated with a LUN being serviced by a particular array port divided by the maximum array port handling capacity.

25. The system of claim 1, the network diagnostics manager further configured for providing an alert when one or both of a configured oversubscription value and an actual oversubscription value exceeds a predetermined threshold.

26. The system of claim 1, the network diagnostics manager further configured for computing a system severity value indicative of a level of impact for selected collected event data.

27. A method for collecting information related to network performance from a plurality of servers in the network, comprising:

configuring an input/output (I/O) controller in each server as an initiator;

employing an agent in each server for collecting initiator event data related to the performance of the network;

tailoring an initiator driver stack in each server to enable communications between the agent and the I/O controller and collecting the initiator event data from the I/O controller; and

identify all agents of a particular type;

periodically polling the agents by a network diagnostics manager to gather the initiator event data and assist in determining root causes of network performance issues

storing the initiator event data; and

retrieving the initiator event data.

28. The method of claim 27, further comprising collecting different types of event data at different times at each agent in response to commands received from the network diagnostics manager.

29. The method of claim 27, further comprising receiving commands at each agent from the network diagnostics manager for one or more of activating, deactivating, configuring, and upgrading the agent.

30. The method of claim 27, further comprising enabling communications between the initiator driver stacks and the agents using an agent application programming interface (API).

31. The method of claim 27, wherein the initiator event data is protocol-based negative event data.

32. The method of claim 31, the protocol-based negative event data including one or more of error messages and observational performance information received from a target.

33. The method of claim 31, the protocol-based negative event data including one or more of error messages and observational performance information received from a switch.

34. The method of claim 27, further comprising collecting information related to the performance of the network free of hardware modifications in the servers.

35. The method of claim 27, further comprising centralizing the network diagnostics manager in a single management server.

36. The method of claim 27, further comprising:

at the network diagnostics manager,

communicating with the agents to determine an agent type,

identifying all agents of a particular type and periodically polling the agents to collect the initiator event data, and

storing the initiator event data in a database.

37. The method of claim 27, further comprising initiating communications with the network diagnostics manager from an agent to request servicing if an amount of initiator event data collected by the agent exceeds a threshold.

38. The method of claim 27, further comprising downloading the agents in an initially inactive state into the servers.

39. The method of claim 27, further comprising establishing communications between the agents and the initiator driver stacks using one or more of vendor-unique, open standard, or operating system application programming interface (API) commands.

40. The method of claim 27, further comprising establishing communications between the agents and an operating system of the servers for collecting performance information seen at the servers.

41. The method of claim 27, further comprising integrating performance data from an operating system (OS), the initiator driver stack, the network, and storage devices to determine which errors to ignore and which errors to address.

42. The method of claim 27, further comprising collecting initiator event data free of a need for access rights to the network or storage components.

43. The method of claim 27, further comprising collecting and managing I/O completion times for at least some of the plurality of servers to enable trending, comparisons, and identification of servers having unacceptable latency.

44. The method of claim 43, further comprising integrating protocol error data, capacity data and latency data to enable the selective disregarding of protocol errors that leave latency or I/O volume unaffected.

45. The method of claim 44, further comprising utilizing latency data including information about an amount of I/Os being completed by a target and logical units (LUNs) to determine whether load balancing software is operating properly.

46. The method of claim 44, further comprising utilizing the integrated data in a predictive manner to prevent problems before they occur.

47. The method of claim 45, wherein the integrated data includes various network paths through which a server can access a target, and wherein the method further comprises determining when one or more of the various network paths is lost from the latency data and the protocol error data.

48. The method of claim 46, further comprising generating an alert when a number of operational paths is reduced to a particular threshold.

49. The method of claim 44, further comprising generating an alert if a Queue Full event came back a predetermined number of times in predetermined time interval.

50. The method of claim 44, the network diagnostics manager further configured for generating an alert if the latency for a particular initiator/target port pair exceeds a predetermined average over a given time interval.

51. The method of claim 27, further comprising:

collecting a maximum configured queue depth of queues in the servers associated with logical units (LUNs) being serviced by a particular array port, and a maximum array port handling capacity; and

computing a configured oversubscription value as the sum of the configured queue depths of the queues in the servers associated with a LUN being serviced by a particular array port divided by the maximum array port handling capacity.

52. The method of claim 27, further comprising:

collecting an actual queue depth of queues in the servers associated with logical units (LUNs) being serviced by a particular array port, and a maximum array port handling capacity; and

computing an actual oversubscription value as the actual queue depths of the queues in the servers associated with a LUN being serviced by a particular array port divided by the maximum array port handling capacity.

53. The method of claim 27, further comprising providing an alert when one or both of a configured oversubscription value and an actual oversubscription value exceeds a predetermined threshold.

54. The method of claim 27, further comprising computing a system severity value indicative of a level of impact for selected collected event data.

55. A server in a network for collecting information related to network performance, comprising:

an input/output (I/O) controller configured as an initiator in the network;

an initiator driver stack tailored for communicating with the I/O controller; and

an agent configured for communicating with the I/O controller through the initiator driver stack to collect initiator event data related to the performance of the network from the I/O controller and transmit the collected event data to a network diagnostics manager,

wherein the network diagnostics manager comprises;

a service locations protocol (SLP) interface operable to communicate with the agent to determine an agent type;

a collector operable to identify one or more agents of a particular type and periodically poll one or more agents to collect the initiator event data;

a database for storing the initiator event data; and

a base application operable to retrieve the initiator event data.

56. The server of claim 55, the agent further configured for collecting different types of event data at different times in response to commands received from the network diagnostics manager.

57. The server of claim 55, the agent further configured for receiving commands from the network diagnostics manager to perform one or more of activating, deactivating, configuring, and upgrading the agent.

58. The server of claim 56, further comprising an agent application programming interface (API) configured for enabling communications between the initiator driver stack and the agent.

59. The server of claim 56, wherein the initiator event data is protocol-based negative event data.

60. The server of claim 59, wherein the protocol-based negative event data includes one or more of error messages and observational performance information received from a target.

61. The server of claim 59, wherein the protocol-based negative event data includes one or more of error messages and observational performance information received from a switch.

62. The server of claim 56, the agent further configured for requesting servicing from the network diagnostics manager if an amount of collected initiator event data exceeds a threshold.

63. The server of claim 58, the agent further configured for communicating with the initiator driver stack using one or more of vendor-unique, open standard, and operating system API commands.

64. The server of claim 56, the agent further configured for collecting performance information seen at the server from an operating system of the server.

65. The server of claim 56, the agent configured for collecting initiator event data free of a need for access rights to the network or storage components.

66. A storage area network comprising a plurality of the servers of claim 56 and a plurality of storage arrays.

67. A method for collecting information related to network performance from a server in the network, comprising:

configuring an input/output (I/O) controller in the server as an initiator;

employing an agent in the server for collecting initiator event data related to the performance of the network;

tailoring an initiator driver stack to enable communications between the agent and the I/O controller and collecting the initiator event data from the I/O controller; and

transmitting the collected event data to a network diagnostics manager;

storing the initiator event data in the network diagnostics manager; and

retrieving the initiator event data from the network diagnostics manager.

68. The method of claim 67, further comprising collecting different types of event data at different times in response to commands received from the network diagnostics manager.

69. The method of claim 67, further comprising receiving commands from the network diagnostics manager for one or more of activating, deactivating, configuring, and upgrading the agent.

70. The method of claim 67, further comprising enabling communications between the initiator driver stack and the agent using an agent application programming interface (API).

71. The method of claim 67, wherein the initiator event data is protocol-based negative event data.

72. The method of claim 67, further comprising collecting information related to the performance of the network free of hardware modifications in the server.

73. The method of claim 67, further comprising collecting initiator event data free of a need for access rights to the network or storage components.

74. The method of claim 67, further comprising initiating communications with the network diagnostics manager to request servicing if an amount of initiator event data collected by the agent exceeds a threshold.

75. The method of claim 67, further comprising downloading the agent in an initially inactive state into the server.

76. The method of claim 67, further comprising establishing communications between the agent and the initiator driver stack using one or more of vendor-unique, open standard, or operating system application programming interface (API) commands.

77. The method of claim 67, further comprising establishing communications between the agent and an operating system of the server for collecting performance information seen at the server.

78. A system for collecting information related to network performance, comprising:

an input/output (I/O) controller configurable as an initiator in the network; and

computer-readable storage media storing

an agent configured for collecting initiator event data related to the performance of the network and transmitting the collected event data to a network diagnostics manager, and

an initiator driver stack tailored for communicating with the agent and the I/O controller and enabling the agent to collect the initiator event data,

wherein the network diagnostics manager comprises;

a service locations protocol (SLP) interface operable to communicate with the agent to determine an agent type;

a collector operable to identify one or more agents of a particular type and periodically poll one or more agents to collect the initiator event data;

a database for storing the initiator event data; and

a base application operable to retrieve the initiator event data.

79. The system of claim 78, the agent further configured for collecting different types of event data at different times in response to commands received from the network diagnostics manager.

80. The system of claim 78, the agent further configured for receiving commands from the network diagnostics manager to perform one or more of activating, deactivating, configuring, and upgrading the agent.

81. The system of claim 78, the computer-readable storage media further storing an agent application programming interface (API) configured for enabling communications between the initiator driver stack and the agent.

82. The system of claim 78, wherein the initiator event data is protocol-based negative event data.

83. The system of claim 78, the agent further configured for collecting initiator event data free of a need for access rights to the network or storage components.

84. The system of claim 78, the agent further configured for requesting servicing from the network diagnostics manager if an amount of collected initiator event data exceeds a threshold.

85. The system of claim 81, the agent further configured for communicating with the initiator driver stack using one or more of vendor-unique, open standard, and operating system API commands.

86. The system of claim 78, the I/O controller forming a part of a server, and the agent further configured for collecting performance information seen at the server from an operating system of the server.

87. A server comprising the system of claim 78.

88. A storage area network comprising the system of claim 78 and further including one or more network switches and a plurality of storage arrays.

89. Non-transitory computer-readable storage media comprising program code for collecting information related to network performance, the program code comprising:

an agent configured for collecting initiator event data related to the performance of the network and transmitting the collected event data to a network diagnostics manager, and

an initiator driver stack tailored for communicating with the agent and the I/O controller and enabling the agent to collect the initiator event data,

wherein the network diagnostics manager comprises;

a service locations protocol (SLP) interface operable to communicate with the agent to determine an agent type;

a collector operable to identify one or more agents of a particular type and periodically poll one or more agents to collect the initiator event data;

a database for storing the initiator event data; and

a base application operable to retrieve the initiator event data.

90. The non-transitory computer-readable storage media of claim 89, the agent further configured for collecting different types of event data at different times in response to commands received from the network diagnostics manager.

91. The non-transitory computer-readable storage media of claim 89, the agent further configured for receiving commands from the network diagnostics manager to perform one or more of activating, deactivating, configuring, and upgrading the agent.

92. The non-transitory computer-readable storage media of claim 89, wherein the initiator event data is protocol-based negative event data.

93. The non-transitory computer-readable storage media of claim 89, the agent further configured for collecting initiator event data free of a need for access rights to the network or storage components.

94. The non-transitory computer-readable storage media of claim 92, wherein the protocol-based negative event data includes one or more of error messages and observational performance information received from a target.

95. Non-transitory computer-readable storage media comprising program code for collecting information related to network performance, the program code comprising:

an agent configured for communicating with an input/output (I/O) controller configured as an initiator in the network through an initiator driver stack to collect initiator event data related to the performance of the network from the I/O controller; and

a network diagnostics manager configured for polling the agent to gather the collected initiator event data and assist in determining root causes of network performance issues,

wherein the network diagnostics manager comprises;

a service locations protocol (SLP) interface operable to communicate with the agent to determine an agent type;

a collector operable to identify one or more agents of a particular type and periodically poll one or more agents to collect the initiator event data;

a database for storing the initiator event data; and

a base application operable to retrieve the initiator event data.

96. The non-transitory computer-readable storage media of claim 95, the agent further configured for collecting different types of event data at different times in response to commands received from the network diagnostics manager.

97. The non-transitory computer-readable storage media of claim 95, the agent further configured for receiving commands from the network diagnostics manager to perform one or more of activating, deactivating, configuring, and upgrading the agent.

98. The non-transitory computer-readable storage media of claim 95, wherein the initiator event data is protocol-based negative event data.

99. The non-transitory computer-readable storage media of claim 95, the agent further configured for collecting initiator event data free of a need for access rights to the network or storage components.

100. The non-transitory computer-readable storage media of claim 98, wherein the protocol-based negative event data includes one or more of error messages and observational performance information received from a target.

101. In a storage area network diagnostic system including servers, server adapters within the servers, and storage systems, the server adapters and storage systems connected in-band to allow data and command messages to flow between the servers and the storage systems, an improvement comprising:

server adapter drivers within the servers for controlling operations of the server adapters, the server adapter drivers configured to collect performance statistics and error statistics concerning the flow of data and command messages through each server adapter and provide statistics report messages upon request, the statistics report messages containing the collected performance statistics and error statistics; and

a network diagnostics manager configured for sending requests for statistics report messages to each of the server adapter drivers, and for receiving and processing the statistics report messages in order to produce an indication to a user,

wherein the network diagnostics manager comprises;

a service locations protocol (SLP) interface operable to communicate with the server adapter drivers;

a collector operable to collect the statistics report messages;

a database for storing information from the statistics report messages; and

a base application operable to retrieve the information.

102. The improvement to the storage area network diagnostic system of claim 101, wherein:

the network diagnostics manager is adapted to selectively produce an activation signal and a deactivation signal;

each server adapter driver has a quiescent mode in which no performance statistics and error statistics are collected and an active mode in which performance statistics and error statistics are collected; and

each server adapter driver automatically changes to the active mode upon receipt of the activation signal, and to the quiescent mode upon receipt of the deactivation signal.

103. The improvement to the storage area network diagnostic system of claim 101, further comprising:

storage adapter drivers within the storage systems adapted for controlling the operation of storage adapters within the storage systems, collecting performance statistics and error statistics concerning the flow of data and command messages through each storage adapter, and providing statistics report messages upon request, wherein the statistics report messages contain the collected performance statistics and error statistics.