PERFORMANCE METRIC COLLECTION AND AUTOMATED ANALYSIS

• US 20080016412A1
• Filed: 08/01/2007
• Published: 01/17/2008
• Est. Priority Date: 07/01/2002
• Status: Active Grant

• Current Assignee: Riverbed Technology, LLC (Riverbed Technology Incorporated)
• Original Assignee: OPNET Technologies Incorporated (Riverbed Technology Incorporated)
• Inventors: WHITE, David, BARON, Steven, LaBATT, Earl, McGEE, John, MACOMBER, Edward
• Granted Patent: US 7,444,263 B2
• Time in Patent Office: Days
• US Class Current: 714/48
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 2201/81 Threshold
• CPC Class Codes: G06F 2201/86 Event-based monitoring
• CPC Class Codes: H04L 41/0631 using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
• CPC Class Codes: H04L 43/08 Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 43/16 Threshold monitoring

Abstract

A metric monitoring and analysis system including dynamic sampling agents located in monitored system elements and a service management platform. Each sampling agent includes a data adapter collecting metric data in a common format, a threshold generator for determining dynamic metric threshold ranges, an alarm detector generating an indicator when a metric deviates outside a dynamic threshold range or a static threshold, and a deviation tracker generating an alarm severity scores. The service platform includes an alarm analyzer identifying root causes of system alarm conditions by correlation of grouped metrics or forensic analysis of temporally or statistically correlated secondary forensic data or data items from a service model of the system.

Claims

1. A method for performing system metric analysis, comprising the steps of:

collecting metric data representing system operations from a plurality of system sources,

determining a dynamic metric threshold range for each metric over successive time periods,

indicating a metric alarm event by

generating a metric threshold alarm indicator when a corresponding metric value deviates outside the dynamic threshold range,

generating an alarm severity score for each metric alarm event, and

performing a root cause analysis identifying a basic cause of a system alarm condition by at least one of

correlation of grouped metrics with alarm conditions, and

forensic analysis of selected secondary forensic data items recorded upon the occurrence of at least selected alarm conditions.

2. The method for performing system metric analysis of claim 1, wherein the step of collecting metric data from system sources further includes the steps of:

collecting the metric data from each source in a source format, and

transforming the metric data from the corresponding source format into at least one common format.

3. The method for performing system metric analysis of claim 1, wherein the step of determining a dynamic metric threshold range for each metric further includes the steps of:

generating an empirical estimate of an actual distribution of values of the metric,

determining a trend in the actual distribution of the values of the metric,

determining an expected range of the values of the metric over the time periods, and

determining a dynamic threshold range for each metric based on the expected range of values of the metric.

4. The method for performing system metric analysis of claim 1, wherein the step of indicating a metric alarm event further includes the steps of:

generating a metric threshold alarm indicator when a corresponding metric value deviates outside a static threshold range.

5. The method for performing system metric analysis of claim 1, wherein the step of generating an alarm severity score for each metric alarm further includes the step of:

determining the severity score as a ratio of a time and magnitude of the metric value deviation outside the threshold range to a width of the threshold range.

6. The method for performing system metric analysis of claim 1, wherein the step of correlation and grouping of metrics with alarm indicators further includes at least one of the steps of:

performing a temporal alarm correlation process by identifying metrics having values resulting in alarm conditions that are correlated in time, and

performing a statistical metric correlation process by correlating pairs of metrics having values resulting in an alarm condition according to the relative ranks of the values of the metrics.

7. The method for performing system metric analysis of claim 1, wherein the step of analysis of recorded secondary forensic data items and alarm indicators further includes at least one of the steps of:

upon occurrence of each of at least one of the selected alarm indicators,

recording selected key metrics defined as bounding the data items with at least one of temporal and statistical correlation of the metrics and

recording data items selected from a service model of the system representing system operations associated with an alarm condition.

8. The method for performing system metric analysis of claim 1, wherein the secondary forensic data items comprise non-metric information pertaining to system conditions at occurrence of an alarm condition.

9. The method for performing system metric analysis of claim 1, wherein:

the steps of

collecting metric data representing system operations from a plurality of system sources,

determining a dynamic metric threshold range for each metric over successive time periods,

indicating a metric alarm event by generating a metric threshold alarm indicator when a corresponding metric value deviates outside the dynamic threshold range, and

generating an alarm severity score for each metric alarm event,

are performed in dynamic sampling agents located at each data source,

the step of performing a root cause analysis identifying a basic cause of a system alarm condition is performed

in a central service management platform, and

the method further includes the step of

communicating each metric alarm event and each alarm severity score for each metric alarm event from the dynamic sampling agent to the service management platform.

10. The method for performing system metric analysis of claim 2, wherein:

the step of collecting metric data from system sources is performed by dynamic sampling agents located at each data source, and

the steps of

collecting the metric data from a source in a source format, and

transforming the metric data from the corresponding source format into at least one common format,

are performed in each dynamic sampling agent by a data adapter.

11. A metric monitoring and analysis system, comprising:

a plurality of dynamic sampling agents, each dynamic sampling agent being located in a system element containing a metric of interest for monitoring and analysis and including

at least one plurality of data collectors for collecting metric data representing system operations from a plurality of system sources,

a threshold generator for determining a dynamic metric threshold range for each metric over successive time periods,

an alarm condition detector indicating a metric alarm event by

generating a metric threshold alarm indicator when a corresponding metric value deviates outside the dynamic threshold range, and

a deviation tracker for generating an alarm severity score for each metric alarm event, and

a single service management platform receiving alarm indicators and severity scores from the dynamic sampling agents and including

an alarm analyzer performing a root cause analysis identifying a basic cause of a system alarm condition by at least one of

correlation of grouped metrics with alarm conditions, and

forensic analysis of selected secondary forensic data items recorded upon the occurrence of at least one or more selected alarm conditions.

12. The metric monitoring and analysis system of claim 11, wherein each dynamic sampling agent further includes:

at least one data adapter for transforming the metric data from a corresponding source format into at least one common format.

13. The metric monitoring and analysis system of claim 11, wherein:

for each metric, the threshold generator

generates an empirical estimate of an actual distribution of values of the metric,

determines a trend in the actual distribution of the values of the metric,

determines an expected range of the values of the metric over the time periods, and

determines a dynamic threshold range for each metric based on the expected range of values of the metric.

14. The metric monitoring and analysis system of claim 11, wherein:

the alarm condition generator generates a metric threshold alarm indicator when a corresponding metric value deviates outside a static threshold range.

15. The metric monitoring and analysis system of claim 11, wherein:

the deviation tracker generates an alarm severity score for each metric alarm by determining the severity score as a ratio of a time and magnitude of the metric value deviation outside the threshold range to a width of the threshold range.

16. The metric monitoring and analysis system of claim 11, wherein the alarm analyzer performs a con-elation of grouped metrics by at least one of

a temporal alarm correlation process identifying metrics having values resulting in alarm conditions that are correlated in time, and

a statistical metric correlation process correlating pairs of metrics having values resulting in an alarm condition according to the relative ranks of the values of the metrics.

17. The metric monitoring and analysis system of claim 11, wherein the alarm analyzer records secondary forensic data items on the occurrence of at least selected alarm indicators by at least one of

recording selected key metrics defined as bounding the data items with at least one of temporal and statistical correlation of the metrics, and

recording data items selected from a service model of the system representing system operations associated with an alarm condition.

18. The metric monitoring and analysis system of claim 12, wherein the secondary forensic data items comprise non-metric information pertaining to system conditions at occurrence of an alarm condition.