System and methods for adaptive threshold determination for performance metrics

• US 7,076,695 B2
• Filed: 07/18/2002
• Issued: 07/11/2006
• Est. Priority Date: 07/20/2001
• Status: Active Grant

• Current Assignee: Riverbed Technology, LLC (Riverbed Technology Incorporated)
• Original Assignee: OPNET Technologies Incorporated (Riverbed Technology Incorporated)
• Inventors: Matthews, James W., McGee, John, Earley, John Michael
• Primary Examiner(s): Beausoliel, Robert
• Assistant Examiner(s): Puente, Emerson
• Application Number:
  US10/198,689
• Publication Number:
  US 20030079160A1
• Time in Patent Office: 1,454 Days
• Field of Search: 714/47, 714/4, 709/223, 709/224, 702/179, 702/186, 702/188
• US Class Current: 714/47.2
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0772 Means for error signaling, e.g. using interrupts, exception flags, dedicated error registers
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/327 Alarm or error message display
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 11/3495 for systems
• CPC Class Codes: G06F 17/15 Correlation function computation including computation of convolution operations arithmetic circuits for sum of products per se, e.g. multiply-accumulators G06F7/5443; digital filters, e.g. FIR, IIR, adaptive filters H03H17/00
• CPC Class Codes: G06F 17/18 for evaluating statistical data , e.g. average values, frequency distributions, probability functions, regression analysis forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04
• CPC Class Codes: G06F 2201/81 Threshold
• CPC Class Codes: G06F 2201/86 Event-based monitoring
• CPC Class Codes: G06F 2201/875 Monitoring of systems including the internet

Abstract

A system and method for dynamically generating alarm thresholds for performance metrics, and for applying those thresholds to generate alarms is described. Statistical methods are used to generate one or more thresholds for metrics that may not fit a Gaussian or normal distribution, or that may exhibit cyclic behavior or persistent shifts in the values of the metrics. The statistical methods used to generate the thresholds may include statistical process control (SPC) methods, normalization methods, and heuristics.

Claims

1. A method for dynamically generating at least one metric threshold associated with a metric and indicating alarm conditions in a monitored system, the method comprising the steps of:

repeatedly receiving data associated with the metric;

statistically analyzing the received data,

determining at least one value from the data using at least one heuristic technique including at least one of a quantile function and weighted linear regression techniques;

equating the at least one value with an updated alarm threshold; and

triggering an alarm on receipt of received data that violate the at least one updated alarm threshold.

2. The method of claim 1, wherein the metric relates to a computer network, and further comprising the step of using the alarm to assess performance in an e-commerce system.

3. The method of claim 1 further comprising the steps of (i) repeating the analysis step and (ii) adjusting the at least one updated alarm threshold based on previously established updated alarm limits.

4. The method of claim 3 further comprising the steps of:

computing at least one value;

filtering the at least one value; and

equating the at last one updated alarm threshold to the at least one value.

5. The method of claim 1 wherein the step of statistically analyzing the received data further comprises the step of categorizing the received data as normal.

6. The method of claim 1 wherein the step of statistically analyzing the received data further comprises the step of categorizing the received data as normalizable.

7. The method of claim 1 wherein the step of statistically analyzing the received data further comprises the step of categorizing the received data as non-normal.

8. The method of claim 1 wherein the step of triggering an alarm further comprises the steps of:

determining a fixed threshold by locking the updated alarm value to a value equal to the at least one value, and

comparing the received data with the fixed threshold.

9. The method of claim 1 wherein the step of triggering an alarm further comprises the step of comparing the mean and standard deviation of the received data with the at least one updated alarm threshold.

10. The method of claim 1 wherein the step of triggering an alarm further comprises the steps of:

normalizing the received data; and

comparing the mean and standard deviation of the normalized received data with the at least one updated alarm threshold.

11. The method of claim 1 wherein the step of triggering an alarm further comprises the step of comparing the mean of the received data with the at least one updated alarm threshold.

12. The method of claim 1 for dynamically generating at least one metric threshold associated with a metric and indicating alarm conditions in a monitored system, further comprising the step of:

filtering the at least one value by generating a time weighted sum of the received data as the at least one alarm threshold wherein each received data decays as a function of time.

13. A method for dynamically generating at least one metric threshold indicating alarm conditions in a monitored system, the method comprising the steps of:

establishing at least one default alarm threshold associated with a metric, including

repeatedly receiving data associated with the metric;

statistically analyzing the received data to establish at least one updated alarm threshold;

including

categorizing the received data as one of normal and normalizable by applying at least one of a chi-square test and an Anderson-Darling test to the received data; and

when the data is categorized as normalizable,

operating on the received data with a function representing the estimated cumulative distribution of the received data, producing a first result; and

operating on the first result with a quantile function of a normal distribution; and

triggering an alarm on receipt of received data that violate the at least one updated alarm threshold.

14. A method for dynamically generating at least one metric threshold indicating alarm conditions in a monitored system, the method comprising the steps of:

establishing at least one default alarm threshold associated with a metric, including

repeatedly receiving data associated with the metric;

statistically analyzing the received data to establish at least one updated alarm threshold;

including

computing at least one value;

filtering the at least one value; and

equating the at least one updated alarm threshold to the at least one value;

wherein

the at least one value is computed using at least one heuristic technique, including one of

a quantile function technique, and

a weighted linear regression technique; and

updating the at least one alarm threshold by

(i) repeating the analysis step and

(ii) adjusting the at least one updated alarm threshold based on previously established updated alarm limits; and

triggering an alarm on receipt of received data that violate the at least one updated alarm threshold.

15. A method for dynamically generating at least one metric threshold indicating alarm conditions in a monitored system, the method comprising the steps of:

establishing at least one default alarm threshold associated with a metric, including

repeatedly receiving data associated with the metric;

statistically analyzing the received data to establish at least one updated alarm threshold;

computing at least one value;

filtering the at least one value; and

equating the at last one updated alarm threshold to the at least one value;

wherein

the step of filtering further comprises computing a weighted sum of the received data;

updating the at least one alarm threshold by

(i) repeating the analysis step and

(ii) adjusting the at least one updated alarm threshold based on previously established updated alarm limits; and

triggering an alarm on receipt of received data that violate the at least one updated alarm threshold.

16. The method of claim 15 wherein the received data comprises historical data.

17. The method of claim 15 wherein the received data comprises a statistical summarization of raw metric data.

18. The method of claim 15 wherein the received data are associated with at least one predetermined time period.

19. Apparatus for dynamically generating at least one metric threshold associated with a metric and indicating alarm conditions in a monitored system, the apparatus comprising:

a data manager for repeatedly receiving data associated with the metric;

a threshold computation module for statistically analyzing the received data to establish a threshold when the data has one of a non-normal distribution and a non-normalizable distribution; and

when the data has one of a non-normal distribution and a non-normalizable distribution,

determining at least one value from the data using a heuristic technique including a quantile function and a linear regression technique, and

equating the at least one value with an updated alarm threshold;

a filter mechanism for filtering the at least one value by generating a time weighted sum of the received data as the at least one alarm threshold wherein each received data decays as a function of time;

a limit store for storing at least one updated alarm threshold; and

an alarm manager for triggering an alarm on receipt of received data that violates the at least one updated alarm threshold.