STATISTICALLY-BASED ANOMALY DETECTION IN UTILITY CLOUDS

• US 20130030761A1
• Filed: 07/29/2011
• Published: 01/31/2013
• Est. Priority Date: 07/29/2011
• Status: Abandoned Application

• Current Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Original Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Inventors: LAKSHMINARAYAN, Choudur, Viswanathan, Krishnamurthy, Wang, Chengwei, Talwar, Vanish
• Application Number:
  US13/194,798
• Publication Number:
  US 20130030761A1
• Time in Patent Office: Days
• US Class Current: 702/179
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 18/231 Hierarchical techniques, i.e. dividing or merging pattern sets so as to obtain a dendrogram
• CPC Class Codes: H04L 41/0604 using filtering, e.g. reduction of information by using priority, element types, position or time

Abstract

Systems and methods for detecting anomalies in a large scale and cloud datacenter are disclosed. Anomaly detection is performed in an automated, statistical-based manner by using a parametric Gini coefficient technique or a non-parametric Tukey technique. In the parametric Gini coefficient technique, sample data is collected within a look-back window. The sample data is normalized to generate normalized data, which is binned into a plurality of bins defined by bin indices. A Gini coefficient and a threshold are calculated for the look-back window and the Gini coefficient is compared to the threshold to detect an anomaly in the sample data. In the non-parametric Tukey technique, collected sample data is divided into quartiles and compared to adjustable Tukey thresholds to detect anomalies in the sample data.

Claims

1. A method for detecting anomalies in a large scale and cloud datacenter, the method comprising:

collecting sample data within a look-back window;

normalizing the sample data to generate normalized data;

binning the normalized data into a plurality of bins defined by bin indices;

calculating a Gini coefficient for the look-back window;

calculating a Gini standard deviation dependent threshold; and

comparing the Gini coefficient to the Gini standard deviation dependent threshold to detect an anomaly in the sample data.

2. The method of claim 1, wherein the sample data comprises a set of performance metrics and monitoring data for the datacenter.

3. The method of claim 1, wherein the normalized data is generated based on the mean and standard deviation of the sample data.

4. The method of claim 1, further comprising generating at least one vector based on the bin indices.

5. The method of claim 1, wherein the Gini coefficient is calculated based on the at least one vector.

6. The method of claim 1, wherein the Gini standard deviation dependent threshold is calculated using the standard deviation of the Gini coefficient over a series of sliding look-back windows.

7. The method of claim 1, further comprising aggregating bin indices for multiple nodes in the datacenter to form a vector representing sample data for the multiple nodes.

8. The method of claim 7, further comprising calculating a Gini coefficient based on the vector representing sample data for the multiple nodes.

9. The method of claim 1, further comprising aggregating Gini coefficients for multiple nodes to form an aggregated Gini coefficient.

10. The method of claim 1, further comprising sliding the look-back window to detect anomalies in sample data within the sliding window.

11. A system for detecting anomalies in a large scale and cloud datacenter, the system comprising:

a metrics collection module to collect metrics and monitoring data across the datacenter within a look-back window;

a statistical-based anomaly detection module for detecting anomalies in the collected data, the statistical-based anomaly detection module comprising;

a normalization module to generate normalized data from the collected data;

a binning module to place the normalized data into a plurality of bins defined by bin indices;

a Gini coefficient module to calculate a Gini coefficient for the look-back window;

a threshold module to calculate a Gini standard deviation dependent threshold; and

an anomaly alarm module to compare the Gini coefficient to the Gini standard deviation dependent threshold and generate an alarm when an anomaly in the collected data is detected; and

a dashboard module to display the look-back window and the detected anomalies.

12. The system of claim 11, wherein the metrics and monitoring data comprise service level metrics, system level metrics, and platform metrics.

13. The system of claim 11, wherein the normalization module generates normalized data based on the mean and standard deviation of the collected data.

14. The system of claim 11, wherein the binning module generates at least one vector based on the bin indices.

15. The system of claim 11, wherein the Gini coefficient is calculated based on the at least one vector.

16. The system of claim 11, wherein the Gini standard deviation dependent threshold is calculated using the standard deviation of the Gini coefficient over a series of sliding look-back windows.

17. The system of claim 11, further comprising an aggregation module to aggregate anomaly detection for multiple nodes in the datacenter.

18. A system for detecting anomalies in a large scale and cloud datacenter, the system comprising:

a metrics collection module to collect metrics and monitoring data across the datacenter within a look-back window;

a data quartile module to divide the collected data in quartiles;

a Tukey threshold module to generate adjustable thresholds; and

an anomaly alarm module to compare the collected data in the quartiles to the thresholds and generate an alarm when an anomaly in the collected data is detected.

19. The system of claim 18, wherein the adjustable thresholds comprise metric-dependent thresholds.

20. The system of claim 18, wherein the alarm is generated when the collected data in the quartiles is outside a range defined by the thresholds.