ANOMALY DETECTION USING CIRCUMSTANCE-SPECIFIC DETECTORS

• US 20160217022A1
• Filed: 10/07/2015
• Published: 07/28/2016
• Est. Priority Date: 01/23/2015
• Status: Active Grant

• Current Assignee: Lightbend, Inc.
• Original Assignee: OpsClarity, Inc. (Lightbend, Inc.)
• Inventors: VELIPASAOGLU, Omer Emre, Surana, Vishal, Sasturkar, Amit
• Granted Patent: US 10,261,851 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0772 Means for error signaling, e.g. using interrupts, exception flags, dedicated error registers
• CPC Class Codes: G06F 11/0787 Storage of error reports, e.g. persistent data storage, storage using memory protection
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/32 with visual or acoustical indication of the functioning of the machine
• CPC Class Codes: G06F 11/323 Visualisation of programs or trace data
• CPC Class Codes: G06F 11/34 Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: H04L 41/064 involving time analysis
• CPC Class Codes: H04L 41/142 using statistical or mathematical methods
• CPC Class Codes: H04L 41/147 for predicting network behaviour
• CPC Class Codes: H04L 41/16 using machine learning or artificial intelligence
• CPC Class Codes: H04L 41/5025 by proactively reacting to service quality change, e.g. by reconfiguration after service quality degradation or upgrade
• CPC Class Codes: H04L 43/045 for graphical visualisation of monitoring data

Abstract

The technology disclosed relates to learning how to efficiently display anomalies in performance data to an operator. In particular, it relates to assembling performance data for a multiplicity of metrics across a multiplicity of resources on a network and training a classifier that implements at least one circumstance-specific detector used to monitor a time series of performance data or to detect patterns in the time series. The training includes producing a time series of anomaly event candidates including corresponding event information used as input to the detectors, generating feature vectors for the anomaly event candidates, selecting a subset of the candidates as anomalous instance data, and using the feature vectors for the anomalous instance data and implicit and/or explicit feedback from users exposed to a visualization of the monitored time series annotated with visual tags for at least some of the anomalous instances data to train the classifier.

Claims

1. A method of learning how to efficiently display anomalies in performance data to an operator, the method including:

assembling performance data for a multiplicity of metrics across a multiplicity of resources on a network; and

training a classifier that implements at least one circumstance-specific detector used to monitor a time series of performance data or to detect patterns in the time series of the performance data, the training including;

producing a time series of anomaly event candidates including corresponding event information using the circumstance-specific detector;

generating feature vectors using the anomaly event candidates;

selecting a subset of the anomaly event candidates as anomalous instance data; and

using the feature vectors for the anomalous instance data and user feedback from users exposed to a visualization of the monitored time series annotated with visual tags for at least some of the anomalous instances data to train the classifier.

2. The method of claim 1, wherein the circumstance-specific detector includes a trend model that represents trend functions of the monitored time series.

3. The method of claim 1, wherein the circumstance-specific detector includes a periodic model that represents periodic variations in the monitored time series.

4. The method of claim 1, wherein the circumstance-specific detector includes a noise model that represents random or systematic fluctuations in the monitored time series.

5. The method of claim 1, wherein the circumstance-specific detector includes a weighted moving average that estimates a smoothed average of values in the monitored time series.

6. The method of claim 1, wherein the circumstance-specific detector includes an exponentially weighted average that estimates a smoothed average of the monitored time series.

7. The method of claim 1, wherein the circumstance-specific detector includes an autoregressive moving average (ARMA) to predict instances in the monitored time series.

8. The method of claim 1, wherein the circumstance-specific detector includes an autoregressive integrated moving average (ARIMA) to predict instances in the monitored time series.

9. The method of claim 1, wherein the circumstance-specific detector includes a Hidden Markov Model (HMM) that infers stage changes in the monitored time series and identifies at least some state changes as anomalies.

10. The method of claim 1, wherein the circumstance-specific detector treats instances in the monitored times series as a Gaussian Process and uses deviations from a mean outcome of the Gaussian Process to identify specific instances as anomalies.

11. The method of claim 1, wherein the circumstance-specific detector generates a frequency domain decomposition of the monitored time series.

12. The method of claim 1, wherein the circumstance-specific detector is a wavelet analysis that generates a time-frequency domain decomposition of the monitored time series.

13. The method of claim 1, further including:

identifying a set of instances in the monitored time series within a predetermined time window as an unordered set of random samples;

estimating at least one of a histogram and an empirical cumulative density function (CDF) using the random samples;

comparing empirical CDFs of at least two or more time-varying instances and determining similarity of underlying probability distribution between the time-varying instances using two-sided Kolmogorov-Smirnov (KS) test; and

determining anomalies in the monitored time series by thresholding a KS distance that marks predetermined deviations as change points.

14. The method of claim 1, wherein the multiplicity of metrics includes at least one of a system level metrics and a service level metrics, including at least one of:

central processing unit (CPU) usage;

disk usage;

memory usage;

process fork rate;

network usage;

system load;

system service level agreements (SLAs);

Hadoop block read latency;

MongoDB read latency; and

Nginx requests per second.

15. The method of claim 1, further including using the explicit feedback from users further includes receiving at least one of emoticons, likes, stars, thumbs up, bonuses, ratings, and badges corresponding to a time series of feature vectors.

16. The method of claim 1, further including using the feature vectors to identify interrelated anomaly event candidates based on at least:

one common time-window within which the anomaly event candidates were detected;

one common metric for which the anomaly event candidates were detected;

one common resource on which the anomaly event candidates occurred;

two or more connected metrics on which two or more respective anomaly event candidates were detected; and

two or more connected resources on which two or more respective anomaly event candidates occurred.

17. The method of claim 1, wherein using the user feedback from users further includes capturing users'"'"' navigation actions during evaluation of a ranked list of anomalies generated using a time series of feature vectors.

18. The method of claim 1, wherein using the user feedback from users further includes determining users'"'"' attention duration to respective anomalies during evaluation of a ranked list of anomalies generated using a time series of feature vectors.

19. A system including one or more processors coupled to memory, the memory loaded with computer instructions to learn how to efficiently display anomalies in performance data to an operator, the instructions, when executed on the processors, implement actions comprising:

assembling performance data for a multiplicity of metrics across a multiplicity of resources on a network; and

training a classifier that implements at least one circumstance-specific detector used to monitor a time series of performance data or to detect patterns in the time series of the performance data, the training including;

producing a time series of anomaly event candidates including corresponding event information using the circumstance-specific detector;

generating feature vectors using the anomaly event candidates;

selecting a subset of the anomaly event candidates as anomalous instance data; and

using the feature vectors for the anomalous instance data and user feedback from users exposed to a visualization of the monitored time series annotated with visual tags for at least some of the anomalous instances data to train the classifier.

20. A non-transitory computer readable storage medium impressed with computer program instructions to learn how to efficiently display anomalies in performance data to an operator, the instructions, when executed on a processor, implement a method comprising:

assembling performance data for a multiplicity of metrics across a multiplicity of resources on a network; and

training a classifier that implements at least one circumstance-specific detector used to monitor a time series of performance data or to detect patterns in the time series of the performance data, the training including;

producing a time series of anomaly event candidates including corresponding event information using the circumstance-specific detector;

generating feature vectors using the anomaly event candidates;

selecting a subset of the anomaly event candidates as anomalous instance data; and

using the feature vectors for the anomalous instance data and user feedback from users exposed to a visualization of the monitored time series annotated with visual tags for at least some of the anomalous instances data to train the classifier.