Anomaly detection for signals populated based on historical data points

• US 10,558,516 B2
• Filed: 10/31/2018
• Issued: 02/11/2020
• Est. Priority Date: 07/08/2016
• Status: Active Grant

• Current Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Original Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Inventors: Leverich, Jacob Barton, Cai, Shang, Zhang, Hongyang, Ganea, Mihai, Cruise, Alex
• Primary Examiner(s): Bryan, Jason B
• Application Number:
  US16/176,186
• Publication Number:
  US 20190065298A1
• Time in Patent Office: 468 Days
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40

Abstract

A continuous anomaly detection service receives data stream and performs continuous anomaly detection on the incoming data streams. This continuous anomaly detection is performed based on anomaly detection definitions, which define a signal used for anomaly detection and an anomaly detection configuration. These anomaly detection definitions can be modified, such that continuous anomaly detection continues to be performed for the data stream and the signal, based on the new anomaly detection definition.

Claims

1. A computer-implemented method for configuring an anomaly detection service, comprising:

accessing an anomaly detection definition comprising an anomaly detection configuration that defines how to populate a sequential set of data points for a corresponding signal from a plurality of historical data points, distinct from the sequential set of data points;

generating the signal based on the anomaly detection configuration and the plurality of historical data points;

determining that one of a plurality of data streams is associated with the signal;

updating the signal by inserting into the signal a set of data points from the data stream associated with the signal; and

periodically analyzing memory storing the signal to determine a corresponding anomaly result for the anomaly detection definition.

2. The computer-implemented method of claim 1, wherein accessing the anomaly detection definition occurs in association with an initial configuration of the anomaly detection definition, and wherein generating the signal comprises pre-loading the signal with a subset of the plurality of historical data points before populating the signal with other data points received after generating the signal.

3. The computer-implemented method of claim 1, wherein generating the signal comprises filling the signal with a subset of the plurality of historical data points based on a determination that the signal is incomplete.

4. The computer-implemented method of claim 1, wherein generating the signal comprises filling the signal with a subset of the plurality of historical data points based on a determination that the signal includes corrupted data.

5. The computer-implemented method of claim 1, wherein generating the signal comprises filling the signal with a subset of the plurality of historical data points based on a determination that the anomaly detection definition includes an updated anomaly detection configuration.

6. The computer-implemented method of claim 1, further comprising:

receiving a request to create a new anomaly detection definition, wherein the request includes a new anomaly detection configuration for a new signal of the new anomaly detection definition;

identifying a subset of the plurality of historical data points based on the new anomaly detection definition; and

generating the new signal by pre-loading the new signal with the subset of the plurality of historical data points before populating the new signal with other data points received after generating the new signal.

7. The computer-implemented method of claim 1, further comprising:

receiving a request to create a new anomaly detection definition, wherein the request includes a new anomaly detection configuration for a new signal of the new anomaly detection definition;

identifying a subset of the plurality of historical data points based on the new anomaly detection definition;

generating the new signal by pre-loading the new signal with the subset of the plurality of historical data points before populating the new signal with other data points received after generating the new signal; and

determining an anomaly result for the new anomaly detection definition based on the new signal and the new anomaly detection configuration.

8. The computer-implemented method of claim 1, wherein the anomaly detection definition is one of a plurality of anomaly detection definitions, and wherein a configuration object includes all of the plurality of anomaly detection definitions.

9. The computer-implemented method of claim 1, wherein the anomaly detection definition is one of a plurality of anomaly detection definitions, and wherein a configuration object includes all of the plurality of anomaly detection definitions, and wherein the configuration object defines a shared search for the plurality of anomaly detection definitions.

10. The computer-implemented method of claim 1, further comprising:

receiving the plurality of data streams; and

determining that the data stream is associated with the signal and identifying the set of data points from the data stream associated with the signal using the anomaly detection configuration.

11. A non-transitory computer-readable storage medium comprising instructions stored thereon, which when executed by one or more processors, cause the one or more processors to perform operations comprising:

accessing an anomaly detection definition comprising an anomaly detection configuration that defines how to populate a sequential set of data points for a corresponding signal from a plurality of historical data points, distinct from the sequential set of data points;

generating the signal based on the anomaly detection configuration and the plurality of historical data points;

determining that one of a plurality of data streams is associated with the signal;

updating the signal by inserting into the signal a set of data points from the data stream associated with the signal; and

periodically analyzing memory storing the signal to determine a corresponding anomaly result for the-anomaly detection definition.

12. The non-transitory computer-readable storage medium of claim 11, wherein accessing the anomaly detection definition is associated with an initial configuration of the anomaly detection definition, and wherein generating the signal comprises pre-loading the signal with a subset of the plurality of historical data points before populating the signal with other data points received after generating the signal.

13. The non-transitory computer-readable storage medium of claim 11, wherein generating the signal comprises filling the signal with a subset of the plurality of historical data points based on a determination that the signal is incomplete.

14. The non-transitory computer-readable storage medium of claim 11, wherein generating the signal comprises filling the signal with a subset of the plurality of historical data points based on a determination that the signal includes corrupted data.

15. The non-transitory computer-readable storage medium of claim 11, wherein generating the signal comprises filling the signal with a subset of the plurality of historical data points based on a determination that the anomaly detection definition includes an updated anomaly detection configuration.

16. The non-transitory computer-readable storage medium of claim 11, the operations further comprising:

receiving a request to create a new anomaly detection definition, wherein the request includes a new anomaly detection configuration for a new signal of the new anomaly detection definition;

identifying a subset of the plurality of historical data points based on the new anomaly detection definition; and

generating the new signal by pre-loading the new signal with the subset of the plurality of historical data points before populating the signal with other data points received after generating the signal.

17. The non-transitory computer-readable storage medium of claim 11, the operations further comprising:

receiving a request to create a new anomaly detection definition, wherein the request includes a new anomaly detection configuration for a new signal of the new anomaly detection definition;

identifying a subset of the plurality of historical data points based on the new anomaly detection definition;

generating the new signal by pre-loading the new signal with the subset of the plurality of historical data points before populating the signal with other data points received after generating the signal; and

determining an anomaly result for the new anomaly detection definition based on the new signal and the new anomaly detection configuration.

18. The non-transitory computer-readable storage medium of claim 11, wherein the anomaly detection definition is one of a plurality of anomaly detection definitions, and wherein a configuration object includes all of the plurality of anomaly detection definitions.

19. The non-transitory computer-readable storage medium of claim 11, wherein the anomaly detection definition is one of a plurality of anomaly detection definitions, and wherein a configuration object includes all of the plurality of anomaly detection definitions, and wherein the first configuration object defines a shared search for the plurality of anomaly detection definitions.

20. The non-transitory computer-readable storage medium of claim 11, the operations further comprising:

receiving the plurality of data streams; and

determining that the data stream is associated with the signal and identifying the set of one or more data points from the data stream associated with the signal using the anomaly detection configuration.

21. A system for performing anomaly detection, comprising:

at least one memory having instructions stored thereon; and

at least one processor configured to execute the instructions to;

access an anomaly detection definition comprising an anomaly detection configuration that defines how to populate a sequential set of data points for a corresponding signal from a plurality of historical data points, distinct from the sequential set of data points;

generate the signal based on the anomaly detection configuration and the plurality of historical data points;

determine that one of a plurality of data streams is associated with the signal;

update the signal by inserting into the signal a set of data points from the data stream associated with the signal; and

periodically analyze a portion of the at least one memory storing the signal to determine a corresponding anomaly result for the-anomaly detection definition.

22. The system of claim 21, wherein the at least one processor is further configured to execute the instructions to access the anomaly detection definition in association with an initial configuration of the anomaly detection definition and generate the signal by pre-loading the signal with a subset of the plurality of historical data points before populating the signal with other data points received after generating the signal.

23. The system of claim 21, wherein the at least one processor is further configured to execute the instructions to generate the signal by filling the signal with a subset of the plurality of historical data points based on a determination that the signal is incomplete.

24. The system of claim 21, wherein the at least one processor is further configured to execute the instructions to generate the signal by filling the signal with a subset of the plurality of historical data points based on a determination that the signal includes corrupted data.

25. The system of claim 21, wherein the at least one processor is further configured to execute the instructions to generate the signal by filling the signal with a subset of the plurality of historical data points based on a determination that the anomaly detection definition includes an updated anomaly detection configuration.

26. The system of claim 21, wherein the at least one processor is further configured to execute the instructions to:

receive a request to create a new anomaly detection definition, wherein the request includes a new anomaly detection configuration for a new signal of the new anomaly detection definition;

identify a subset of the plurality of historical data points based on the new anomaly detection definition; and

generate the new signal by pre-loading the new signal with the subset of the plurality of historical data points before populating the signal with other data points received after generating the signal.

27. The system of claim 21, wherein the at least one processor is further configured to execute the instructions to:

receive a request to create a new anomaly detection definition, wherein the request includes a new anomaly detection configuration for a new signal of the new anomaly detection definition;

identify a subset of the plurality of historical data points based on the new anomaly detection definition;

generate the new signal by pre-loading the new signal with the subset of the plurality of historical data points before populating the signal with other data points received after generating the signal; and

determine an anomaly result for the new anomaly detection definition based on the new signal and the new anomaly detection configuration.

28. The system of claim 21, wherein the anomaly detection definition is one of a plurality of anomaly detection definitions, and wherein the at least one memory includes a configuration object includes all of the plurality of anomaly detection definitions.

29. The system of claim 21, wherein the anomaly detection definition is one of a plurality of anomaly detection definitions, wherein the at least one memory includes a configuration object includes all of the plurality of anomaly detection definitions, and wherein the configuration object defines a shared search for the plurality of anomaly detection definitions.

30. The system of claim 21, wherein the at least one processor is further configured to execute the instructions to:

receive the plurality of data streams; and

determine that the data stream is associated with the signal and identify the set of data points from the data stream associated with the signal using the anomaly detection configuration.