Anomaly detection

• US 10,445,311 B1
• Filed: 06/27/2014
• Issued: 10/15/2019
• Est. Priority Date: 09/11/2013
• Status: Active Grant

• Current Assignee: Sumo Logic, Sumo Logic, Inc.
• Original Assignee: Sumo Logic, Inc.
• Inventors: Saurabh, Kumar, Andrzejewski, David M., Zhao, Yuchen, Beedgen, Christian Friedrich, Kurtic, Bruno
• Primary Examiner(s): Yen, Syling
• Application Number:
  US14/318,409
• Time in Patent Office: 1,936 Days
• Field of Search: 707706, 707722, 707736, 707758, 707781
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection

Abstract

Analyzing log data, such as security log data and machine data, is disclosed. A baseline is built for a set of machine data. The baseline is built at least in part by determining a plurality of signature profiles for a plurality of respective time slices. An occurrence of an anomaly associated with the source of the machine data is determined. The occurrence is determined at least in part by determining that received machine data does not conform to the baseline within a threshold.

Claims

1. A system, comprising:

a processor configured to;

build a baseline for a set of machine data at least in part by determining a plurality of signature profiles for a plurality of respective time slices, wherein determining a signature profile for a given time slice included in the plurality of time slices comprises determining a distribution of signatures to which machine data for the given time slice matches;

determine an occurrence of an anomaly associated with a source of the set of machine data at least in part by determining that received machine data does not conform to the baseline within a threshold;

present the anomaly to a user via one or more interfaces;

receive, from the user and via the one or more interfaces, information associated with the anomaly, wherein the information provided by the user comprises a severity of the anomaly;

assign the severity provided by the user to the anomaly;

store a representation of the anomaly and the information provided by the user as a first event;

determine an occurrence of a subsequent anomaly;

automatically classify the subsequent anomaly as a same type of anomaly as the first event;

in response to automatically classifying the subsequent anomaly as the same type of anomaly as the first event, automatically annotate the subsequent anomaly with the severity previously provided by the user; and

perform an action with respect to the subsequent anomaly based at least in part on the annotation of the subsequent anomaly with the severity previously provided by the user; and

a memory coupled to the processor and configured to provide the processor with instructions.

2. The system of claim 1 wherein the processor is further configured to receive a set of signatures associated with an application.

3. The system of claim 2 wherein receiving the set of signatures includes determining the set of signatures at least in part by grouping a plurality of similar log entries included in a set of log entries together and associating those log entries included in the group with a signature usable to identify additional log entries which should be included in the group.

4. The system of claim 1 wherein a signature profile included in the plurality of signature profiles comprises a histogram of signatures.

5. The system of claim 1 wherein the processor is further configured to alert an administrator of the occurrence of the anomaly.

6. The system of claim 1 wherein the processor is further configured to cause the representation of the anomaly to be displayed to a first administrator.

7. The system of claim 6 wherein the first administrator is prompted to provide the severity of the anomaly and save the anomaly as the first event.

8. The system of claim 7 wherein the severity provided by the first administrator is editable by a second administrator.

9. The system of claim 7 wherein the first event is stored in an event repository, wherein the event repository is accessible by a first and second user, and wherein log data accessible to the first user and is not accessible to the second user can be classified using the event repository.

10. The system of claim 1 wherein the baseline is built using machine data received from a first data source and wherein the subsequent anomaly is present in additional machine data received from the first data source.

11. The system of claim 1 wherein the subsequent anomaly is present in machine data received from a different source of machine data than was used to build the baseline.

12. The system of claim 1 wherein the baseline is one of a plurality of segmented baselines and wherein the baseline is selected from the plurality of segmented baselines for use in detecting anomalies based at least in part on a current condition matching a condition associated with the selected baseline.

13. The system of claim 12 wherein the baseline and the condition share at least one of a common date and a common time.

14. A method, comprising:

building a baseline for a set of machine data at least in part by determining a plurality of signature profiles for a plurality of respective time slices, wherein determining a signature profile for a given time slice included in the plurality of time slices comprises determining a distribution of signatures to which machine data for the given time slice matches;

determining an occurrence of an anomaly associated with a source of the set of machine data at least in part by determining that received machine data does not conform to the baseline within a threshold;

presenting the anomaly to a user via one or more interfaces;

receiving, from the user and via the one or more interfaces, information associated with the anomaly, wherein the information provided by the user comprises a severity of the anomaly;

assigning the severity provided by the user to the anomaly;

storing a representation of the anomaly and the information provided by the user as a first event;

determining an occurrence of a subsequent anomaly;

automatically classifying the subsequent anomaly as a same type of anomaly as the first event;

in response to automatically classifying the subsequent anomaly as the same type of anomaly as the first event, automatically annotating the subsequent anomaly with the severity previously provided by the user; and

performing an action with respect to the subsequent anomaly based at least in part on the annotation of the subsequent anomaly with the severity previously provided by the user.

15. The method of claim 14 further comprising receiving a set of signatures associated with an application.

16. The method of claim 15 wherein receiving the set of signatures includes determining the set of signatures at least in part by grouping a plurality of similar log entries included in a set of log entries together and associating those log entries included in the group with a signature usable to identify additional log entries which should be included in the group.

17. The method of claim 14 wherein a signature profile included in the plurality of signature profiles comprises a histogram of signatures.

18. The method of claim 14 further comprising alerting an administrator of the occurrence of the anomaly.

19. The method of claim 14 further comprising causing the representation of the anomaly to be displayed to a first administrator.

20. The method of claim 19 wherein the first administrator is prompted to provide the severity of the anomaly and save the anomaly as the first event.

21. The method of claim 20 wherein the severity provided by the first administrator is editable by a second administrator.

22. The method of claim 20 wherein the first event is stored in an event repository, wherein the event repository is accessible by a first and second user, and wherein log data accessible to the first user and is not accessible to the second user can be classified using the event repository.

23. The method of claim 14 wherein the baseline is built using machine data received from a first data source and wherein the subsequent anomaly is present in additional machine data received from the first data source.

24. The method of claim 14 wherein the subsequent anomaly is present in machine data received from a different source of machine data than was used to build the baseline.

25. The method of claim 14 wherein the baseline is one of a plurality of segmented baselines and wherein the baseline is selected from the plurality of segmented baselines for use in detecting anomalies based at least in part on a current condition matching a condition associated with the selected baseline.

26. The method of claim 25 wherein the baseline and the condition share at least one of a common date and a common time.

27. A computer program product embodied in a tangible computer readable storage medium and comprising computer instructions for:

building a baseline for a set of machine data at least in part by determining a plurality of signature profiles for a plurality of respective time slices, wherein determining a signature profile for a given time slice included in the plurality of time slices comprises determining a distribution of signatures to which machine data for the given time slice matches; and

determining an occurrence of an anomaly associated with a source of the set of machine data at least in part by determining that received machine data does not conform to the baseline within a threshold;

presenting the anomaly to a user via one or more interfaces;

receiving, from the user and via the one or more interfaces, information associated with the anomaly, wherein the information provided by the user comprises a severity of the anomaly;

assigning the severity provided by the user to the anomaly;

storing a representation of the anomaly and the information provided by the user as a first event;

determining an occurrence of a subsequent anomaly;

automatically classifying the subsequent anomaly as a same type of anomaly as the first event;

in response to automatically classifying the subsequent anomaly as the same type of anomaly as the first event, automatically annotating the subsequent anomaly with the severity previously provided by the user; and

performing an action with respect to the subsequent anomaly based at least in part on the annotation of the subsequent anomaly with the severity previously provided by the user.