Anomaly detection to identify security threats

• US 10,673,880 B1
• Filed: 09/26/2016
• Issued: 06/02/2020
• Est. Priority Date: 09/26/2016
• Status: Active Grant

• Current Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Original Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Inventors: Pratt, Robert Winslow, Bulusu, Ravi Prasad
• Primary Examiner(s): Patel, Haresh N
• Application Number:
  US15/276,647
• Time in Patent Office: 1,345 Days
• Field of Search: 726 1
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06F 21/554 involving event detection and direct action
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/045 Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence
• CPC Class Codes: H04L 2463/121 Timestamp
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1433 Vulnerability analysis
• CPC Class Codes: H04L 63/20 for managing network security; network security policies in general filtering policies H04L63/0227

Abstract

Techniques are described for processing anomalies detected using user-specified rules with anomalies detected using machine-learning based behavioral analysis models to identify threat indicators and security threats to a computer network. In an embodiment, anomalies are detected based on processing event data at a network security system that used rules-based anomaly detection. These rules-based detected anomalies are acquired by a network security system that uses machine-learning based anomaly detection. The rules-based detected anomalies are processed along with machine learning detected anomalies to detect threat indicators or security threats to the computer network. The threat indicators and security threats are output as alerts to the network security system that used rules-based anomaly detection.

Claims

1. A computer-implemented method comprising:

processing events associated with an information technology environment using a machine-learning anomaly detection model to generate first anomaly data indicative of a first anomaly detected in the events;

processing the events using an anomaly detection rule to generate second anomaly data indicative of a second anomaly detected in the events, the anomaly detection rule having been defined by a user;

inputting the first anomaly data and the second anomaly data into a machine-learning threat indicator model; and

processing the first anomaly data concurrently with the second anomaly data using the machine-learning threat indicator model to identify a threat indicator associated with a potential security threat to the information technology environment.

2. The method of claim 1, wherein each of the events comprises:

a timestamp;

a portion of raw machine data, associated with the timestamp, generated by a component of the information technology environment related to activity in the information technology environment.

3. The method of claim 1, wherein the second anomaly is acquired from a network security system that uses rule-based anomaly detection based on the anomaly detection rule defined by the user.

4. The method of claim 1, further comprising:

causing display of information about the first anomaly or the second anomaly via a graphical user interface of a network security system.

5. The method of claim 1, further comprising:

outputting the threat indicator for display via a graphical user interface of a network security system that uses rule-based anomaly detection.

6. The method of claim 1, wherein generating the first anomaly data comprises:

determining that the first anomaly is associated with a particular category of anomalous activity; and

wherein the first anomaly data includes a tag indicative that the first anomaly is associated with the particular category of anomalous activity.

7. The method of claim 1, wherein the second anomaly data includes a user-specified tag indicative that the second anomaly is associated with a particular category of anomalous activity.

8. The method of claim 1, further comprising:

based on a tag in the second anomaly data associated with the second anomaly, determining that the second anomaly corresponds to the first anomaly, before processing the first anomaly data concurrently with the second anomaly data using the machine-learning threat indicator model to identify the threat indicator.

9. The method of claim 1, further comprising:

causing display of an option to the user to define the anomaly detection rule.

10. The method of claim 1, wherein the machine-learning anomaly detection model includes:

model processing logic defining a process for assigning an anomaly score to the events; and

a model state defining a set of parameters for applying the model processing logic; and

wherein generating the first anomaly data includes;

applying the model processing logic to assign the anomaly score to the events; and

outputting an indicator of the first anomaly if the anomaly score satisfies a specified scoring criterion.

11. The method of claim 1, such that the second anomaly is detected at another computer system by:

searching the events for particular events that include data associated with a particular entity in the information technology environment;

applying the anomaly detection rule to the particular events; and

outputting an indicator of the second anomaly if the particular events satisfy the anomaly detection rule.

12. The method of claim 1, wherein the machine-learning threat indicator model includes:

model processing logic defining a process for assigning a threat indicator score to the processed anomaly data; and

a model state defining a set of parameters for applying the model processing logic; and

wherein identifying the threat indicator includes;

assigning the threat indicator score based on the processing of the first anomaly data concurrently with the second anomaly data using the machine-learning threat indicator model;

wherein the threat indicator is identified if the threat indicator score satisfies a specified scoring criterion.

13. The method of claim 1, wherein identifying the threat indicator includes one or more of:

aggregating the first anomaly data and the second anomaly data;

correlating the first anomaly data and the second anomaly data;

or

enhancing the first anomaly data and the second anomaly.

14. The method of claim 1, wherein identifying the threat indicator includes:

determining a measure of anomalies associated with a particular entity in the information technology environment over a time period, the particular entity including any of a user, a device, or an application;

wherein the threat indicator is identified if the measure of anomalies associated with the particular entity satisfies a specified criterion.

15. The method of claim 1, wherein the machine-learning threat indictor model includes model processing logic for detecting a third anomaly, the third anomaly different than the first anomaly and the second anomaly;

and

wherein the threat indicator is identified in response to detecting the third anomaly based on processing the first anomaly data concurrently with the second anomaly data using the machine-learning threat indicator model.

16. The method of claim 1, wherein identifying the threat indicator includes:

determining a number of entities in the information technology environment associated with a particular category of anomaly over a time period;

wherein the threat indicator is identified if the number of entities associated with the particular category of anomaly satisfies a specified criterion.

17. The method of claim 1, wherein identifying the threat indicator includes:

monitoring a duration of the first anomaly or the second anomaly over a time period;

wherein the threat indicator is identified if the duration of the first anomaly or the second anomaly satisfies a specified criterion.

18. The method of claim 1, further comprising:

acquiring, by the computer system, the anomaly detection rule from a network security system that uses rule-based anomaly detection.

19. The method of claim 1, further comprising:

determining an identity of a particular entity associated with the first or second anomaly based on the events;

applying, by the computer system, a tag to the first anomaly data or the second anomaly data indicative of an association of the particular entity with the first or second anomaly.

20. The method of claim 1, further comprising:

identifying a plurality of entities associated with the information technology environment by processing the events, the plurality of entities including a user represented by a user identifier included in the events and a machine represented by a machine identifier included in the events;

determining a probability of association between the machine identifier and the user identifier, based on the events;

detecting, by the computer system, that the probability of association satisfies a specified criterion; and

in response to detecting that the probability of association satisfies the specified criterion, creating a user association record indicative that a particular event is associated with the particular user based on the presence of the machine identifier in the particular event.

21. The method of claim 1, further comprising:

accessing a user association record including data associating a particular user identifier to a particular machine identifier;

determining that a particular user is associated with the first anomaly based on the accessed user association record; and

applying a tag to the first anomaly data, the tag indicative of an association of the particular user with the first anomaly.

22. The method of claim 1, further comprising:

accessing a user association record including data associating a particular user identifier to a particular machine identifier;

determining that a particular user is associated with the second anomaly based on the accessed user association record; and

applying a tag to the second anomaly data, the tag indicative of an association of the particular user with the second anomaly.

23. The method of claim 1, further comprising:

outputting the threat indicator for display via a graphical user interface of a network security system;

receiving a request via the graphical user interface for raw machine data related to the threat indicator; and

searching the events for particular events related to the threat indicator, in response to receiving the request; and

outputting the raw machine data included in the particular events for display via the graphical user interface.

24. The method of claim 1, processing the first anomaly data with the second anomaly data includes using a processing engine running Apache Storm.

25. The method of claim 1, wherein processing the first anomaly data with the second anomaly is performed in any of real-time or batch mode.

26. The method of claim 1, further comprising:

identifying an actual security threat to the information technology environment by processing the threat indicator with another threat indicator.

27. The method of claim 1, further comprising:

correlating the threat indicator against a plurality of pre-defined security scenarios;

generating a set of candidate security threats based on the correlation; and

for each of the candidate security threats;

comparing the threat indicator against pre-configured patterns associated with the candidate security threat;

generating a pattern matching score based on a result of the comparing; and

identifying a security threat to the information technology environment if the pattern matching score satisfies a specified scoring criterion.

28. A computer system comprising:

processor; and

a

a storage device having instructions stored thereon, which when executed by the processor cause the computer system to;

process events associated with an information technology environment using a machine-learning anomaly detection model to generate first anomaly data indicative of a first anomaly detected in the events;

process the events using an anomaly detection rule to generate second anomaly data indicative of a second anomaly detected in the events, the anomaly detection rule having been defined by a user;

input the first anomaly data and the second anomaly data into a machine-learning threat indicator model; and

process the first anomaly data concurrently with the second anomaly data using the machine-learning threat indicator model to identify a threat indicator associated with a potential security threat to the information technology environment.

29. A non-transitory computer readable medium containing instructions, execution of which in a computer system causes the computer system to:

process events associated with an information technology environment using a machine-learning anomaly detection model to generate first anomaly data indicative of a first anomaly detected in the events;

process the events using an anomaly detection rule to generate second anomaly data indicative of a second anomaly detected in the events, the anomaly detection rule having been defined by a user;

input the first anomaly data and the second anomaly data into a machine-learning threat indicator model; and

process the first anomaly data concurrently with the second anomaly data using the machine-learning threat indicator model to identify a threat indicator associated with a potential security threat to the information technology environment.