DETECTION OF ANOMALIES, THREAT INDICATORS, AND THREATS TO NETWORK SECURITY

• US 20170063905A1
• Filed: 10/30/2015
• Published: 03/02/2017
• Est. Priority Date: 08/31/2015
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Muddu, Sudhakar, Tryfonas, Christos
• Granted Patent: US 10,419,450 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/444 Spatial browsing, e.g. 2D maps, 3D or virtual spaces
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 40/134 Hyperlinking
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06V 10/225 based on a marking or identifier characterising the area
• CPC Class Codes: H04L 2463/121 Timestamp cryptographic mechanisms or cryptographic arrangements involving time stamps H04L9/3297
• CPC Class Codes: H04L 41/0893 Assignment of logical groups to network elements
• CPC Class Codes: H04L 41/145 involving simulating, designing, planning or modelling of a network
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 43/00 : Arrangements for monitoring or testing data switching networks
• CPC Class Codes: H04L 43/045 : for graphical visualisation of monitoring data
• CPC Class Codes: H04L 43/062 : related to network traffic
• CPC Class Codes: H04L 43/08 : Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 63/06 : for supporting key management in a packet data network cryptographic mechanisms or cryptographic arrangements for key management H04L9/08
• CPC Class Codes: H04L 63/1408 : by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/1416 : Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 : Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1433 : Vulnerability analysis
• CPC Class Codes: H04L 63/1441 : Countermeasures against malicious traffic countermeasures against attacks on cryptographic mechanisms H04L9/002
• CPC Class Codes: H04L 63/20 : for managing network security; network security policies in general filtering policies H04L63/0227
• CPC Class Codes: H05K 999/99 : dummy group

Abstract

A security platform employs a variety techniques and mechanisms to detect security related anomalies and threats in a computer network environment. The security platform is “big data” driven and employs machine learning to perform security analytics. The security platform performs user/entity behavioral analytics (UEBA) to detect the security related anomalies and threats, regardless of whether such anomalies/threats were previously known. The security platform can include both real-time and batch paths/modes for detecting anomalies and threats. By visually presenting analytical results scored with risk ratings and supporting evidence, the security platform enables network security administrators to respond to a detected anomaly or threat, and to take action promptly.

Claims

1. A method comprising:

detecting, by a computer system, anomalies in activity on a computer network, based on received event data associated with the activity;

generating, by the computer system, anomaly data indicative of the anomalies in response to said detecting;

identifying, by the computer system, threat indicators by processing the anomaly data;

generating, by the computer system, threat indicator data indicative of the threat indicators in response to said identifying; and

identifying, by the computer system, the security threat to the computer network by processing the threat indicator data.

2. The method of claim 1, wherein said processing anomaly data includes one or more of:

aggregating the anomaly data;

correlating the anomaly data;

or enhancing the anomaly data.

3. The method of claim 1, wherein said processing the anomaly data includes:

determining a measure of anomalies associated with a particular entity of the computer network over a time period, the particular entity including a user or a device;

wherein said identifying threat indicators includes identifying a threat indicator if the measure of anomalies associated with the particular entity satisfies a specified criterion.

4. The method of claim 1, wherein said processing the anomaly data includes:

determining a number of entities of the computer network associated with a particular category of anomaly over a time period, the entities including users and/or devices;

wherein said identifying threat indicators includes identifying a threat indicator if the number of entities associated with the particular category of anomaly satisfies a specified criterion.

5. The method of claim 1, wherein said processing the anomaly data includes:

monitoring a duration of a particular anomaly over a time period; and

wherein said identifying threat indicators includes identifying a threat indicator if the duration of the particular anomaly satisfies a specified criterion.

6. The method of claim 1, wherein processing the anomaly data includes:

determining anomalies that have substantially matching profiles over a time period, the profiles based on the underlying event data associated with each anomaly; and

wherein said identifying threat indicators includes identifying a threat indicator if the number of anomalies with substantially matching profiles satisfies a specified criterion.

7. The method of claim 1, wherein said identifying threat indicators includes identifying that a pattern of detected anomalies satisfies both a local rarity criterion and a global rarity criterion.

8. The method of claim 1, wherein processing the anomaly data includes:

performing a global rarity analysis across a subset of the anomaly data over a time period, each of the detected anomalies associated with the subset of the anomaly data having been detected using a local rarity analysis;

wherein said identifying threat indicators includes identifying a threat indicator if a pattern in the detected anomalies satisfies a global rarity criterion.

9. The method of claim 1, wherein identifying threat indicators includes identifying a threat indicator if a combination of different types of anomalies satisfies a specified scoring criterion.

10. The method of claim 1, wherein processing the anomaly data includes:

combining anomaly data associated with different types of anomalies; and

assigning a threat indicator score based on a result of the combining;

wherein said identifying threat indicators includes identifying a threat indicator if the threat indicator score satisfies a specified scoring criterion.

11. The method of claim 1, wherein processing the anomaly data to identify threat indicators includes:

inputting anomaly data associated with a detected first type of anomaly into a model configured to detect a second type of anomaly;

wherein said identifying threat indicators includes identifying a threat indicator if the second type of anomaly is detected based on processing the anomaly data associated with the first type of anomaly according to the model.

12. The method of claim 1, wherein said processing the anomaly data includes:

identifying a particular entity associated with the anomaly data; and

comparing the particular entity against data stored in an external database of known security risks;

wherein said identifying threat indicators includes identifying a threat indicator if the particular entity substantially matches a known security risk from the database.

13. The method of claim 1, wherein said processing the anomaly data includes annotating the anomaly data with data from an external database of known security risks.

14. The method of claim 1, wherein said processing the anomaly data includes:

identifying an anomaly associated with a connection to a domain considered to be unfamiliar; and

in response to identifying the anomaly associated with the connection to the domain considered to be unfamiliar, determining whether the domain matches a domain known to be a security risk;

wherein said identifying threat indicators includes identifying a threat indicator if the domain substantially matches a domain known to be a security risk.

15. The method of claim 1, wherein said processing the anomaly data includes:

identifying an anomaly associated with a beacon type; and

in response to identifying the anomaly associated with the beacon type, determining whether the beacon type matches a beacon type known to be a security risk;

wherein said identifying threat indicators includes identifying a threat indicator if the beacon type substantially matches a beacon type known to be a security risk.

16. The method of claim 1, wherein detecting anomalies in activity on the computer network includes:

receiving event data indicative of activity by a particular entity associated with the computer network;

processing the event data through an anomaly model, the anomaly model including;

model processing logic defining a process for assigning an anomaly score to the event data; and

a model state defining a set of parameters for applying the model processing logic;

assigning an anomaly score based on the processing of the event data; and

outputting an indicator of the particular anomaly if the anomaly score satisfies a specified scoring criterion.

17. The method of claim 1, wherein said processing anomaly data includes:

processing the anomaly data through a threat indicator model, the threat indicator model including;

model processing logic defining a process for assigning a threat indicator score to the anomaly data; and

a model state defining a set of parameters for applying the model processing logic; and

assigning a threat indicator score based on the processing of the anomaly data;

wherein said identifying threat indicators includes identifying a threat indicator if the threat indicator score satisfies a specified scoring criterion.

18. The method of claim 1, wherein said processing the threat indicator data includes:

correlating a subset of the threat indicator data against a plurality of pre-defined security scenarios;

generating a set of candidate security threats based on the correlation; and

for each of the candidate threats;

comparing the subset of the threat indicator data against pre-configured patterns associated with the candidate threat; and

generating a pattern matching score based on a result of the comparing;

wherein said identifying the security threat to the computer network includes identifying a security threat if the pattern matching score satisfies a specified scoring criterion.

19. The method of claim 1, further comprising:

outputting, via a user interface, the identified threat indicators to a user; and

providing an alert to the user, via the user interface, in response to identifying the security threat.

20. The method of claim 1, further comprising:

outputting, via the user interface, the identified threat indicators as a summary of key anomalies from a set of detected anomalies across the computer network.

21. The method of claim 1, further comprising:

computing, by a processor, a score for a particular identified threat indicator, the score based on the anomaly data on which the particular threat indicator is based;

wherein the score represents a confidence level that the particular threat indictor is indicative of a security threat to the computer network.

22. The method of claim 1, further comprising:

incorporating the threat indicator data into a network security graph, the network security graph including;

a plurality of vertices representing entities associated with the computer network, the entities including users and/or devices; and

a plurality of edges, each of the plurality of edges linking two of the plurality of vertices and representing an association between the entities represented by the vertices;

wherein each threat indicator defined in the threat indicator data is incorporated as a vertex linked to one or more of the entities by one or more edges.

23. The method of claim 1, wherein the processing of event data is performed using a processing engine running Apache Storm.

24. The method of claim 1, wherein the processing of anomaly data and/or threat indicator data is performed in batch mode.

25. The method of claim 1, wherein the processing of anomaly data and/or threat indicator data is performed using a processing engine running Apache Spark on a Hadoop distributed computing cluster.

26. The method of claim 1, further comprising:

receiving the event data from a plurality of entities associated with the computer network via an extract, transform, and load (ETL) pipeline.

27. The method of claim 1, wherein the event data is machine-generated data.

28. The method of claim 1, further comprising storing the anomaly data in a data structure as an anomaly graph that includes:

a plurality of vertices representing entities associated with the computer network, the entities including users and/or devices; and

a plurality of edges, each of the plurality of edges representing an anomaly linking two of the plurality of vertices.

29. A system comprising:

a processor; and

a memory unit having instructions stored thereon, which when executed by the processor cause the system to;

detect anomalies in activity on a computer network, based on received event data associated with the activity;

generate anomaly data indicative of the anomalies in response to said detecting;

identify threat indicators by processing the anomaly data;

generate threat indicator data indicative of the threat indicators in response to said identifying; and

identify the security threat to the computer network by processing the threat indicator data.

30. A non-transient computer readable medium containing instructions for causing a computer system to:

detect anomalies in activity on a computer network, based on received event data associated with the activity;

generate anomaly data indicative of the anomalies in response to said detecting;

identify threat indicators by processing the anomaly data;

generate threat indicator data indicative of the threat indicators in response to said identifying; and

identify a security threat to the computer network by processing the threat indicator data.