Detection of anomalies, threat indicators, and threats to network security

• US 10,419,450 B2
• Filed: 10/30/2015
• Issued: 09/17/2019
• Est. Priority Date: 08/31/2015
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Muddu, Sudhakar, Tryfonas, Christos
• Primary Examiner(s): Dada, Beemnet W
• Application Number:
  US14/929,037
• Publication Number:
  US 20170063905A1
• Time in Patent Office: 1,418 Days
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/444 Spatial browsing, e.g. 2D maps, 3D or virtual spaces
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 40/134 Hyperlinking
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06V 10/225 based on a marking or identifier characterising the area
• CPC Class Codes: H04L 2463/121 Timestamp cryptographic mechanisms or cryptographic arrangements involving time stamps H04L9/3297
• CPC Class Codes: H04L 41/0893 Assignment of logical groups to network elements
• CPC Class Codes: H04L 41/145 involving simulating, designing, planning or modelling of a network
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 43/00 : Arrangements for monitoring or testing data switching networks
• CPC Class Codes: H04L 43/045 : for graphical visualisation of monitoring data
• CPC Class Codes: H04L 43/062 : related to network traffic
• CPC Class Codes: H04L 43/08 : Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 63/06 : for supporting key management in a packet data network cryptographic mechanisms or cryptographic arrangements for key management H04L9/08
• CPC Class Codes: H04L 63/1408 : by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/1416 : Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 : Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1433 : Vulnerability analysis
• CPC Class Codes: H04L 63/1441 : Countermeasures against malicious traffic countermeasures against attacks on cryptographic mechanisms H04L9/002
• CPC Class Codes: H04L 63/20 : for managing network security; network security policies in general filtering policies H04L63/0227
• CPC Class Codes: H05K 999/99 : dummy group

Abstract

A security platform employs a variety techniques and mechanisms to detect security related anomalies and threats in a computer network environment. The security platform is “big data” driven and employs machine learning to perform security analytics. The security platform performs user/entity behavioral analytics (UEBA) to detect the security related anomalies and threats, regardless of whether such anomalies/threats were previously known. The security platform can include both real-time and batch paths/modes for detecting anomalies and threats. By visually presenting analytical results scored with risk ratings and supporting evidence, the security platform enables network security administrators to respond to a detected anomaly or threat, and to take action promptly.

Claims

1. A method comprising:

detecting, by a computer system, anomalies in activity on a computer network, by processing received event data associated with the activity using a plurality of machine-learning anomaly models, each of the plurality of machine-learning anomaly models configured to detect a different one of a plurality of categories of anomalous activity on the computer network;

generating, by the computer system, anomaly data indicative of the anomalies in response to said detecting;

identifying, by the computer system, threat indicators by processing the anomaly data using a plurality of machine-learning threat indicator models, each of the plurality of machine-learning threat indicator models configured to analyze relationships between anomalies detected across the computer network;

generating, by the computer system, threat indicator data indicative of the threat indicators in response to said identifying; and

identifying, by the computer system, a security threat to the computer network by processing the threat indicator data using a plurality of machine-learning security threat models, each of the plurality of machine-learning security threat models configured to analyze relationships between threat indicators identified across the computer network.

2. The method of claim 1, wherein said processing anomaly data includes one or more of:

aggregating the anomaly data;

correlating the anomaly data;

or enhancing the anomaly data.

3. The method of claim 1, wherein said processing the anomaly data includes:

determining a measure of anomalies associated with a particular entity of the computer network over a time period, the particular entity including a user or a device;

wherein said identifying threat indicators includes identifying a threat indicator if the measure of anomalies associated with the particular entity satisfies a specified criterion.

4. The method of claim 1, wherein said processing the anomaly data includes:

determining a number of entities of the computer network associated with a particular category of anomaly over a time period, the entities including users and/or devices;

wherein said identifying threat indicators includes identifying a threat indicator if the number of entities associated with the particular category of anomaly satisfies a specified criterion.

5. The method of claim 1, wherein said processing the anomaly data includes:

monitoring a duration of a particular anomaly over a time period; and

wherein said identifying threat indicators includes identifying a threat indicator if the duration of the particular anomaly satisfies a specified criterion.

6. The method of claim 1, wherein processing the anomaly data includes:

determining anomalies that have substantially matching profiles over a time period, the profiles based on the underlying event data associated with each anomaly; and

wherein said identifying threat indicators includes identifying a threat indicator if the number of anomalies with substantially matching profiles satisfies a specified criterion.

7. The method of claim 1, wherein processing the anomaly data to identify threat indicators includes:

inputting anomaly data associated with a detected first type of anomaly into a model configured to detect a second type of anomaly;

wherein said identifying threat indicators includes identifying a threat indicator if the second type of anomaly is detected based on processing the anomaly data associated with the first type of anomaly according to the model.

8. The method of claim 1, wherein said processing the anomaly data includes:

identifying a particular entity associated with the anomaly data; and

comparing the particular entity against data stored in an external database of known security risks;

wherein said identifying threat indicators includes identifying a threat indicator if the particular entity substantially matches a known security risk from the database.

9. The method of claim 1, wherein said processing the anomaly data includes annotating the anomaly data with data from an external database of known security risks.

10. The method of claim 1, wherein said processing the anomaly data includes:

identifying an anomaly associated with a connection to a domain considered to be unfamiliar; and

in response to identifying the anomaly associated with the connection to the domain considered to be unfamiliar, determining whether the domain matches a domain known to be a security risk;

wherein said identifying threat indicators includes identifying a threat indicator if the domain substantially matches a domain known to be a security risk.

11. The method of claim 1, wherein said processing the anomaly data includes:

identifying an anomaly associated with a beacon type; and

in response to identifying the anomaly associated with the beacon type, determining whether the beacon type matches a beacon type known to be a security risk;

wherein said identifying threat indicators includes identifying a threat indicator if the beacon type substantially matches a beacon type known to be a security risk.

12. The method of claim 1, further comprising:

outputting, via a user interface, the identified threat indicators to a user; and

providing an alert to the user, via the user interface, in response to identifying the security threat.

13. The method of claim 1, further comprising:

outputting, via the user interface, the identified threat indicators as a summary of key anomalies from a set of detected anomalies across the computer network.

14. The method of claim 1, further comprising:

computing, by a processor, a score for a particular identified threat indicator, the score based on the anomaly data on which the particular threat indicator is based;

wherein the score represents a confidence level that the particular threat indictor is indicative of a security threat to the computer network.

15. The method of claim 1, wherein the processing of event data is performed using a processing engine running Apache Storm.

16. The method of claim 1, wherein the processing of anomaly data and/or threat indicator data is performed in batch mode.

17. The method of claim 1, wherein the processing of anomaly data and/or threat indicator data is performed using a processing engine running Apache Spark on a Hadoop distributed computing cluster.

18. The method of claim 1, further comprising:

receiving the event data from a plurality of entities associated with the computer network via an extract, transform, and load (ETL) pipeline.

19. The method of claim 1, wherein the event data is machine-generated data.

20. A system comprising:

a processor; and

a memory unit having instructions stored thereon, which when executed by the processor cause the system to;

detect anomalies in activity on a computer network, by processing received event data associated with the activity using a plurality of machine-learning anomaly models, each of the plurality of machine-learning anomaly models configured to detect a different one of a plurality of categories of anomalous activity on the computer network;

generate anomaly data indicative of the anomalies in response to said detecting;

identify threat indicators by processing the anomaly data using a plurality of machine-learning threat indicator models, each of the plurality of machine-learning threat indicator models configured to analyze relationships between anomalies detected across the computer network;

generate threat indicator data indicative of the threat indicators in response to said identifying; and

identify a security threat to the computer network by processing the threat indicator data using a plurality of machine-learning security threat models, each of the plurality of machine-learning security threat models configured to analyze relationships between threat indicators identified across the computer network.

21. A non-transient computer readable medium containing instructions for causing a computer system to:

detect anomalies in activity on a computer network by processing received event data associated with the activity using a plurality of machine-learning anomaly models, each of the plurality of machine-learning anomaly models configured to detect a different one of a plurality of categories of anomalous activity on the computer network;

generate anomaly data indicative of the anomalies in response to said detecting;

identify threat indicators by processing the anomaly data using a plurality of machine-learning threat indicator models, each of the plurality of machine-learning threat indicator models configured to analyze relationships between anomalies detected across the computer network;

generate threat indicator data indicative of the threat indicators in response to said identifying; and

identify a security threat to the computer network by processing the threat indicator data using a plurality of machine-learning security threat models, each of the plurality of machine-learning security threat models configured to analyze relationships between threat indicators identified across the computer network.