Network security threat detection using shared variable behavior baseline

• US 10,135,848 B2
• Filed: 01/23/2017
• Issued: 11/20/2018
• Est. Priority Date: 08/31/2015
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Muddu, Sudhakar, Tryfonas, Christos
• Primary Examiner(s): Shiferaw, Eleni A
• Assistant Examiner(s): Holmes, Angela R
• Application Number:
  US15/413,336
• Publication Number:
  US 20170134415A1
• Time in Patent Office: 666 Days
• Field of Search: 726 23
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/444 Spatial browsing, e.g. 2D maps, 3D or virtual spaces
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 40/134 Hyperlinking
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06V 10/225 based on a marking or identifier characterising the area
• CPC Class Codes: H04L 2463/121 Timestamp cryptographic mechanisms or cryptographic arrangements involving time stamps H04L9/3297
• CPC Class Codes: H04L 41/0893 Assignment of logical groups to network elements
• CPC Class Codes: H04L 41/145 involving simulating, designing, planning or modelling of a network
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 43/00 : Arrangements for monitoring or testing data switching networks
• CPC Class Codes: H04L 43/045 : for graphical visualisation of monitoring data
• CPC Class Codes: H04L 43/062 : related to network traffic
• CPC Class Codes: H04L 43/08 : Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 63/06 : for supporting key management in a packet data network cryptographic mechanisms or cryptographic arrangements for key management H04L9/08
• CPC Class Codes: H04L 63/1408 : by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/1416 : Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 : Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1433 : Vulnerability analysis
• CPC Class Codes: H04L 63/1441 : Countermeasures against malicious traffic countermeasures against attacks on cryptographic mechanisms H04L9/002
• CPC Class Codes: H04L 63/20 : for managing network security; network security policies in general filtering policies H04L63/0227
• CPC Class Codes: H05K 999/99 : dummy group

Abstract

A security platform employs a variety techniques and mechanisms to detect security related anomalies and threats in a computer network environment. The security platform is “big data” driven and employs machine learning to perform security analytics. The security platform performs user/entity behavioral analytics (UEBA) to detect the security related anomalies and threats, regardless of whether such anomalies/threats were previously known. The security platform can include both real-time and batch paths/modes for detecting anomalies and threats. By visually presenting analytical results scored with risk ratings and supporting evidence, the security platform enables network security administrators to respond to a detected anomaly or threat, and to take action promptly.

Claims

1. A method comprising:

receiving, at a computer system, first event data indicative of computer network activity of an entity that is part of or has interacted with a computer network, the computer system including a real-time event processing engine and a batch event processing engine;

constructing a variable behavior baseline of the entity by using one of the real-time event processing engine or the batch event processing engine in the computer system, based on the first event data, the variable behavior baseline being representative of computer network activity of a particular type by the entity;

sharing the variable behavior baseline between the real-time event processing engine and the batch event processing engine;

receiving, at the computer system, second event data indicative of additional computer network activity associated with the entity;

comparing the second event data to the shared variable behavior baseline of the entity, by the one of the real-time event processing engine and the batch event processing engine that was not used to construct the variable behavior baseline of the entity; and

detecting at least one of a network security threat or a network security anomaly, when said comparing results in a determination that the second event data has a specified relationship to the shared variable behavior baseline of the entity.

2. A method as recited in claim 1, wherein the first event data comprise machine data.

3. A method as recited in claim 1, wherein the first event data comprise timestamped machine data.

4. A method as recited in claim 1, wherein the first event data comprise data indicative of performance or operation of an element of an information technology environment.

5. A method as recited in claim 1, wherein the entity is a device on the computer network.

6. A method as recited in claim 1, wherein the entity is a user of a device on the computer network.

7. A method as recited in claim 1, wherein constructing the first behavior baseline of the entity comprises using a machine learning process to construct the first behavior baseline of the entity.

8. A method as recited in claim 1, further comprising:

adjusting, by a first automated process, the variable behavior baseline of the entity based on the first event data.

9. A method as recited in claim 1, further comprising:

adjusting the variable behavior baseline of the entity based on a machine learning model.

10. A method as recited in claim 1, wherein the first event data comprises event data representing a plurality of events on the computer network over a period of time, each of the events being associated with network activity of the entity, and wherein constructing the variable behavioral baseline of the entity comprises constructing the variable behavioral baseline of the entity based on the first event data event data representing the plurality of events.

11. A method as recited in claim 1, further comprising:

generating anomaly data indicative that the additional computer network activity associated with the entity represents a network security anomaly;

detecting, by a threat decision engine, that the additional computer network activity associated with the entity represents a network security threat, based on the anomaly data; and

outputting, to a user, an indication that the additional computer network activity associated with the entity represents a network security threat.

12. A method as recited in claim 1, wherein said constructing the variable behavior baseline is performed in real time as the first event data are received.

13. A method as recited in claim 1, the first event data being stored in a persistent storage facility, wherein said constructing the variable behavior baseline is performed in a batch processing mode based on the stored first event data.

14. A method as recited in claim 1,

said comparing comprises comparing, in real-time by the real-time event processing engine, the second event data to the shared variable behavior baseline of the entity;

and

said detecting comprises determining, in real-time by the real-time event processing engine, that the additional computer network activity associated with the entity represents a network security anomaly or a network security threat, when said comparing results in a determination that the second event data has a specified relationship to the shared variable behavior baseline of the entity.

15. A method as recited in claim 1, further comprising:

storing the second event data in a persistent storage facility;

wherein said comparing comprises comparing, by the batch event processing engine, the stored second event data to the shared variable behavior baseline of the entity; and

wherein said determining comprises determining, by the batch event processing engine, that the additional computer network activity associated with the entity represents a network security anomaly or a network security threat, when said comparing results in a determination that the second event data has a specified relationship to the shared variable behavior baseline of the entity.

16. A method as recited in claim 1, wherein said constructing the variable behavior baseline is performed in a batch mode by the batch event processing engine,

said comparing comprises comparing, in real-time by the real-time event processing engine, the second event data to the shared variable behavior baseline of the entity;

and

said determining comprises determining, in real-time by the real-time event processing engine, that the additional computer network activity associated with the entity represents a network security anomaly or a network security threat, when said comparing results in a determination that the second event data has a specified relationship to the shared variable behavior baseline of the entity.

17. A method as recited in claim 1, wherein said constructing the variable behavior baseline is performed in a real-time mode by the real-time event processing engine;

wherein said comparing comprises comparing, in a batch mode by the batch event processing engine, the second event data to the shared variable behavior baseline of the entity; and

wherein said determining comprises determining, in a batch mode by the batch event processing engine, that the additional computer network activity associated with the entity represents a network security anomaly or a network security threat, when said comparing results in a determination that the second event data has a specified relationship to the shared variable behavior baseline of the entity.

18. A method as recited in claim 1, wherein the variable behavioral baseline is a first variable behavioral baseline of the entity, further comprising:

constructing a second variable behavior baseline of the entity, based on the second event data, the second variable behavior baseline being representative of the computer network activity of the entity;

receiving, at the computer system, third event data resulting from additional computer network activity associated with the entity;

comparing, by the computer system, the third event data to the first variable behavior baseline and the second variable behavior baseline; and

determining, by an automated process in the computer system, that the additional computer network activity associated with the entity represents a network security anomaly or a network security threat, in response to a determination that the third event data has a specified relationship to at least one of the first variable behavior baseline or the second variable behavior baseline.

19. A method as recited in claim 1, wherein:

the first event data and the second event data comprise time stamped data indicative of performance or operation of an element of an information technology environment.

20. A method as recited in claim 1, wherein the batch event processing engine is an implementation of Apache Spark.

21. A method as recited in claim 1, wherein the real-time event processing engine is an implementation of Apache Storm.

22. A method as recited in claim 1, wherein the real-time event processing engine is an implementation of Apache Spark Streaming.

23. A method as recited in claim 1, wherein the batch event processing engine is an implementation of Apache Spark, and the real-time event processing engine is an implementation of Apache Storm or Apache Spark Streaming.

24. A computer system comprising:

a processor; and

a communication device, operatively coupled to the processor, through which to receive first event data indicative of computer network activity of an entity that is part of or interacts with a computer network;

a real-time event processing engine;

a batch event processing engine;

wherein the processor is configured to

construct a variable behavior baseline of the entity by using one of the real-time event processing engine or the batch event processing engine in the computer system, based on the first event data, the variable behavior baseline being representative of computer network activity of a particular type by the entity;

share the variable behavior baseline between the real-time event processing engine and the batch event processing engine;

receive second event data indicative of additional computer network activity associated with the entity;

compare the second event data to the shared variable behavior baseline of the entity, by the one of the real-time event processing engine and the batch event processing engine that was not used to construct the variable behavior baseline of the entity; and

detect at least one of a network security threat or a network security anomaly, when comparison of the second event data to the shared variable behavior baseline of the entity results in a determination that the second event data has a specified relationship to the shared variable behavior baseline of the entity.

25. A non-transitory machine-readable storage medium for use in a processing system, the non-transitory machine-readable storage medium storing instructions, an execution of which in the processing system causes the processing system to perform operations comprising:

receiving first event data indicative of computer network activity of an entity that is part of or has interacted with a computer network, the computer system including a real-time event processing engine and a batch event processing engine;

constructing a variable behavior baseline of the entity by using one of the real-time event processing engine or the batch event processing engine in the computer system, based on the first event data, the variable behavior baseline being representative of computer network activity of a particular type by the entity;

sharing the variable behavior baseline between the real-time event processing engine and the batch event processing engine;

receiving, at the processing system, second event data indicative of additional computer network activity associated with the entity;

comparing the second event data to the shared variable behavior baseline of the entity, by the one of the real-time event processing engine and the batch event processing engine that was not used to construct the variable behavior baseline of the entity; and

detecting at least one of a network security threat or a network security anomaly, when said comparing results in a determination that the second event data has a specified relationship to the shared variable behavior baseline of the entity.