Network security analysis using real-time and batch detection engines

• US 9,813,435 B2
• Filed: 01/25/2017
• Issued: 11/07/2017
• Est. Priority Date: 08/31/2015
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Muddu, Sudhakar, Tryfonas, Christos, Bulusu, Ravi Prasad
• Primary Examiner(s): Schwartz, Darren B
• Application Number:
  US15/415,747
• Publication Number:
  US 20170134410A1
• Time in Patent Office: 286 Days
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/444 Spatial browsing, e.g. 2D maps, 3D or virtual spaces
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 40/134 Hyperlinking
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06V 10/225 based on a marking or identifier characterising the area
• CPC Class Codes: H04L 2463/121 Timestamp cryptographic mechanisms or cryptographic arrangements involving time stamps H04L9/3297
• CPC Class Codes: H04L 41/0893 Assignment of logical groups to network elements
• CPC Class Codes: H04L 41/145 involving simulating, designing, planning or modelling of a network
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 43/00 : Arrangements for monitoring or testing data switching networks
• CPC Class Codes: H04L 43/045 : for graphical visualisation of monitoring data
• CPC Class Codes: H04L 43/062 : related to network traffic
• CPC Class Codes: H04L 43/08 : Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 63/06 : for supporting key management in a packet data network cryptographic mechanisms or cryptographic arrangements for key management H04L9/08
• CPC Class Codes: H04L 63/1408 : by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/1416 : Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 : Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1433 : Vulnerability analysis
• CPC Class Codes: H04L 63/1441 : Countermeasures against malicious traffic countermeasures against attacks on cryptographic mechanisms H04L9/002
• CPC Class Codes: H04L 63/20 : for managing network security; network security policies in general filtering policies H04L63/0227
• CPC Class Codes: H05K 999/99 : dummy group

Abstract

A security platform employs a variety techniques and mechanisms to detect security related anomalies and threats in a computer network environment. The security platform is “big data” driven and employs machine learning to perform security analytics. The security platform performs user/entity behavioral analytics (UEBA) to detect the security related anomalies and threats, regardless of whether such anomalies/threats were previously known. The security platform can include both real-time and batch paths/modes for detecting anomalies and threats. By visually presenting analytical results scored with risk ratings and supporting evidence, the security platform enables network security administrators to respond to a detected anomaly or threat, and to take action promptly.

Claims

1. A network security breach detection system comprising:

a real-time path including a real-time analysis engine configured to receive first event data indicative of first activity on a computer network, the real-time event analysis engine configured to detect first indicia of possible security breaches in a real-time processing mode based on the first event data, and to generate real-time analysis result data representing the first indicia for output to a user;

a non-volatile storage system to store the real-time analysis result data; and

a batch path including a batch analysis engine configured to operate concurrently with the real-time analysis engine, the batch analysis engine further configured to retrieve, from the non-volatile storage system, the real-time analysis result data and second event data indicative of second activity on the computer network, the first event data and the second event data each including timestamped machine data indicative of performance or operation of a component in an information technology environment, the second event data having been stored in the non-volatile storage system prior to analysis of the first event data by the real-time analysis engine, the batch analysis engine further configured to detect, in a batch mode, second indicia of possible security breaches based on the second event data and the real-time analysis result data.

2. The network security breach detection system as recited in claim 1, wherein the real-time analysis engine is further configured to use outputs of the batch analysis engine, in conjunction with the first event data, to detect the first indicia of possible security breaches.

3. The network security breach detection system as recited in claim 1, wherein the first event data is a portion of an unbounded stream of event data.

4. The network security breach detection system as recited in claim 1, wherein the real-time path further includes a data intake and preparation engine configured to receive the first event data and the second event data from a plurality of heterogeneous data sources in the computer network, and to perform preprocessing of the first event data and the second event data before the first event data and the second event data are provided to the real-time analysis engine and the batch analysis engine, respectively, wherein the preprocessing includes at least one of:

parsing the first and second event data, enriching the first and second data, and filtering the first and second event data.

5. The network security breach detection system as recited in claim 1, wherein the real-time path further includes a message broker to receive the first event data and the second event data and to pass the first event data to the real-time analysis engine and to pass the second event data to the batch analysis engine.

6. The network security breach detection system as recited in claim 1, wherein the real-time path further includes:

a data intake and preparation engine configured to receive the first event data and the second event data from a plurality of heterogeneous data sources in the computer network, and to perform preprocessing of the first event data and the second event data before the first event data and the second event data are provided to the real-time analysis engine and the batch analysis engine, respectively; and

a message broker to receive the preprocessed first event data and second event data from the data intake and preparation engine and to pass the preprocessed first event data to the real-time analysis engine and to pass the preprocessed second event data to the batch analysis engine.

7. The network security breach detection system as recited in claim 1, wherein the real-time analysis engine includes a real-time anomaly detection engine and a real-time threat detection engine, wherein some of the first indicia of possible security breaches are detected by the real-time anomaly detection engine as security related anomalies and others of the first indicia are detected by the real-time threat detection engine as security related threats based on the detected anomalies.

8. The network security breach detection system as recited in claim 1, wherein the real-time analysis engine includes a real-time anomaly detection engine and a real-time threat detection engine, wherein some of the first indicia of possible security breaches are detected by the real-time anomaly detection engine as security related anomalies and others of the first indicia are detected by the real-time threat detection engine as security related threats based on the detected anomalies;

and

wherein the batch analysis engine includes a batch anomaly detection engine and a batch threat detection engine, wherein some of the second indicia of possible security breaches are detected by the batch anomaly detection engine as security related anomalies and others of the second indicia are detected by the batch threat detection engine as security related threats based on the anomalies detected by the batch anomaly detection engine.

9. The network security breach detection system as recited in claim 1, wherein the second event data includes a larger amount of data than the first event data and has been generated over a longer time period than the first event data.

10. The network security breach detection system as recited in claim 1, wherein:

the real-time analysis engine executes a first plurality of versions of a plurality of machine learning models to detect the first indicia of possible security breaches in the real-time mode; and

the batch analysis engine executes a second plurality of versions of said plurality of machine learning models to detect the second indicia of possible security breaches in the batch mode.

11. The network security breach detection system as recited in claim 1, wherein:

the real-time analysis engine and the batch analysis engine collectively execute a plurality of machine learning models to detect the first indicia and second indicia of possible security breaches; and

the real-time analysis engine and the batch analysis engine share a model state of a particular machine learning model of the plurality of machine learning models.

12. The network security breach detection system as recited in claim 1, wherein:

the real-time analysis engine and the batch analysis engine collectively execute a plurality of machine learning models to detect the first indicia and second indicia of possible security breaches; and

a result from the batch analysis engine is used to update a model state of a machine learning model used by the real-time analysis engine.

13. The network security breach detection system as recited in claim 1, wherein:

the real-time analysis engine and the batch analysis engine collectively execute a plurality of machine learning models to detect the first indicia and second indicia of possible security breaches; and

a result from the real-time analysis engine is used to update a model state of a machine learning model used by the batch analysis engine.

14. The network security breach detection system as recited in claim 1, wherein:

the real-time analysis engine and the batch analysis engine collectively execute a plurality of machine learning models to detect the first indicia and second indicia of possible security breaches;

a result from the batch analysis engine is used to update a model state of a machine learning model used by the real-time analysis engine; and

a result from the real-time analysis engine is used to update a model state of a machine learning model used by the batch analysis engine.

15. The network security breach detection system as recited in claim 1, wherein the real-time analysis engine is implemented on a first distributed data processing platform, the batch analysis engine is implemented on a second distributed processing platform, and the non-volatile storage system is a distributed storage system.

16. The network security breach detection system as recited in claim 1, wherein the real-time analysis engine is implemented on Apache Storm, the batch analysis engine is implemented on Apache Spark, and the non-volatile storage system is implemented using Hadoop Distributed File System.

17. The network security breach detection system as recited in claim 1, wherein:

the first event data and the second event data include timestamped machine data;

the real-time analysis engine is implemented on a first distributed processing platform and includes a real-time anomaly detection engine and a real-time threat detection engine, wherein some of the first indicia of possible security breaches are detected by the real-time anomaly detection engine as security related anomalies and others of the first indicia are detected by the real-time threat detection engine as security related threats based on the detected anomalies;

the batch analysis engine is implemented on a second distributed processing platform and includes a batch anomaly detection engine and a batch threat detection engine, wherein some of the second indicia of possible security breaches are detected by the batch anomaly detection engine as security related anomalies and others of the second indicia are detected by the batch threat detection engine as security related threats based on the anomalies detected by the batch anomaly detection engine;

the real-time analysis engine and the batch analysis engine collectively execute a plurality of machine learning models to detect, respectively, the first indicia and the second indicia of possible security breaches; and

a result from the batch analysis engine is used to update a model state of a machine learning model used by the real-time analysis engine.

18. The network security breach detection system as recited in claim 1, wherein:

the first event data and the second event data include timestamped machine data;

the real-time path further includes;

a data intake and preparation engine configured to receive the first event data and the second event data from a plurality of heterogeneous data sources in the computer network, and to perform preprocessing of the first event data and the second event data before the first event data and the second event data are provided to the real-time analysis engine and the batch analysis engine, respectively; and

a message broker to receive the preprocessed first event data and second event data from the data intake and preparation engine and to pass the preprocessed first event data to the real-time analysis engine and to pass the preprocessed second event data to the batch analysis engine;

the real-time analysis engine is implemented on a first distributed processing platform and includes a real-time anomaly detection engine and a real-time threat detection engine, wherein some of the first indicia of possible security breaches are detected by the real-time anomaly detection engine as security related anomalies and others of the first indicia are detected by the real-time threat detection engine as security related threats based on the detected anomalies;

the batch analysis engine is implemented on a second distributed processing platform and includes a batch anomaly detection engine and a batch threat detection engine, wherein some of the second indicia of possible security breaches are detected by the batch anomaly detection engine as security related anomalies and others of the second indicia are detected by the batch threat detection engine as security related threats based on the anomalies detected by the batch anomaly detection engine;

the real-time analysis engine and the batch analysis engine collectively execute a plurality of machine learning models to detect the first indicia and second indicia, respectively, of possible security breaches; and

a result from the batch analysis engine is used to update a model state of a machine learning model used by the real-time analysis engine.

19. A method comprising:

detecting, in a real-time processing mode, first indicia of possible security breaches based on first event data indicative of first activity on a computer network, by using a real-time analysis engine;

generating real-time analysis result data representing the first indicia for output to a user;

storing the real-time analysis result data in a non-volatile storage system;

retrieving, from the non-volatile storage system, the real-time analysis result data and second event data indicative of second activity on the computer network, the first event data and the second event data each including timestamped machine data indicative of performance or operation of a component in an information technology environment, the second event data having been stored in the non-volatile storage system prior to analysis of the first event data by the real-time analysis engine; and

detecting, in a batch mode, second indicia of possible security breaches based on the second event data and the real-time analysis result data, by using a batch analysis engine concurrently with use of the real-time analysis engine.

20. The method as recited in claim 19, further comprising:

using outputs of the batch analysis engine, in conjunction with the first event data, to detect the first indicia of possible security breaches.

21. The method as recited in claim 19, wherein using the real-time analysis engine includes using a real-time anomaly detection engine and a real-time threat detection engine, wherein some of the first indicia of possible security breaches are detected by the real-time anomaly detection engine as security related anomalies and others of the first indicia are detected by the real-time threat detection engine as security related threats based on the detected anomalies;

and

wherein using the batch analysis engine includes using a batch anomaly detection engine and a batch threat detection engine, wherein some of the second indicia of possible security breaches are detected by the batch anomaly detection engine as security related anomalies and others of the second indicia are detected by the batch threat detection engine as security related threats based on the anomalies detected by the batch anomaly detection engine.

22. The method as recited in claim 19, wherein:

using the real-time analysis engine includes executing a first plurality of versions of a plurality of machine learning models to detect the first indicia of possible security breaches in the real-time mode; and

using the batch analysis engine includes executing a second plurality of versions of said plurality of machine learning models to detect the second indicia of possible security breaches in the batch mode.

23. The method as recited in claim 19, wherein:

using the real-time analysis engine and using the batch analysis engine collectively include using a plurality of machine learning models to detect the first indicia and second indicia of possible security breaches; and

a result from the batch analysis engine is used to update a model state of a machine learning model used by the real-time analysis engine.

24. A non-transitory machine-readable storage medium storing instructions, execution of which in a computer system causes the computer system to perform operations comprising:

detecting, in a real-time processing mode, first indicia of possible security breaches based on first event data indicative of first activity on a computer network, by executing a real-time analysis engine;

generating real-time analysis result data representing the first indicia for output to a user;

storing the real-time analysis result data in a non-volatile storage system;

retrieving, from the non-volatile storage system, the real-time analysis result data and second event data indicative of second activity on the computer network, the first event data and the second event data each including timestamped machine data indicative of performance or operation of a component in an information technology environment, the second event data having been stored in the non-volatile storage system prior to analysis of the first event data by the real-time analysis engine; and

detecting, in a batch mode, second indicia of possible security breaches based on the second event data and the real-time analysis result data, by executing a batch analysis engine concurrently with the executing of the real-time analysis engine.

25. The non-transitory machine-readable storage medium as recited in claim 24, the operations further comprising:

using outputs of the batch analysis engine, in conjunction with the first event data, to detect the first indicia of possible security breaches.

26. The non-transitory machine-readable storage medium as recited in claim 24, such that:

using the real-time analysis engine includes using a real-time anomaly detection engine and a real-time threat detection engine, wherein some of the first indicia of possible security breaches are detected by the real-time anomaly detection engine as security related anomalies and others of the first indicia are detected by the real-time threat detection engine as security related threats based on the detected anomalies; and

using the batch analysis engine includes using a batch anomaly detection engine and a batch threat detection engine, wherein some of the second indicia of possible security breaches are detected by the batch anomaly detection engine as security related anomalies and others of the second indicia are detected by the batch threat detection engine as security related threats based on the anomalies detected by the batch anomaly detection engine.

27. The non-transitory machine-readable storage medium as recited in claim 24, such that:

using the real-time analysis engine includes executing a first plurality of versions of a plurality of machine learning models to detect the first indicia of possible security breaches in the real-time mode; and

using the batch analysis engine includes executing a second plurality of versions of said plurality of machine learning models to detect the second indicia of possible security breaches in the batch mode.

28. The non-transitory machine-readable storage medium as recited in claim 24, wherein:

using the real-time analysis engine and using the batch analysis engine collectively include using a plurality of machine learning models to detect the first indicia and second indicia of possible security breaches; and

a result from the batch analysis engine is used to update a model state of a machine learning model used by the real-time analysis engine.