Sharing model state between real-time and batch paths in network security anomaly detection

• US 10,158,652 B2
• Filed: 10/30/2015
• Issued: 12/18/2018
• Est. Priority Date: 08/31/2015
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Muddu, Sudhakar, Tryfonas, Christos, Bulusu, Ravi Prasad
• Primary Examiner(s): Hirl, Joseph P
• Assistant Examiner(s): GYORFI, THOMAS A
• Application Number:
  US14/929,141
• Publication Number:
  US 20170063908A1
• Time in Patent Office: 1,145 Days
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/444 Spatial browsing, e.g. 2D maps, 3D or virtual spaces
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 40/134 Hyperlinking
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06V 10/225 based on a marking or identifier characterising the area
• CPC Class Codes: H04L 2463/121 Timestamp cryptographic mechanisms or cryptographic arrangements involving time stamps H04L9/3297
• CPC Class Codes: H04L 41/0893 Assignment of logical groups to network elements
• CPC Class Codes: H04L 41/145 involving simulating, designing, planning or modelling of a network
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 43/00 : Arrangements for monitoring or testing data switching networks
• CPC Class Codes: H04L 43/045 : for graphical visualisation of monitoring data
• CPC Class Codes: H04L 43/062 : related to network traffic
• CPC Class Codes: H04L 43/08 : Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 63/06 : for supporting key management in a packet data network cryptographic mechanisms or cryptographic arrangements for key management H04L9/08
• CPC Class Codes: H04L 63/1408 : by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/1416 : Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 : Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1433 : Vulnerability analysis
• CPC Class Codes: H04L 63/1441 : Countermeasures against malicious traffic countermeasures against attacks on cryptographic mechanisms H04L9/002
• CPC Class Codes: H04L 63/20 : for managing network security; network security policies in general filtering policies H04L63/0227
• CPC Class Codes: H05K 999/99 : dummy group

Abstract

A security platform employs a variety techniques and mechanisms to detect security related anomalies and threats in a computer network environment. The security platform is “big data” driven and employs machine learning to perform security analytics. The security platform performs user/entity behavioral analytics (UEBA) to detect the security related anomalies and threats, regardless of whether such anomalies/threats were previously known. The security platform can include both real-time and batch paths/modes for detecting anomalies and threats. By visually presenting analytical results scored with risk ratings and supporting evidence, the security platform enables network security administrators to respond to a detected anomaly or threat, and to take action promptly.

Claims

1. A method comprising:

implementing a real-time event processing engine on a distributed data processing platform, wherein the real-time event processing engine is configured to process an unbounded stream of event data to detect a plurality of network security-related issues and/or to train a machine learning model;

implementing a batch event processing engine on the distributed data processing platform, wherein the batch event processing engine is configured to process a batch of historic event data to detect a plurality of network security-related issues and/or to train a machine learning model; and

enabling the real-time event processing engine and the batch event processing engine to share a model state of a particular machine learning model, the particular machine learning model being configured to process a time slice of data to produce a score for detecting a network security-related issue,

wherein the real-time event processing engine and the batch event processing engine each utilize the shared model state to share, with the other engine, network security-related knowledge gained from processing one'"'"'s respective data.

2. The method of claim 1, wherein a size of the time slice is controlled by whichever one of the real-time processing engine and the batch processing engine utilizes the particular machine learning model.

3. The method of claim 1, wherein the shared model state enables the batch event processing engine to use knowledge gained by the real-time event processing engine to discover a security-related issue in the historic event data.

4. The method of claim 1, wherein the shared model state enables the batch event processing engine to use knowledge gained by the real-time event processing engine to discover a security-related issue in the historic event data that is undetectable by the batch event processing engine without the knowledge.

5. The method of claim 1, wherein the shared model state enables the batch event processing engine to use new knowledge gained by the real-time event processing engine from processing the unbounded stream of event data, to inspect the historic event data to discover a security-related issue that would be undetectable by the batch event processing engine without the new knowledge,

wherein the new knowledge is knowledge gained after a last time the batch event processing engine performs historic event data inspection using the particular machine learning model.

6. The method of claim 1, further comprising:

performing, by the batch event processing engine, an analysis on the historic event data to detect a security-related issue.

7. The method of claim 1, further comprising:

performing, by the batch event processing engine, an analysis on the historic event data to detect a security-related issue, wherein the analysis includes at least one of;

a lateral movement anomaly analysis, a behavioral peer analysis, a label propagation analysis, or a time-series anomaly analysis.

8. The method of claim 1, further comprising:

performing, by the batch event processing engine, an analysis on the historic event data to detect a security-related issue, wherein the analysis results in acquisition of new knowledge of event data; and

updating, by the batch event processing engine, the shared model state of the particular machine learning model to incorporate the new knowledge;

and

performing, by the real-time event processing engine, an inspection of newly received event data based on the new knowledge, by using the shared model state.

9. The method of claim 1, further comprising:

locating, in the batch of historic event data, data representing a plurality of events associated with an entity; and

performing, by the batch event processing engine, a behavioral analysis of the entity to detect a behavioral anomaly.

10. The method of claim 1, further comprising:

locating, in the batch of historic event data, data representing a plurality of events that are associated with an entity; and

performing, by the batch event processing engine, a behavioral analysis of the entity to detect a behavioral anomaly, wherein said performing the behavioral analysis includes;

utilizing the particular machine learning model to compute a degree of behavioral deviation in behaviors within a given time slice as compared to a behavioral baseline specific to the entity; and

determining, based on the degree of behavioral deviation, whether the behavioral anomaly exists.

11. The method of claim 1, further comprising:

locating, in the batch of historic event data, data representing a plurality of events that are associated with an entity; and

performing, by the batch event processing engine, a behavioral analysis of the entity to detect a behavioral anomaly, wherein said performing behavioral analysis includes;

utilizing the particular machine learning model to compute a degree of behavioral deviation in behaviors within a given time slice as compared to a behavioral baseline specific to the entity; and

determining, based on the degree of behavioral deviation, whether the behavioral anomaly exists,

wherein the particular machine learning model is trained by the real-time event processing engine.

12. The method of claim 1, further comprising:

locating, in the batch of historic event data, data representing a plurality of events that are associated with an entity; and

performing, by the batch event processing engine, a behavioral analysis of the entity to detect a behavioral anomaly, wherein said performing behavioral analysis includes;

utilizing the particular machine learning model to compute a degree of behavioral deviation in behaviors within a given time slice as compared to a behavioral baseline specific to the entity; and

determining, based on the degree of behavioral deviation, whether the behavioral anomaly exists,

wherein the particular machine learning model is trained by the real-time event processing engine, and

wherein the behavioral baseline is established by the real-time event processing engine using the particular machine learning model.

13. The method of claim 1, further comprising:

locating, in the batch of historic event data, data representing a plurality of events that are associated with behaviors performed by a plurality of entities; and

performing, by the batch event processing engine, anomaly analysis on the behaviors performed by a plurality of entities to detect a particular security-related anomaly.

14. The method of claim 1, further comprising:

locating, in the batch of historic event data, data representing a plurality of events associated with behaviors performed by a plurality of entities; and

performing, by the batch event processing engine, anomaly analysis on the behaviors performed by a plurality of entities to detect a particular security-related anomaly,

wherein said anomaly analysis includes comparing a behavior of one entity to the behaviors of other entities in determining whether a behavioral deviation of the one entity represents the particular security-related anomaly.

15. The method of claim 1, further comprising:

locating, by the batch event processing engine, a composite relationship graph associated with the historic event data; and

obtaining a projection of the composite relationship graph, based on a requirement of the particular machine learning model.

16. The method of claim 1, further comprising:

locating, by the batch event processing engine, a composite relationship graph associated with the historic event data; and

obtaining a projection of the composite relationship graph, based on a requirement of the particular machine learning model,

wherein the composite relationship graph is based on information generated by the real-time event processing engine processing the unbounded stream of event data.

17. The method of claim 1, further comprising:

locating, by the batch event processing engine, a composite relationship graph associated with the historic event data; and

obtaining a projection of the composite relationship graph, based on a requirement of the particular machine learning model,

wherein the projection includes a graph of users associated with machines to facilitate tracking, by the particular machine learning model, of user lateral movement.

18. The method of claim 1, further comprising:

locating, by the batch event processing engine, a composite relationship graph associated with the historic event data; and

obtaining a projection of the composite relationship graph, based on a requirement of the particular machine learning model,

wherein the projection includes a graph that associates entities identified as having security-related issues to facilitate correlating, by the particular machine learning model, of user anomalies so as to identify sophisticated threats.

19. The method of claim 1, further comprising:

locating, by the batch event processing engine, a composite relationship graph associated with the historic event data; and

obtaining a projection of the composite relationship graph, based on a requirement of the particular machine learning model,

wherein the projection includes a graph of website visitation activities of users to facilitate identifying, by the particular machine learning model, of commonly accessed websites by potentially security-compromised users.

20. The method of claim 1, further comprising:

receiving user feedback regarding a determination of a detected security-related issue; and

updating the particular machine learning model based on the user feedback.

21. The method of claim 1, further comprising:

using a job controller to cause a retrieval of the historic event data; and

determining, by the job controller and based on a directory catalog, an order by which the batch event processing engine is to process the historic event data.

22. The method of claim 1, wherein the security-related issues include at least one of:

a security-related anomaly or a security-related threat, wherein the security-related anomaly represents a detected fact, and wherein the security-related threat represents a security-related interpretation of one or more detected anomalies.

23. The method of claim 1, wherein the event data comprise machine data.

24. The method of claim 1, wherein the event data comprise timestamped machine data.

25. The method of claim 1, wherein the real-time event processing engine is implemented using Apache Storm™

or Apache Spark Streaming™

.

26. The method of claim 1, wherein the batch event processing engine is implemented using Apache Spark™

.

27. The method of claim 1, wherein the particular machine learning model is a reducible model.

28. The method of claim 1, wherein the particular machine learning model is a reducible model being reducible in at least one of:

a training phase, or a scoring phase.

29. A computer system comprising:

a real-time event processing engine implemented on a distributed data processing platform, wherein the real-time event processing engine is configured to process an unbounded stream of event data to detect a plurality of network security-related issues and/or to train a machine learning model;

a batch event processing engine implemented on the distributed data processing platform, wherein the batch event processing engine is configured to process a batch of historic event data to detect a plurality of network security-related issues and/or to train a machine learning model; and

wherein the real-time event processing engine and the batch event processing engine shares a model state of a particular machine learning model, the particular machine learning model being configured to process a time slice of data to produce a score for detecting a network security-related issue, and

wherein the real-time event processing engine and the batch event processing engine each utilize the shared model state to share, with the other engine, network security-related knowledge gained from processing one'"'"'s respective data.

30. A non-transitory machine-readable storage medium for use in a processing system, the non-transitory machine-readable storage medium storing instructions, an execution of which in the processing system causes the processing system to perform operations comprising:

implementing a real-time event processing engine on a distributed data processing platform, wherein the real-time event processing engine is configured to process an unbounded stream of event data to detect a plurality of network security-related issues and/or to train a machine learning model;

implementing a batch event processing engine on the distributed data processing platform, wherein the batch event processing engine is configured to process a batch of historic event data to detect a plurality of network security-related issues and/or to train a machine learning model; and

enabling the real-time event processing engine and the batch event processing engine to share a model state of a particular machine learning model, the particular machine learning model being configured to process a time slice of data to produce a score for detecting a network security-related issue,

wherein the real-time event processing engine and the batch event processing engine each utilize the shared model state to share, with the other engine, network security-related knowledge gained from processing one'"'"'s respective data.