Data retrieval in security anomaly detection platform with shared model state between real-time and batch paths

• US 10,419,465 B2
• Filed: 11/06/2018
• Issued: 09/17/2019
• Est. Priority Date: 08/31/2015
• Status: Active Grant

• Current Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Original Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Inventors: Muddu, Sudhakar, Tryfonas, Christos, Bulusu, Ravi Prasad
• Primary Examiner(s): To, Baotran N
• Assistant Examiner(s): Gyorfi, Thomas A
• Application Number:
  US16/182,469
• Publication Number:
  US 20190075126A1
• Time in Patent Office: 315 Days
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/444 Spatial browsing, e.g. 2D maps, 3D or virtual spaces
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 40/134 Hyperlinking
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06V 10/225 based on a marking or identifier characterising the area
• CPC Class Codes: H04L 2463/121 Timestamp
• CPC Class Codes: H04L 41/0893 Assignment of logical groups to network elements
• CPC Class Codes: H04L 41/145 involving simulating, designing, planning or modelling of a network
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 43/00 : Arrangements for monitoring or testing data switching networks
• CPC Class Codes: H04L 43/045 : for graphical visualisation of monitoring data
• CPC Class Codes: H04L 43/062 : related to network traffic
• CPC Class Codes: H04L 43/08 : Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 63/06 : for supporting key management in a packet data network cryptographic mechanisms or cryptographic arrangements for key management H04L9/08
• CPC Class Codes: H04L 63/1408 : by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/1416 : Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 : Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1433 : Vulnerability analysis
• CPC Class Codes: H04L 63/1441 : Countermeasures against malicious traffic countermeasures against attacks on cryptographic mechanisms H04L9/002
• CPC Class Codes: H04L 63/20 : for managing network security; network security policies in general filtering policies H04L63/0227

Abstract

A security platform employs a variety techniques and mechanisms to detect security related anomalies and threats in a computer network environment. The security platform is “big data” driven and employs machine learning to perform security analytics. The security platform performs user/entity behavioral analytics (UEBA) to detect the security related anomalies and threats, regardless of whether such anomalies/threats were previously known. The security platform can include both real-time and batch paths/modes for detecting anomalies and threats. By visually presenting analytical results scored with risk ratings and supporting evidence, the security platform enables network security administrators to respond to a detected anomaly or threat, and to take action promptly.

Claims

1. A method comprising:

implementing a batch event processing engine on a distributed data processing platform, wherein the batch event processing engine is configured to process a batch of historic event data;

performing an interaction with a datastore to retrieve specific event data;

scheduling the batch event processing engine to process the specific event data; and

enabling the batch event processing engine to share a model state of a particular machine learning model, with a real-time event processing engine on the distributed data processing platform, the real-time event processing engine being configured to process an unbounded stream of event data, the particular machine learning model being configured to process a time slice of data for detecting a security-related issue,

wherein the real-time event processing engine and the batch event processing engine each utilize the shared model state to share, with the other engine, security-related knowledge gained from processing respective data.

2. The method of claim 1, wherein the interaction with a datastore includes sending, to the datastore, a query or a command to be executed by the datastore.

3. The method of claim 1, wherein the interaction with a datastore includes specifying, in a query or a command to be executed by the datastore, a manner in which resulting data from the datastore is to be delivered.

4. The method of claim 1, wherein the interaction with a datastore includes specifying, in a query or a command to be executed by the datastore, a manner in which resulting data from the datastore is to be delivered, and

wherein the manner includes selecting a particular set of the resulting data as the specific event data.

5. The method of claim 1, wherein the interaction with a datastore includes specifying, in a query or a command to be executed by the datastore, a manner in which resulting data from the datastore is to be delivered, and

wherein the manner includes specifying a time range of the resulting data.

6. The method of claim 1, wherein the interaction with a datastore includes specifying, in a query or a command to be executed by the datastore, a manner in which resulting data from the datastore is to be delivered, and

wherein the manner includes specifying an order of a set of the resulting data.

7. The method of claim 1, wherein the interaction with a datastore includes specifying, in a query or a command to be executed by the datastore, a manner in which resulting data from the datastore is to be delivered, and

wherein the manner includes specifying an order of a set of the resulting data,

wherein the order includes one or more of:

event time, data format, or a type of event that a particular resulting data represent.

8. The method of claim 1, wherein the interaction with a datastore includes specifying, in a query or a command to be executed by the datastore, a manner in which resulting data from the datastore is to be delivered, and

wherein the manner includes specifying an order of a set of the resulting data,

wherein the order specifies that log files with device information are to be delivered first, followed by log files that associate user data with devices, followed by other files.

9. The method of claim 1, wherein the interaction with a datastore includes specifying, in a query or a command to be executed by the datastore, a manner in which resulting data from the datastore is to be delivered, and

wherein the manner includes specifying an order of a set of the resulting data,

wherein the order specifies that DHCP logs are to be delivered first, followed by AD or VPN logs, followed by other files.

10. The method of claim 1, wherein the interaction with a datastore includes specifying, in a query or a command to be executed by the datastore, a manner in which resulting data from the datastore is to be delivered, and

wherein the manner includes specifying that a particular type of the resulting data is to be delivered on a higher priority than another particular type.

11. The method of claim 1, further comprising:

determining a priority of data type in the resulting data from the datastore; and

determining, based on the priority of the data type, a manner in which resulting data from the datastore is to be delivered.

12. The method of claim 1, further comprising:

determining a priority of data type in the resulting data from the datastore; and

determining, based on the priority of the data type, a manner in which resulting data from the datastore is to be delivered,

wherein the interaction with a datastore includes specifying, in a query or a command to be executed by the datastore, the manner in which resulting data from the datastore is to be delivered.

13. The method of claim 1, wherein said scheduling the batch event processing engine to process the specific event data includes:

initiating a job for the batch event processing engine with the specific event data;

tracking a progress of the job; and

recording, based on the progress, an analytical result for events represented by the specific event data.

14. The method of claim 1, further comprising:

determining a time schedule for performing said interaction with the datastore.

15. The method of claim 1, further comprising:

receiving an initial result from the datastore regarding the specific event data;

determining, based on the initial result, a subsequent interaction with the datastore.

16. The method of claim 1, further comprising:

receiving an initial result from the datastore regarding the specific event data; and

determining, based on the initial result, a subsequent interaction with the datastore,

wherein the subsequent interaction with the datastore includes causing the datastore to modify a delivery order of resulting data.

17. The method of claim 1, wherein the datastore is a Hadoop™

Distributed File System (HDFS) datastore.

18. The method of claim 1, wherein the shared model state enables the batch event processing engine to use knowledge gained by the real-time event processing engine to discover a security-related issue in the historic event data that is undetectable by the batch event processing engine without the knowledge.

19. The method of claim 1, further comprising:

performing, by the batch event processing engine, an analysis on the historic event data to detect a security-related issue, wherein the analysis includes at least one of;

a lateral movement anomaly analysis, a behavioral peer analysis, a label propagation analysis, or a time-series anomaly analysis.

20. The method of claim 1, further comprising:

locating, in the batch of historic event data, data representing a plurality of events associated with an entity; and

performing, by the batch event processing engine, a behavioral analysis of the entity to detect a behavioral anomaly.

21. The method of claim 1, further comprising:

locating, in the batch of historic event data, data representing a plurality of events that are associated with behaviors performed by a plurality of entities; and

performing, by the batch event processing engine, anomaly analysis on the behaviors performed by a plurality of entities to detect a particular security-related anomaly.

22. The method of claim 1, further comprising:

locating, by the batch event processing engine, a composite relationship graph associated with the historic event data; and

obtaining a projection of the composite relationship graph, based on a requirement of the particular machine learning model.

23. The method of claim 1, further comprising:

receiving user feedback regarding a determination of a detected security-related issue; and

updating the particular machine learning model based on the user feedback.

24. The method of claim 1, wherein the security-related issues include at least one of:

a security-related anomaly or a security-related threat, wherein the security-related anomaly represents a detected fact, and wherein the security-related threat represents a security-related interpretation of one or more detected anomalies.

25. The method of claim 1, wherein the event data comprise machine data.

26. The method of claim 1, wherein the event data comprise timestamped machine data.

27. The method of claim 1, wherein at least one of the event processing engines utilizes a particular machine learning model that is a reducible model.

28. The method of claim 1, wherein at least one of the event processing engines utilizes a particular machine learning model that is a reducible model being reducible in at least one of:

a training phase, or a scoring phase.

29. A computer system comprising:

a real-time event processing engine on the distributed data processing platform, the real-time event processing engine being configured to process an unbounded stream of event data; and

a batch event processing engine on a distributed data processing platform, wherein the batch event processing engine is configured to process a batch of historic event data;

wherein the system is configured to;

perform an interaction with a datastore to retrieve specific event data;

schedule the batch event processing engine to process the specific event data; and

enable the batch event processing engine to share a model state of a particular machine learning model, with the real-time event processing engine, the particular machine learning model being configured to process a time slice of data for detecting a security-related issue,

wherein the real-time event processing engine and the batch event processing engine each utilize the shared model state to share, with the other engine, security-related knowledge gained from processing respective data.

30. A non-transitory machine-readable storage medium for use in a processing system, the non-transitory machine-readable storage medium storing instructions, an execution of which in the processing system causes the processing system to perform operations comprising:

implementing a batch event processing engine on a distributed data processing platform, wherein the batch event processing engine is configured to process a batch of historic event data;

performing an interaction with a datastore to retrieve specific event data;

scheduling the batch event processing engine to process the specific event data; and

enabling the batch event processing engine to share a model state of a particular machine learning model, with a real-time event processing engine on the distributed data processing platform, the real-time event processing engine being configured to process an unbounded stream of event data, the particular machine learning model being configured to process a time slice of data for detecting a security-related issue,

wherein the real-time event processing engine and the batch event processing engine each utilize the shared model state to share, with the other engine, security-related knowledge gained from processing respective data.