COMPLEX EVENT PROCESSING OF COMPUTER NETWORK DATA

• US 20170063906A1
• Filed: 10/30/2015
• Published: 03/02/2017
• Est. Priority Date: 08/31/2015
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Muddu, Sudhakar, Tryfonas, Christos
• Granted Patent: US 9,667,641 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/444 Spatial browsing, e.g. 2D maps, 3D or virtual spaces
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 40/134 Hyperlinking
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06V 10/225 based on a marking or identifier characterising the area
• CPC Class Codes: H04L 2463/121 Timestamp cryptographic mechanisms or cryptographic arrangements involving time stamps H04L9/3297
• CPC Class Codes: H04L 41/0893 Assignment of logical groups to network elements
• CPC Class Codes: H04L 41/145 involving simulating, designing, planning or modelling of a network
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 43/00 : Arrangements for monitoring or testing data switching networks
• CPC Class Codes: H04L 43/045 : for graphical visualisation of monitoring data
• CPC Class Codes: H04L 43/062 : related to network traffic
• CPC Class Codes: H04L 43/08 : Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 63/06 : for supporting key management in a packet data network cryptographic mechanisms or cryptographic arrangements for key management H04L9/08
• CPC Class Codes: H04L 63/1408 : by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/1416 : Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 : Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1433 : Vulnerability analysis
• CPC Class Codes: H04L 63/1441 : Countermeasures against malicious traffic countermeasures against attacks on cryptographic mechanisms H04L9/002
• CPC Class Codes: H04L 63/20 : for managing network security; network security policies in general filtering policies H04L63/0227
• CPC Class Codes: H05K 999/99 : dummy group

Abstract

A security platform employs a variety techniques and mechanisms to detect security related anomalies and threats in a computer network environment. The security platform is “big data” driven and employs machine learning to perform security analytics. The security platform performs user/entity behavioral analytics (UEBA) to detect the security related anomalies and threats, regardless of whether such anomalies/threats were previously known. The security platform can include both real-time and batch paths/modes for detecting anomalies and threats. By visually presenting analytical results scored with risk ratings and supporting evidence, the security platform enables network security administrators to respond to a detected anomaly or threat, and to take action promptly.

Claims

1. A method comprising:

transforming in real time raw event data, representing a sequence of events and captured from multiple sources in a computer network, into a stream of event feature sets without a known end-point to the stream, where the raw event data includes time-stamped machine data;

computing in real-time a score by processing a time slice of the stream of event feature sets through an active version of a machine learning model, wherein the time slice includes a most recent event feature set in the stream of event feature sets;

training, in parallel with said processing the time slice and responsive in real-time to said transforming the raw event data, a non-active version of the machine learning model with the time slice that is being processed through the active version for scoring, wherein the machine learning model is trained to represent a particular entity involved in a computer network activity represented by the raw event data;

identifying, by comparing the score against a threshold, a security-related anomaly or a security-related threat to enable remediation of the security-related anomaly or the security-related threat in the computer network as the stream of event feature sets is processed in real-time;

determining that the non-active version of the machine learning model is ready for active deployment based on at least one of;

a number of event feature sets that have been used to train the non-active version, length of time that the non-active version has been in training, or whether a model state of the non-active version is converging; and

live-swapping in the non-active version as the active version to compute another score by processing a subsequent time slice from the stream of event feature sets through the live-swapped-in active version of the machine learning model.

2. The method of claim 1, wherein the time slice is the most recent time slice received from the stream of event feature sets.

3. The method of claim 1, wherein the time slice corresponds to an event in the sequence of events;

and wherein said identifying the security-related anomaly or the security-related threat includes determining that the security-related anomaly or the security-related threat corresponds to the event.

4. The method of claim 1, further comprising training the machine learning model by processing a previous time slice of the stream of event feature sets, prior to said computing the score.

5. The method of claim 1, wherein said processing the time slice through the active version of the machine learning model includes processing the time slice through a model deliberation process logic configured by the active version of the machine learning model;

wherein said training the non-active version of the machine learning model includes processing the time slice through a model training process logic; and

wherein the model training process logic and the model deliberation process logic correspond to a particular machine learning model type.

6. The method of claim 1,

wherein

said live swapping includes re-configuring, without first terminating, a model deliberation process thread that is performing said computing in real-time.

8. The method of claim 1, wherein the machine learning model is an unsupervised machine learning model.

9. The method of claim 1, furthering comprising processing the stream of event feature sets through a plurality of machine learning models of different types to detect security-related anomalies or threats of different types.

10. The method of claim 1, wherein the machine learning model is a supervised machine learning model or a semi-supervised machine learning model.

11. The method of claim 1, wherein the machine learning model is a deep machine learning model.

13. The method of claim 1, wherein the machine learning model is specific to a particular entity involved in the sequence of events, wherein the entity is a user, a device, a system, a network resource locator, an application, a process thread, or any combination thereof.

14. The method of claim 1, wherein the machine learning model performs behavioral analysis of a particular entity involved in the sequence of events.

15. The method of claim 1, wherein the machine learning model performs peer group analysis amongst entities involved in the sequence of events.

16. The method of claim 1, wherein the machine learning model performs time series analysis on a sequence of the event feature sets.

17. The method of claim 1, wherein the machine learning model performs graph correlation analysis amongst entities involved in the sequence of events.

18. The method of claim 1, wherein the machine learning model includes a state machine specific to an entity involved in an event represented in the time slice.

19. The method of claim 1, wherein said computing the score includes processing the stream of event feature sets through the machine learning model according to model deliberation processing logic specified by a model type associated with the machine learning model.

20. The method of claim 1, further comprising training the machine learning model according to model training processing logic specified by a model type associated with the machine learning model.

21. The method of claim 1, further comprising incrementally updating the machine learning model in real-time using the time slice of the event feature sets, wherein said incremental updating includes:

isolating a portion of a model state representative of the machine learning model affected by the event feature sets; and

re-training only the portion of the model state.

22. The method of claim 1, further comprising simultaneously training multiple machine learning models, associated with different purposes or different entities, in real-time.

23. The method of claim 1, wherein computing the score is performed in a distributed computation system implementing a task-parallel distributed data processing engine.

24. The method of claim 1, further comprising training the machine learning model in a distributed computation system implementing a task-parallel distributed data processing engine.

25. The method of claim 1, further comprising training the machine learning model in real-time using single-pass training processing logic.

26. The method of claim 1, further comprising:

training the machine learning model; and

storing a model state of the machine learning model, resulting from said training, in a distributed cache for use in said computing of the score.

27. The method of claim 1, further comprising outputting, to a user via a computer output device, a representation of the security-related anomaly or the security-related threat, in response to determining the security-related anomaly or the security-related threat.

28. The method of claim 1, wherein the threshold is defined by the machine learning model.

7. (canceled)

12. (canceled)

29. A system comprising:

at least one hardware processor implementing an extract transform load (ETL) engine configured to transform in real time raw event data, representing a sequence of events and captured from multiple sources in a computer network, into an stream of event feature sets without a known end-point to the stream, where the raw event data includes time-stamped machine data;

at least one hardware processor implementing a model execution engine configured to compute in real-time a score by processing a time slice of the stream of event feature sets through an active version of a machine learning model and to train, in parallel with said processing the time slice and responsive in real-time to said transforming the raw event data, a non-active version of the machine learning model with the time slice that is being processed through the active version for scoring, wherein the machine learning model is trained to represent a particular entity involved in a computer network activity represented by the raw event data, and wherein the time slice includes a most recent event feature set in the stream of event feature sets; and

at least one hardware processor implementing a computing node configured to identify, by comparing the score against a threshold, a security-related anomaly or a security-related threat to enable remediation of the security-related anomaly or the security-related threat in the computer network as the stream of event feature sets is processed in real-time;

wherein the model execution engine is configured to determine that the non-active version of the machine learning model is ready for active deployment based on at least one of;

a number of event feature sets that have been used to train the non-active version, length of time that the non-active version has been in training, or whether a model state of the non-active version is converging, and to live-swap in the non-active version as the active version to compute another score by processing a subsequent time slice from the stream of event feature sets through the live-swapped-in active version of the machine learning model.

30. A non-transitory computer readable medium storing instructions there on which, when executed by a processor, cause the processor to:

transform in real time raw event data, representing a sequence of events and captured from multiple sources in a computer network, into an stream of event feature sets without a known end-point to the stream, where the raw event data includes time-stamped machine data;

compute in real-time a score by processing a time slice of the stream of event feature sets through an active version of a machine learning model;

train, in parallel with said processing the time slice and responsive in real-time to said transforming the raw event data, a non-active version of the machine learning model with the time slice that is being processed through the active version for scoring, wherein the machine learning model is trained to represent a particular entity involved in a computer network activity represented by the raw event data and wherein the time slice includes a most recent event feature set in the stream of event feature sets;

identify, by comparing the score against a threshold, a security-related anomaly or a security-related threat to enable remediation of the security-related anomaly or the security-related threat in the computer network as the stream of event feature sets is processed in real-time;

determine that the non-active version of the machine learning model is ready for active deployment based on at least one of;

a number of event feature sets that have been used to train the non-active version, length of time that the non-active version has been in training, or whether a model state of the non-active version is converging; and

live-swap in the non-active version as the active version to compute another score by processing a subsequent time slice from the stream of event feature sets through the live-swapped-in active version of the machine learning model.