Event mini-graphs in data intake stage of machine data processing platform

• US 9,596,254 B1
• Filed: 10/30/2015
• Issued: 03/14/2017
• Est. Priority Date: 08/31/2015
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Muddu, Sudhakar, Tryfonas, Christos, Bulusu, Ravi Prasad
• Primary Examiner(s): Shiferaw, Eleni
• Assistant Examiner(s): Li, Mary
• Application Number:
  US14/929,203
• Publication Number:
  US 20170063912A1
• Time in Patent Office: 501 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/444 Spatial browsing, e.g. 2D maps, 3D or virtual spaces
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 40/134 Hyperlinking
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06V 10/225 based on a marking or identifier characterising the area
• CPC Class Codes: H04L 2463/121 Timestamp cryptographic mechanisms or cryptographic arrangements involving time stamps H04L9/3297
• CPC Class Codes: H04L 41/0893 Assignment of logical groups to network elements
• CPC Class Codes: H04L 41/145 involving simulating, designing, planning or modelling of a network
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 43/00 : Arrangements for monitoring or testing data switching networks
• CPC Class Codes: H04L 43/045 : for graphical visualisation of monitoring data
• CPC Class Codes: H04L 43/062 : related to network traffic
• CPC Class Codes: H04L 43/08 : Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 63/06 : for supporting key management in a packet data network cryptographic mechanisms or cryptographic arrangements for key management H04L9/08
• CPC Class Codes: H04L 63/1408 : by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/1416 : Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 : Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1433 : Vulnerability analysis
• CPC Class Codes: H04L 63/1441 : Countermeasures against malicious traffic countermeasures against attacks on cryptographic mechanisms H04L9/002
• CPC Class Codes: H04L 63/20 : for managing network security; network security policies in general filtering policies H04L63/0227
• CPC Class Codes: H05K 999/99 : dummy group

Abstract

A security platform employs a variety techniques and mechanisms to detect security related anomalies and threats in a computer network environment. The security platform is “big data” driven and employs machine learning to perform security analytics. The security platform performs user/entity behavioral analytics (UEBA) to detect the security related anomalies and threats, regardless of whether such anomalies/threats were previously known. The security platform can include both real-time and batch paths/modes for detecting anomalies and threats. By visually presenting analytical results scored with risk ratings and supporting evidence, the security platform enables network security administrators to respond to a detected anomaly or threat, and to take action promptly.

Claims

1. A method for processing data at data intake for detection of an anomaly in a distributed computer environment, the method comprising:

receiving event data representing an event on a computer network, the event data being indicative of a plurality of entities and an action involved in the event;

identifying the entities and a relationship between the entities, based on the action in the event data;

creating, for the event, a record of the relationship between the entities by using a data structure representing a relationship graph, the relationship graph including at least two nodes and an edge between the two nodes, each node representing one of the entities, the edge representing the relationship between the entities; and

creating an updated event data by appending the event data representing the event with the record of the relationship;

wherein the record of the relationship is specific to the event;

sending the updated event data to an event processing engine for further processing;

generating a composite relationship graph that is combined from a plurality of the relationship graph corresponding to a plurality of the updated event data; and

using the event processing engine to perform analytics on the plurality of the updated event data and the composite relationship graph;

wherein the anomaly detection is performed based on applying a machine learning model to perform analytics on at least a portion of the composite relationship graph.

2. The method of claim 1, wherein the relationship between the entities represents an activity, recorded in the event data, performed by one entity with respect to another entity.

3. The method of claim 1, further comprising:

sending the updated event data to a distributed messaging system.

4. The method of claim 1, further comprising:

using the event processing engine to identify security-oriented anomalies in the plurality of events.

5. The method of claim 1, further comprising:

sending the record of the relationship to the event processing engine for further processing; and

using the event processing engine to use a downstream entity to perform analytics on the plurality of the updated event data and the composite relationship graph.

6. The method of claim 1, further comprising:

repeating said receiving and identifying steps for event data representing each of a plurality of events on the computer network to generate the relationship graph for each of the plurality of events.

7. The method of claim 1, further comprising:

repeating said receiving and identifying steps for event data representing each of a plurality of events on the computer network to generate the relationship graph for each of the plurality of events; and

combining additional relationship graphs for the plurality of events into the composite relationship graph.

8. The method of claim 1, further comprising:

repeating said receiving and identifying steps for event data representing each of a plurality of events on the computer network to generate the relationship graph for each of the plurality of events; and

combining additional relationship graphs for the plurality of events into the composite relationship graph;

wherein the composite relationship graph includes all identified relationships among all identified entities involved in the plurality of events.

9. The method of claim 1, wherein said identifying step comprises:

tokenizing the event data by extracting, as tokens, a key or a value or a key-value pair in the event data; and

parsing the event data based on a predetermined data format that specifies which tokens represent the entities and which tokens represent actions in the extracted tokens.

10. The method of claim 1, wherein said identifying step comprises:

parsing the event data based on a predetermined data format that specifies which data represent the entities and which data represent the action in the event.

11. The method of claim 1, wherein said identifying step comprises:

parsing the event data based on a predetermined data format that specifies which data represent the entities and which data represent the action in the event,

wherein the predetermined data format comprises a data format representing output from at least one of;

an active directory, a proxy, a firewall, a web gateway, a virtual private network (VPN) connection, an intrusion detection system, a network traffic analyzer, or a malware engine.

12. The method of claim 1, wherein identifying the entities and the action in the event comprises:

tokenizing the event data by extracting, as tokens, a key or a value or a key-value pair in the event.

13. The method of claim 1, wherein the action is performed by one entity with respect to another entity, and wherein the identified relationship between the entities is identified based on the action.

14. The method of claim 1, wherein the identified relationship between the entities is indicative of the action, and wherein the identified relationship is identified based on comparing the action with a table of identifiable relationships.

15. The method of claim 1, wherein said identifying step comprises:

detecting a data format of the event data.

16. The method of claim 1, further comprising:

issuing a query to a data processing system that performs the query against data stored in a distributed file system,

wherein said receiving step includes receiving the event data as a result of the query from the processing system.

17. The method of claim 1, further comprising:

issuing a query to a data processing system that performs the query against data stored in a distributed file system,

wherein said receiving step includes receiving the event data as a result of the query from the processing system, and

wherein the data processing system includes a framework that provides methods including;

a map method that performs a data processing operation on local data on distributed nodes, and a reduce method that performs a summary operation to generate a result based on the processed local data.

18. The method of claim 1, further comprising:

requesting a machine on the computer network to transmit the event data, and wherein said receiving event data occurs in response to the machine on the computer network transmitting the event data.

19. The method of claim 1, wherein said identifying step comprises identifying a timestamp in the event.

20. The method of claim 1, further comprising:

configuring said identifying step by making an adjustment to a configuration file.

21. The method of claim 1, wherein the event data comprises timestamped machine data.

22. The method of claim 1, wherein the method is performed as part of an extract-transform-load stage of at least one of a distributed event processing system or an anomaly detection system.

23. The method of claim 1, further comprising:

identifying a plurality of attributes of the event, based on the event data; and

adding a view identifier to the event data to allow a downstream entity, by having designated the view identifier, to receive select information extracted from and/or generated based on the plurality of attributes of the event, through an interface identified by the view identifier.

24. The method of claim 1, further comprising:

identifying a plurality of attributes of the event, based on the event data; and

adding a view identifier to the event data to allow a downstream entity, by having designated the view identifier, to receive select information extracted from and/or generated based on the plurality of attributes of the event, through an interface identified by the view identifier,

wherein the interface further includes logic performs an action on the attributes of the event to generate the information.

25. The method of claim 1, wherein each of the entities is of a type that is one of:

user, device, application, uniform resource locator (URL), session, or threat; and

wherein at least two of the entities are not of the same type.

26. The method of claim 1, wherein each of the entities is of a type that is one of:

user, device, application, uniform resource locator (URL), session, or threat; and

wherein the action includes at least one of;

uses, visits, connects to, logs in, or logs out.

29. The method of claim 1, wherein the event processing engine is in a real-time path or a batch path.

27. A computer system for detection of an anomaly in a distributed computer environment, the system comprising:

a communication device; and

a processor configured to;

receive, via the communication device, event data representing an event on a computer network, the event data being indicative of a plurality of entities and an action involved in the event;

identify the entities and a relationship between the entities, based on the action in the event data;

create, for the event, a record of the relationship between the entities by using a data structure representing a relationship graph, the relationship graph including at least two nodes and an edge between the two nodes, each node representing one of the entities, the edge representing the relationship between the entities;

create an updated event data by appending the event data representing the event with the record of the relationship,

wherein the record of the relationship is specific to the event;

send the updated event data to an event processing engine for further processing;

generate a composite relationship graph that is combined from a plurality of the relationship graph corresponding to a plurality of the updated event data; and

use the event processing engine to perform analytics on the plurality of the updated event data and the composite relationship graph,

wherein the anomaly detection is performed based on applying a machine learning model to perform analytics on at least a portion of a the composite relationship graph.

28. A non-transitory machine-readable storage medium for use in a processing system for detection of an anomaly in a distributed computer environment, the non-transitory machine-readable storage medium storing instructions, an execution of which in the processing system causes the processing system to perform operations comprising:

receiving event data representing an event on a computer network, the event data being indicative of a plurality of entities and an action involved in the event;

identifying the entities and a relationship between the entities, based on the action in the event data;

creating, for the event, a record of the relationship between the entities by using a data structure representing a relationship graph, the relationship graph including at least two nodes and an edge between the two nodes, each node representing one of the entities, the edge representing the relationship between the entities; and

creating an updated event data by appending the event data representing the event with the record of the relationship,

wherein the record of the relationship is specific to the event;

sending the updated event data to an event processing engine for further processing;

generating a composite relationship graph that is combined from a plurality of the relationship graph corresponding to a plurality of the updated event data; and

using the event processing engine to perform analytics on the plurality of the updated event data and the composite relationship graph,

wherein the anomaly detection is performed based on applying a machine learning model to perform analytics on at least a portion of the composite relationship graph.