Event specific entity relationship discovery in data intake stage of a distributed data processing system

• US 10,419,463 B2
• Filed: 03/20/2018
• Issued: 09/17/2019
• Est. Priority Date: 08/31/2015
• Status: Active Grant

• Current Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Original Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Inventors: Muddu, Sudhakar, Tryfonas, Christos, Bulusu, Ravi Prasad
• Primary Examiner(s): Song, Hosuk
• Assistant Examiner(s): Murphy, J. Brant
• Application Number:
  US15/926,961
• Publication Number:
  US 20180219897A1
• Time in Patent Office: 546 Days
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/444 Spatial browsing, e.g. 2D maps, 3D or virtual spaces
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 40/134 Hyperlinking
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06V 10/225 based on a marking or identifier characterising the area
• CPC Class Codes: H04L 2463/121 Timestamp
• CPC Class Codes: H04L 41/0893 Assignment of logical groups to network elements
• CPC Class Codes: H04L 41/145 involving simulating, designing, planning or modelling of a network
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 43/00 : Arrangements for monitoring or testing data switching networks
• CPC Class Codes: H04L 43/045 : for graphical visualisation of monitoring data
• CPC Class Codes: H04L 43/062 : related to network traffic
• CPC Class Codes: H04L 43/08 : Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 63/06 : for supporting key management in a packet data network cryptographic mechanisms or cryptographic arrangements for key management H04L9/08
• CPC Class Codes: H04L 63/1408 : by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/1416 : Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 : Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1433 : Vulnerability analysis
• CPC Class Codes: H04L 63/1441 : Countermeasures against malicious traffic countermeasures against attacks on cryptographic mechanisms H04L9/002
• CPC Class Codes: H04L 63/20 : for managing network security; network security policies in general filtering policies H04L63/0227

Abstract

A security platform employs a variety techniques and mechanisms to detect security related anomalies and threats in a computer network environment. The security platform is “big data” driven and employs machine learning to perform security analytics. The security platform performs user/entity behavioral analytics (UEBA) to detect the security related anomalies and threats, regardless of whether such anomalies/threats were previously known. The security platform can include both real-time and batch paths/modes for detecting anomalies and threats. By visually presenting analytical results scored with risk ratings and supporting evidence, the security platform enables network security administrators to respond to a detected anomaly or threat, and to take action promptly.

Claims

1. A method comprising:

accessing machine data representing an event, the event resulting from an activity that occurred on a computer network;

identifying, in the event, a first entity that participated in the activity and a relationship between the first entity and a second entity, the relationship being indicative of the activity; and

annotating, by using a graph data structure, raw machine data of the event to include the identified relationship between the first and second entities in the event, wherein annotation in the event enables an event processing engine to detect an anomaly in the computer network.

2. The method of claim 1, wherein the method is performed by a number of machines internal to the computer network.

3. The method of claim 1, wherein the method is performed, by a number of machines external to the computer network, as a cloud service to a client.

4. The method of claim 1, wherein the method is performed, by a number of machines external to the computer network, as a cloud service to a client, and wherein the cloud service also hosts a software application of the client.

5. The method of claim 1, further comprising:

utilizing a data connector for receiving the machine data from a data source,

wherein the data connector supports accessing or receiving one or more of;

(1) indexed data, (2) unindexed data, (3) data from a third-party provider, or (4) data from a distributed file system.

6. The method of claim 1, further comprising:

utilizing a data connector for receiving the machine data from a data source,

wherein the data connector supports a pull mechanism, a push mechanism, and a hybrid mechanism.

7. The method of claim 1, further comprising:

performing said method for a plurality of heterogeneous events.

8. The method of claim 1, further comprising:

receiving the machine data from a data source,

wherein the data source is internal to the computer network.

9. The method of claim 1, further comprising:

receiving the machine data from a data source,

wherein the data source is external to the computer network.

10. The method of claim 1, further comprising:

receiving the machine data from a data source,

wherein the event includes one or more of;

(1) an identity or authentication type event;

(2) a network activity type event;

(3) a security software generated type event;

(4) an event from a software service source external to the computer network;

or (5) a third-party threat feed.

11. The method of claim 1, further comprising:

sending the annotated raw machine data of the event to a distributed messaging system.

12. The method of claim 1, further comprising:

generating a composite relationship graph that is based on a combination of a plurality of the identified relationships in a plurality of events.

13. The method of claim 1, further comprising:

generating a composite relationship graph that is based on a combination of a plurality of the identified relationships in a plurality of events, wherein the detection of an anomaly in the information technology environment is performed based on a projection of the composite relationship graph.

14. The method of claim 1, wherein the relationship between the first and second entities is deterministically identified according to actual records of the event.

15. The method of claim 1, further comprising:

utilizing, by an anomaly detection module, machine learning models to detect behavioral-based anomalies.

16. The method of claim 1, further comprising:

extracting, at a subsequent stage, the data indicative of the relationship from the event in response to the event being examined.

17. The method of claim 1, wherein said identifying comprises:

tokenizing the raw machine data of the event by extracting, as tokens, a key, a value or a key-value pair from the raw machine data of the event; and

parsing the raw machine data of the event based on a predetermined data format that specifies which tokens represent the first and second entities and which tokens represent activities in the extracted tokens.

18. The method of claim 1, wherein said identifying comprises:

tokenizing the raw machine data of the event by extracting, as tokens, a key or a value or a key-value pair of the event.

19. The method of claim 1, wherein said identifying comprises:

detecting a data format of the raw machine data of the event.

20. The method of claim 1, further comprising:

issuing a query to a data processing system configured to perform the query against data stored in a distributed file system; and

receiving the raw machine data of the event as a result of the query from the processing system.

21. The method of claim 1, further comprising:

causing a data processing system to execute a query against data stored in a distributed file system; and

receiving the raw machine data of the event as a result of the query from the processing system,

wherein the data processing system includes a framework that provides methods including;

a map method to perform a data processing operation on local data on distributed nodes, and a reduce method to perform a summary operation to generate a result based on the processed local data.

22. The method of claim 1, wherein said identifying comprises identifying a timestamp of the event.

23. The method of claim 1, further comprising:

configuring future executions of said identifying by making an adjustment to a configuration file.

24. The method of claim 1, wherein the raw machine data of the event comprises timestamped machine data.

25. The method of claim 1, wherein the method is performed as part of an extract-transform-load stage of at least one of a distributed event processing system or an anomaly detection system.

26. The method of claim 1, further comprising:

identifying a plurality of attributes of the event, based on the raw machine data of the event; and

adding a view identifier to the raw machine data of the event to allow a downstream entity, by having designated the view identifier, to receive select information extracted from and/or generated based on the plurality of attributes of the event.

27. The method of claim 1, further comprising:

identifying a plurality of attributes of the event, based on the raw machine data of the event; and

adding a view identifier to the raw machine data of the event to allow a downstream entity, by having designated the view identifier, to receive select information extracted from and/or generated based on the plurality of attributes of the event,

wherein the interface further includes logic performs an activity on the attributes of the event to generate the information.

28. The method of claim 1, wherein (1) each of the entities is of a type that is one of:

user, device, application, network resource locator, session, or threat; and

wherein at least two of the entities are not of the same type;

or, wherein (2) each of the entities is of a type that is one of;

user, device, application, uniform resource locator (URL), session, or threat; and

wherein the activity includes at least one of;

uses, visits, connects to, logs in, or logs out.

29. A computer system for detection of an anomaly in a distributed computer environment, the system comprising:

a communication device; and

a processor configured to;

access machine data representing an event, the event resulting from an activity that occurred on a computer network;

identify, in the event, a first entity that participated in the activity and a relationship between the first entity and a second entity, the relationship being indicative of the activity; and

annotate, by using a graph data structure, raw machine data of the event to include the identified relationship between the first and second entities in the event, wherein annotation in the event enables an event processing engine to detect an anomaly in the computer network.

30. A non-transitory machine-readable storage medium for use in a processing system for detection of an anomaly in a distributed computer environment, the non-transitory machine-readable storage medium storing instructions, execution of which in the processing system causes the processing system to perform operations comprising:

accessing machine data representing an event, the event resulting from an activity that occurred on a computer network;

identifying, in the event, a first entity that participated in the activity and a relationship between the first entity and a second entity, the relationship being indicative of the activity; and

annotating, by using a graph data structure, raw machine data of the event to include the identified relationship between the first and second entities in the event, wherein annotation in the event enables an event processing engine to detect an anomaly in the computer network.