Anomaly detection based on communication between entities over a network

• US 11,258,807 B2
• Filed: 07/03/2019
• Issued: 02/22/2022
• Est. Priority Date: 08/31/2015
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Muddu, Sudhakar, Tryfonas, Christos, Zadeh, Joseph Auguste, Bond, Alexander Beebe, Athalye, Ashwin
• Primary Examiner(s): Dada, Beemnet W
• Application Number:
  US16/503,181
• Publication Number:
  US 20190327251A1
• Time in Patent Office: 965 Days
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/444 Spatial browsing, e.g. 2D maps, 3D or virtual spaces
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 40/134 Hyperlinking
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06V 10/225 based on a marking or identifier characterising the area
• CPC Class Codes: H04L 2463/121 Timestamp cryptographic mechanisms or cryptographic arrangements involving time stamps H04L9/3297
• CPC Class Codes: H04L 41/0893 Assignment of logical groups to network elements
• CPC Class Codes: H04L 41/145 involving simulating, designing, planning or modelling of a network
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 43/00 : Arrangements for monitoring or testing data switching networks
• CPC Class Codes: H04L 43/045 : for graphical visualisation of monitoring data
• CPC Class Codes: H04L 43/062 : related to network traffic
• CPC Class Codes: H04L 43/08 : Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 63/06 : for supporting key management in a packet data network cryptographic mechanisms or cryptographic arrangements for key management H04L9/08
• CPC Class Codes: H04L 63/1408 : by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/1416 : Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 : Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1433 : Vulnerability analysis
• CPC Class Codes: H04L 63/1441 : Countermeasures against malicious traffic countermeasures against attacks on cryptographic mechanisms H04L9/002
• CPC Class Codes: H04L 63/20 : for managing network security; network security policies in general filtering policies H04L63/0227
• CPC Class Codes: H05K 999/99 : dummy group

Abstract

A security platform employs a variety techniques and mechanisms to detect security related anomalies and threats in a computer network environment. The security platform is “big data” driven and employs machine learning to perform security analytics. The security platform performs user/entity behavioral analytics (UEBA) to detect the security related anomalies and threats, regardless of whether such anomalies/threats were previously known. The security platform can include both real-time and batch paths/modes for detecting anomalies and threats. By visually presenting analytical results scored with risk ratings and supporting evidence, the security platform enables network security administrators to respond to a detected anomaly or threat, and to take action promptly.

Claims

1. A method comprising:

receiving, by a computer system, event data associated with a communication between an internal entity within a computer network and an external entity outside the computer network, the event data including an identifier associated with a particular entity, wherein the particular entity is the internal entity or the external entity;

analyzing, by the computer system, a plurality of characters in the identifier by processing the event data;

assigning, by the computer system, a feature score based on the analysis, wherein the feature score is indicative of a level of confidence that the identifier is machine generated and a level of randomness in a sequence of characters in the identifier;

determining, by the computer system, that the level of randomness satisfies a specified criterion; and

detecting, by the computer system, an anomaly based on the feature score, wherein the anomaly is detected in response to determining that the level of randomness satisfies the specified criterion.

2. The method of claim 1, wherein the detected anomaly is indicative of malware communications.

3. The method of claim 1, wherein the feature score is representative of a quantified evaluation of risk associated with the particular entity.

4. The method of claim 1, wherein analyzing the plurality of characters in the identifier includes:

analyzing a sequencing of the plurality of characters in the identifier.

5. The method of claim 1, wherein analyzing the plurality of characters in the identifier includes:

performing an n-gram analysis of the identifier to determine a level of randomness in a sequence of characters in the identifier.

6. The method of claim 1, wherein analyzing the plurality of characters in the identifier includes:

performing a lexical analysis of the identifier using natural language processing.

7. The method of claim 1, further comprising:

assigning, by the computer system, a second feature score based on any of;

timing of communications associated with the particular entity;

sequencing of communications associated with the particular entity;

data transmission statistics associated with the particular entity;

or

referral strings associated with the particular entity;

wherein the anomaly is further detected based on the second feature score.

8. The method of claim 1, wherein detecting the anomaly includes:

assigning an anomaly score based on the feature score; and

determining that the anomaly score satisfies a specified criterion;

wherein the anomaly is detected in response to determining that the anomaly score satisfies the specified criterion.

9. The method of claim 1, wherein detecting the anomaly includes:

assigning an anomaly score based on a weighted combination of the feature score and one or more other feature scores associated with the particular entity; and

determining that the anomaly score satisfies a specified criterion;

wherein the anomaly is detected in response to determining that the anomaly score satisfies the specified criterion.

10. The method of claim 1, wherein assigning the feature score includes:

processing the event data using a machine learning model, the machine learning model including;

model processing logic defining a process for assigning the feature score based on the event data; and

a model state defining a set of parameters for applying the model processing logic.

11. The method of claim 1, wherein detecting the anomaly includes:

processing the feature score using an anomaly model, the anomaly model including;

model processing logic defining a process for assigning an anomaly score based on the feature score; and

a model state defining a set of parameters for applying the model processing logic; and

assigning an anomaly score based on the processing of the feature score; and

determining that the anomaly score satisfies a specified criterion;

wherein the anomaly is detected in response to determining that the anomaly score satisfies the specified criterion.

12. The method of claim 1, wherein detecting the anomaly includes:

determining a volume of event data associated with the communication between the internal entity and the external entity;

processing the feature score using;

a first anomaly model if the volume of event data is at or above a threshold volume;

or

a second anomaly model if the volume of event data is below the threshold volume.

13. The method of claim 1, wherein detecting the anomaly includes:

processing the feature score using a plurality of machine-learning models;

assigning a plurality of intermediate anomaly scores, each of the plurality of intermediate anomaly scores based on processing of the feature score according to one of the plurality of machine-learning models;

processing the plurality of intermediate anomaly scores according to an ensemble-learning model;

assigning an anomaly score based on processing the plurality of intermediate anomaly scores; and

determining that the anomaly score satisfies a specified criterion;

wherein the anomaly is detected in response to determining that the anomaly score satisfies a specified criterion.

14. The method of claim 1, further comprising:

annotating, by the computer system, the detected anomaly with data from an external data source external to the computer network.

15. The method of claim 1, further comprising:

outputting, by the computer system, via a user interface, an indication of the detected anomaly to a user.

16. The method of claim 1, further comprising:

outputting, by the computer system, via a user interface, in response to detecting the anomaly, an incident response output, the incident response output including;

the identifier associated with the external or internal entity;

the feature score; and

a recommended response based on the feature score.

17. The method of claim 1, further comprising:

Incorporating, by the computer system, the detected anomaly into a network security graph, the network security graph including;

a plurality of nodes representing entities associated with the computer network, the entities including the particular entity; and

a plurality of edges, each of the plurality of edges linking two of the plurality of nodes and representing an association between the entities represented by the nodes;

wherein the detected anomaly is incorporated as a node linked to the particular entity by an edge.

18. The method of claim 1, wherein detecting the anomaly is performed in real time as the event data are received.

19. The method of claim 1, wherein detecting the anomaly is performed using Apache Storm or Apache Spark Streaming as a processing engine.

20. The method of claim 1, wherein receiving the event data includes:

adaptively filtering the event data according to a dynamic whitelist.

21. The method of claim 1, wherein the event data are receiving received from a plurality of entities associated with the computer network via an extract, transform, and load (ETL) pipeline.

22. The method of claim 1, wherein the event data are timestamped machine data.

23. The method of claim 1, wherein the event data include one or more of:

domain name system (DNS) generated log data, firewall generated log data, or proxy generated log data.

24. The method of claim 1, further comprising:

storing, by the computer system, anomaly data indicative of the detected anomaly in an anomaly graph data structure that includes;

a plurality of nodes, each of the plurality of nodes representing entities associated with the computer network, the entities including users and/or devices; and

a plurality of edges, each of the plurality of edges representing an anomaly linking two of the plurality of nodes.

25. The method of claim 1, wherein the feature score is one of a plurality of feature scores included in an entity profile associated with the particular entity, and wherein detecting the anomaly includes processing the entity profile using an anomaly model.

26. The method of claim 1, further comprising:

generating, by the computer system, an entity profile for the particular entity, the entity profile including a plurality of feature scores, the plurality of feature scores including the feature score assigned based on the analysis of the plurality of characters in the identifier, each of the plurality of feature scores assigned based on a different one of a plurality of different analyses of the event data;

wherein detecting the anomaly includes processing the entity profile using an anomaly model.

27. The method of claim 1, wherein the identifier is any of:

a domain name, a uniform resource locater (URL), uniform resource identifier (URI), an Internet Protocol (IP) address, a unique identifier (UID), a device identification, or a user identification.

28. A system comprising:

a processor; and

a memory unit having instructions stored thereon, which when executed by the processor cause the system to;

receive event data associated with a communication between an internal entity within a computer network and an external entity outside the computer network, the event data including an identifier associated with a particular entity, wherein the particular entity is the internal entity or the external entity;

analyze a plurality of characters in the identifier by processing the event data;

assign a feature score based on the analysis, wherein the feature score is indicative of a level of confidence that the identifier is machine generated and a level of randomness in a sequence of characters in the identifier;

determine that the level of randomness satisfies a specified criterion; and

detect an anomaly based on the feature score, wherein the anomaly is detected in response to determining that the level of randomness satisfies the specified criterion.

29. A non-transient computer readable medium containing instructions, execution of which by a computer system cause the computer system to:

receive event data associated with a communication between an internal entity within a computer network and an external entity outside the computer network, the event data including an identifier associated with a particular entity, wherein the particular entity is the internal entity or the external entity;

analyze a plurality of characters in the identifier by processing the event data;

assign a feature score based on the analysis, wherein the feature score is indicative of a level of confidence that the identifier is machine generated and a level of randomness in a sequence of characters in the identifier;

determine that the level of randomness satisfies a specified criterion; and

detect an anomaly based on the feature score, wherein the anomaly is detected in response to determining that the level of randomness satisfies the specified criterion.