Rarity analysis in network security anomaly/threat detection

• US 10,038,707 B2
• Filed: 10/30/2015
• Issued: 07/31/2018
• Est. Priority Date: 08/31/2015
• Status: Active Grant

• Current Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Original Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Inventors: Muddu, Sudhakar, Tryfonas, Christos, Li, Yijiang
• Primary Examiner(s): DADA, BEEMNET W
• Application Number:
  US14/929,204
• Publication Number:
  US 20170063890A1
• Time in Patent Office: 1,005 Days
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/444 Spatial browsing, e.g. 2D maps, 3D or virtual spaces
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 40/134 Hyperlinking
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06V 10/225 based on a marking or identifier characterising the area
• CPC Class Codes: H04L 2463/121 Timestamp
• CPC Class Codes: H04L 41/0893 Assignment of logical groups to network elements
• CPC Class Codes: H04L 41/145 involving simulating, designing, planning or modelling of a network
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 43/00 : Arrangements for monitoring or testing data switching networks
• CPC Class Codes: H04L 43/045 : for graphical visualisation of monitoring data
• CPC Class Codes: H04L 43/062 : related to network traffic
• CPC Class Codes: H04L 43/08 : Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 63/06 : for supporting key management in a packet data network cryptographic mechanisms or cryptographic arrangements for key management H04L9/08
• CPC Class Codes: H04L 63/1408 : by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/1416 : Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 : Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1433 : Vulnerability analysis
• CPC Class Codes: H04L 63/1441 : Countermeasures against malicious traffic countermeasures against attacks on cryptographic mechanisms H04L9/002
• CPC Class Codes: H04L 63/20 : for managing network security; network security policies in general filtering policies H04L63/0227

Abstract

A security platform employs a variety techniques and mechanisms to detect security related anomalies and threats in a computer network environment. The security platform is “big data” driven and employs machine learning to perform security analytics. The security platform performs user/entity behavioral analytics (UEBA) to detect the security related anomalies and threats, regardless of whether such anomalies/threats were previously known. The security platform can include both real-time and batch paths/modes for detecting anomalies and threats. By visually presenting analytical results scored with risk ratings and supporting evidence, the security platform enables network security administrators to respond to a detected anomaly or threat, and to take action promptly.

Claims

1. A method comprising:

analyzing, by a computer system, event data representative of data traffic associated with a computer network to identify a feature of the data traffic, the data traffic including a plurality of occurrences of the feature, each occurrence of the feature having one of a plurality of values of the feature;

identifying, by the computer system, a set of the values whose probability of occurrence does not exceed a probability of occurrence of a particular value of the plurality of values, the set of the values being those values of the feature that have occurred not more than the number of times of the particular value;

determining, by the computer system, a rarity score for the particular value as a function of the probability of occurrence of the set of the values;

detecting, by the computer system, that activity of an entity on the computer network is anomalous in a security context, by determining that an occurrence of the particular value corresponds to an anomaly, based on the rarity score, wherein said determining the rarity score of the particular value and said determining that the occurrence of the particular value corresponds to an anomaly comprise executing a machine learning model at the computer system; and

enabling, by the computer system, a targeted response to a security threat indicated by the activity by causing a display of an indication that the activity is anomalous.

2. The method of claim 1, wherein the feature is an attribute of the data traffic that can assume one of a finite number of values, and wherein the feature includes at least one of (i) an Internet Protocol (IP) address, (ii) a port, (iii) a username of a user, (iv) a device identification (ID), (v) an application name, or (vi) a geo location of a device and/or user.

3. The method of claim 1, wherein determining the rarity score of the particular value is performed as part of execution of a machine learning model executing at the computer system.

4. The method of claim 1, wherein determining the occurrence of the particular value as an anomaly is performed as part of execution of a machine learning model executing at the computer system.

5. The method of claim 1, wherein analyzing the data traffic of the device includes analyzing the data traffic in real-time.

6. The method of claim 1, wherein determining the rarity score of the particular value includes determining the rarity score as a function of a number of occurrences of the particular value, the set of the values and a total number of occurrences of the feature.

7. The method of claim 1, wherein determining the rarity score of the particular value includes:

determining a rarity of the particular value as a function of a number of occurrences of the particular value and the set of the values and a total number of occurrences of the feature, and

determining the rarity score for the particular value based on a confidence interval for the rarity.

8. The method of claim 1, wherein determining the rarity score of the particular value includes:

determining a rarity of the particular value as a sum of a probability of occurrence of the particular value and the set of the values, and

determining the rarity score of the particular value based on a confidence interval for the rarity.

9. The method of claim 1, wherein the rarity score is a tuple including a score threshold and a count threshold, the count threshold indicative of a number of times the particular value can be indicated as an anomaly;

the method further comprising;

using the count threshold to determine whether the particular value is an anomaly.

10. The method of claim 1, wherein determining the occurrence of the particular value as an anomaly includes incrementing a count of a number of times the particular value is indicated as an anomaly.

11. The method of claim 1, wherein determining the occurrence of the particular value as an anomaly includes determining that the rarity score of the particular value is less than a score threshold and that a count of a number of times the particular value is indicated as an anomaly is less than a count threshold.

12. The method of claim 1, wherein determining the occurrence of the particular value as an anomaly includes:

determining that the occurrence of the particular value is not an anomaly if a count of a number of times the particular value is indicated as an anomaly exceeds a count threshold.

13. The method of claim 1, wherein determining the occurrence of the particular value as an anomaly includes determining the particular value as an anomaly based on a score threshold and a count threshold;

the method further comprising;

dynamically adjusting the score threshold and the count threshold based on a number of times the particular value is identified as an anomaly in a predefined period.

14. The method of claim 1, wherein analyzing the data traffic includes:

obtaining information regarding the data traffic from a log, the log representing a plurality of events of the data traffic, each of the events including at least one feature.

15. The method of claim 1, wherein the feature is one of a plurality of features occurring in an event of the data traffic, and wherein determining the rarity score of the particular value of the feature includes determining the rarity score of the particular value when a first feature of the features occurs at a first value in the event.

16. The method of claim 1, wherein determining the rarity score includes:

determining the rarity score for a feature pair, the feature pair including a first feature and a second feature, wherein the first feature is the feature, the determining including determining the rarity score for the occurrence of the particular value of the first feature when the second feature occurs at a first value.

17. The method of claim 1 further comprising:

determining that an event of which the feature is a part as an anomaly based on a rarity score for at least some features in the event and a number of the features whose rarity scores do not satisfy a score threshold for the event.

18. The method of claim 1 further comprising:

determining the rarity score for each of a set of features in an event of which the feature is a part,

determining a number of the set of features whose rarity scores do not satisfy a score threshold for the event, and

determining the event as an anomaly if the number of set of features satisfy a feature count threshold for the event.

19. The method of claim 1 further comprising:

determining an event of which the feature is a part as an anomaly based on whether a particular feature of a plurality of features of the event is determined as anomalous.

20. The method of claim 1 further comprising:

determining an event of which the feature is a part as an anomaly based on whether a particular feature pair of a plurality of features of the event is determined as anomalous.

21. The method of claim 1, wherein determining the occurrence of the particular value as an anomaly includes:

determining an event of which the feature is a part as an anomaly based on at least one of (i) a number of a plurality of features of the event that are determined as anomalous, (ii) whether a particular feature of the features is determined as anomalous, or (iii) a number of times the event has been identified as an anomaly.

22. The method of claim 1 further comprising:

determining an event of which the feature is a part as an anomaly based on a plurality of thresholds, the thresholds being dynamically adjusted by the computer system based on a number of times the event is identified as an anomaly in a predefined period.

23. The method of claim 1, wherein analyzing the data traffic includes:

tracking for each of the occurrences of the feature a corresponding value of the feature, and

storing a count of occurrences of the corresponding value.

24. The method of claim 1, wherein analyzing the data traffic includes:

determining an occurrence of a feature pair, the feature pair including a first feature and a second feature, wherein the first feature is the feature,

tracking for each of the occurrences of the feature pair a corresponding value of the first feature when the second feature occurs at a first value, and

storing a count of the occurrences of the corresponding value.

25. A non-transitory machine-readable storage medium storing instructions, execution of which in a computer system causes the computer system to perform operations comprising:

analyzing event data representative of data traffic associated with a computer network to identify a feature of the data traffic, the data traffic including a plurality of occurrences of the feature, each occurrence of the feature having one of a plurality of values of the feature;

identifying a set of the values whose probability of occurrence does not exceed a probability of occurrence of a particular value of the plurality of values, the set of the values being those values of the feature that have occurred not more than the number of times of the particular value;

determining a rarity score for the particular value as a function of the probability of occurrence of the set of the values;

detecting that activity of an entity on the computer network is anomalous in a security context, by determining that an occurrence of the particular value corresponds to an anomaly, based on the rarity score, wherein said determining the rarity score of the particular value and said determining that the occurrence of the particular value corresponds to an anomaly comprise executing a machine learning model at the computer system; and

enabling a targeted response to a security threat indicated by the activity by causing a display of an indication that the activity is anomalous.

26. The non-transitory machine-readable storage medium of claim 25, wherein determining the rarity score of the particular value comprises:

determining the rarity score as a function of a number of occurrences of the particular value and the set of the values and a total number of occurrences of the feature.

27. The non-transitory machine-readable storage medium of claim 25, wherein determining the rarity score of the particular value comprises:

determining the rarity score of the particular value based on a confidence interval for a first parameter and a second parameter, the first parameter determined as a function of a number of occurrences of the particular value and the set of the values, the second parameter determined as a function of a total number of occurrences of the feature.

28. A system, comprising:

a processor;

a memory operatively coupled to the processor;

a first module operatively coupled to the processor and configured to analyze event data representative of data traffic associated with a computer network to identify a feature of the data traffic, the data traffic including a plurality of occurrences of the feature, each occurrence of the feature having one of a plurality of values of the feature;

a second module operatively coupled to the processor and configured to identify a set of the values whose probability of occurrence does not exceed a probability of occurrence of a particular value of the plurality of values, the set of the values being those values of the feature that have occurred not more than the number of times of the particular value;

a third module operatively coupled to the processor and configured to determine a rarity score of the particular value as a function of the probability of occurrence of the set of values;

a fourth module operatively coupled to the processor and configured to detect that activity of an entity on the computer network is anomalous in a security context, by determining that an occurrence of the particular value corresponds to an anomaly, based on the rarity score, wherein determination of the rarity score of the particular value and determination that the occurrence of the particular value corresponds to an anomaly comprise executing a machine learning model; and

a fifth module operatively coupled to the processor and configured to enable a targeted response to a security threat indicated by the activity by causing a display of an indication that the activity is anomalous.

29. The system of claim 28, wherein the fourth module is further configured to:

determine an event of which the feature is a part as an anomaly based on a rarity score of at least some features in the event and a number of the features whose rarity scores do not satisfy a score threshold for the event.