Identity resolution in data intake of a distributed data processing system

• US 10,291,635 B2
• Filed: 10/31/2017
• Issued: 05/14/2019
• Est. Priority Date: 08/31/2015
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Muddu, Sudhakar, Tryfonas, Christos, Bulusu, Ravi Prasad
• Primary Examiner(s): Hirl, Joseph P
• Assistant Examiner(s): Murphy, J. Brant
• Application Number:
  US15/800,010
• Publication Number:
  US 20180069888A1
• Time in Patent Office: 560 Days
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/444 Spatial browsing, e.g. 2D maps, 3D or virtual spaces
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 40/134 Hyperlinking
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06V 10/225 based on a marking or identifier characterising the area
• CPC Class Codes: H04L 2463/121 Timestamp cryptographic mechanisms or cryptographic arrangements involving time stamps H04L9/3297
• CPC Class Codes: H04L 41/0893 Assignment of logical groups to network elements
• CPC Class Codes: H04L 41/145 involving simulating, designing, planning or modelling of a network
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 43/00 : Arrangements for monitoring or testing data switching networks
• CPC Class Codes: H04L 43/045 : for graphical visualisation of monitoring data
• CPC Class Codes: H04L 43/062 : related to network traffic
• CPC Class Codes: H04L 43/08 : Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 63/06 : for supporting key management in a packet data network cryptographic mechanisms or cryptographic arrangements for key management H04L9/08
• CPC Class Codes: H04L 63/1408 : by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/1416 : Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 : Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1433 : Vulnerability analysis
• CPC Class Codes: H04L 63/1441 : Countermeasures against malicious traffic countermeasures against attacks on cryptographic mechanisms H04L9/002
• CPC Class Codes: H04L 63/20 : for managing network security; network security policies in general filtering policies H04L63/0227
• CPC Class Codes: H05K 999/99 : dummy group

Abstract

A security platform employs a variety techniques and mechanisms to detect security related anomalies and threats in a computer network environment. The security platform is “big data” driven and employs machine learning to perform security analytics. The security platform performs user/entity behavioral analytics (UEBA) to detect the security related anomalies and threats, regardless of whether such anomalies/threats were previously known. The security platform can include both real-time and batch paths/modes for detecting anomalies and threats. By visually presenting analytical results scored with risk ratings and supporting evidence, the security platform enables network security administrators to respond to a detected anomaly or threat, and to take action promptly.

Claims

1. A computer-implemented method comprising:

identifying a plurality of entities involved in a plurality of events on a computer network, the plurality of entities including a particular user represented by a user identifier and a machine represented by a machine identifier;

determining that the machine identifier is to be associated with the particular user, based on a statistical analysis of events, of the plurality of events, that satisfy a criterion; and

in response to a determination that the machine identifier is to be associated with the particular user, annotating raw, unstructured machine data of a particular event to include an indication that the particular event is associated with the particular user.

2. The method of claim 1, wherein the criterion comprises a probability of association between the particular user and the machine identifier exceeding a confidence threshold.

3. The method of claim 1, wherein the annotating is performed regardless of whether the particular event includes the user identifier.

4. The method of claim 1, wherein the annotating is performed when the particular event includes the machine identifier.

5. The method of claim 1, wherein the annotating is performed when the particular event includes the machine identifier but not the user identifier.

6. The method of claim 1, wherein the annotating is performed when the particular event is received during a valid time period.

7. The method of claim 1, wherein said determining step comprises:

creating a probabilistic graph to generate and track a probability of association between the particular user and the machine identifier,

wherein a result from the probabilistic graph has a time-based dependence on current and past inputs.

8. The method of claim 1, wherein said determining step comprises:

creating a probabilistic graph to record a probability of association between the particular user and the machine identifier,

wherein the probabilistic graph includes a peripheral node, a center node, and an edge, the peripheral node representing the machine identifier, the center node representing the particular user, and the edge representing the probability of association between the machine identifier and the particular user.

9. The method of claim 1, wherein said determining step comprises:

creating a probabilistic graph to record a probability of association between the particular user and the machine identifier,

wherein the probabilistic graph is in the form of a stored data structure, and

wherein the stored data structure is configured to include additional machine identifiers.

10. The method of claim 1, further comprising:

updating a probability of association between the particular user and the machine identifier upon receiving a new event having at least one of;

the machine identifier or the user identifier.

11. The method of claim 1, further comprising:

updating a probability of association between the particular user and the machine identifier upon receiving a new event having at least one of;

the machine identifier or the user identifier;

wherein the new event comprises an authentication event that includes the user identifier.

12. The method of claim 1, further comprising:

updating a probability of association between the particular user and the machine identifier upon receiving a new event having at least one of;

the machine identifier or the user identifier;

wherein the new event comprises an authentication event that includes the user identifier, and

wherein said updating step adjusts a weight assigned to the new event based on a type of authentication event.

13. The method of claim 1, further comprising:

updating a probability of association between the particular user and the machine identifier upon receiving a new event having at least one of;

the machine identifier or the user identifier;

wherein the new event comprises an authentication event that includes the user identifier,

wherein said updating step assigns more weight to a physical login type of authentication event than to any other type of authentication event.

14. The method of claim 1, further comprising:

creating, by a machine learning model, a probabilistic graph to record a probability of association between the particular user and the machine identifier.

15. The method of claim 1, wherein said determining step is performed only on events that have occurred during a lifetime of a particular version of a machine learning model that is used to generate and track a probability of association between the particular user and the machine identifier.

16. The method of claim 1, wherein the plurality of events is received in an order different from a temporal order of the events.

17. The method of claim 1, further comprising:

sending a user association record to a cache server.

18. The method of claim 1, further comprising:

sending a user association record to a cache server that stores structured data,

wherein the user association record is stored in the cache server using a data structure representing a probability of association between the particular user and each of a plurality of machine identifiers.

19. The method of claim 1, wherein the events further include a second machine identifier, the method further comprising:

determining a probability of association between the machine identifier and the second machine identifier.

20. The method of claim 1, wherein the events further include a second machine identifier, the method further comprising:

determining a probability of machine association between the machine identifier and the second machine identifier; and

upon the probability of machine association satisfying a second specified criterion, creating a machine association record indicative that a particular event having the second machine identifier is associated with the machine identifier.

21. The method of claim 1, further comprising:

resolving a user identity of the particular user by querying, using the user identifier as a key, a database having records indicating a plurality of user identifiers registered to the user identity.

22. The method of claim 1, wherein the machine identifier comprises at least one of:

a media access control (MAC) address or an Internet Protocol (IP) address.

23. The method of claim 1, wherein the user identifier comprises at least one of:

a user login identifier (ID), a username, or an electronic mail address.

24. The method of claim 1, wherein identifying the plurality of entities in the events comprises:

parsing the events based on a specified data format that specifies which data represent entities in the events.

25. The method of claim 1, wherein identifying the plurality of entities in the events comprises:

detecting a data format of events.

26. The method of claim 1, wherein identifying the plurality of entities in the events comprises:

detecting a data format of the events by;

comparing the data format of the events to a list of known event data formats; and

determining a highest probability data format based on a result of said comparing.

27. A computer system comprising:

a communication device; and

a processor configured to;

identify a plurality of entities involved in a plurality of events on a computer network, the plurality of entities including a particular user represented by a user identifier and a machine represented by a machine identifier;

determine that the machine identifier is to be associated with the particular user, based on a statistical analysis of events, of the plurality of events, that satisfy a criterion; and

in response to a determination that the machine identifier is to be associated with the particular user, annotate raw, unstructured machine data of a particular event to include an indication that the particular event is associated with the particular user.

28. The computer system of claim 27, wherein the processor is further configured to:

update a probability of association between the particular user and the machine identifier upon receiving a new event having at least one of;

the machine identifier or the user identifier.

29. A non-transitory machine-readable storage medium for use in a processing system, the non-transitory machine-readable storage medium storing instructions, an execution of which in the processing system causes the processing system to perform operations comprising:

identifying a plurality of entities involved in a plurality of events on a computer network, the plurality of entities including a particular user represented by a user identifier and a machine represented by a machine identifier;

determining that the machine identifier is to be associated with the particular user, based on a statistical analysis of events, of the plurality of events, that satisfy a criterion; and

in response to a determination that the machine identifier is to be associated with the particular user, annotating raw, unstructured machine data of a particular event to include an indication that the particular event is associated with the particular user.

30. The medium of claim 29, the operations further comprising:

updating a probability of association between the particular user and the machine identifier upon receiving a new event having at least one of;

the machine identifier or the user identifier.