Using network locations obtained from multiple threat lists to evaluate network data or machine data

• US 10,367,827 B2
• Filed: 12/19/2013
• Issued: 07/30/2019
• Est. Priority Date: 12/19/2013
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Seward, Mark, Coates, John Robert
• Primary Examiner(s): Najjar, Saleh
• Assistant Examiner(s): Pan, Peiliang
• Application Number:
  US14/135,427
• Publication Number:
  US 20150180891A1
• Time in Patent Office: 2,049 Days
• Field of Search: 726 22- 25, 709224, 709225
• CPC Class Codes: G06F 16/212 with details for data modelling support
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection

Abstract

Systems and methods are provided for identifying network addresses and/or IDs of a deduplicated list among network data, machine data, and/or events derived from network data and/or machine data, and for identifying notable events by searching for the presence of network addresses and/or network IDs that are deduplicated across lists received from multiple external sources. One method includes receiving a plurality of lists of network locations, wherein each list is received from over a network, wherein each of the network locations includes a domain name or an IP address, and wherein at least two of the plurality of lists each include a same network location; aggregating the plurality of lists of network locations into a deduplicated list of unique network locations; and searching network data or machine data for a network location included in the deduplicated list of unique network locations.

Claims

1. A computer-implemented method for identifying notable events in a set of events to facilitate identification of computer or network security-related events, the set of events including a plurality of subsets of events, an event in the set of events comprising a portion of raw machine data representing activity involving at least one host in a plurality of hosts distributed across an enterprise'"'"'s network, the method comprising:

accessing a plurality of threat location lists from third-party sources;

receiving, from a user;

(i) criteria for at least one correlation search for notable events that occur on the enterprise'"'"'s network, and

(ii) designation of a set of threat location lists from the accessed plurality of threat location lists;

generating an aggregated threat location list by merging and deduplicating the designated set of threat location lists from the plurality of threat location lists, the aggregated threat list stored in an index;

in response to receiving the criteria for the at least one correlation search, for a subset of events in the set of events;

extracting a network location and values for one or more fields identified in the criteria from at least one event in the subset of events, at analysis time, by using an extraction rule or regular expression that is associated with an identified field and with the subset of events, the extraction rule or regular expression defining how to extract the network location from the portion of raw machine data, and

determining notable events based on the criteria by determining that;

(i) the extracted network location for the at least one event in the subset of events matches a threat location in the aggregated threat location list stored in the index, and

(ii) the extracted values for the one or more identified fields for the at least one event in the subset of events match user-specified values in the criteria;

generating, for each of a plurality of threat locations from the aggregated threat location list, a count of events from the identified notable events that include a network location matching the threat location; and

causing display of a graphical user interface (GUI) including at least one interface element indicating an amount of activity associated with threat locations from the aggregated threat location list, the amount of activity based on the generated count of events, from the determined notable events, for each of the plurality of threat locations from the aggregated threat location list, the amount of activity associated with threat locations used to facilitate operation performance or security associated with at least one component in an IT environment.

2. The computer-implemented method of claim 1, wherein the aggregated threat location list identifies the threat locations by URL.

3. The computer-implemented method of claim 1, wherein the aggregated threat location list identifies the threat locations by IP address.

4. The computer-implemented method of claim 1, wherein the raw machine data includes log data.

5. The computer-implemented method of claim 1, wherein the raw machine data includes network packet data.

6. The computer-implemented method of claim 1, wherein the raw machine data is included in time-stamped searchable events.

7. The computer-implemented method of claim 1, wherein the raw machine data is included in a set of time-stamped searchable events, and wherein the GUI displays information associated with each event in the set that satisfies the criteria.

8. The computer-implemented method of claim 1, further comprising:

causing display of data on the GUI that identifies the plurality of threat location lists accessed from third-party sources; and

receiving, via the GUI, input indicating the designation of the set of threat location lists from the accessed plurality of threat location lists.

9. The computer-implemented method of claim 1, further comprising:

receiving a user-generated list of threat locations; and

including the threat locations in the user-generated list in the aggregated threat location list.

10. The computer-implemented method of claim 1, wherein each event is associated with a time stamp indicating a time of occurrence of activity reflected in raw data in the event.

11. The computer-implemented method of claim 1, wherein causing display of the GUI includes displaying a threat location corresponding to one of the identified notable events.

12. The computer-implemented method of claim 1, wherein causing display of the GUI includes displaying information associated with the event from which at least one of the matching network locations was extracted.

13. The computer-implemented method of claim 1, wherein the one or more identified notable events comprise a suspicious network location access pattern.

14. The computer-implemented method of claim 1, further comprising determining that the one or more identified notable events involve suspicious network activity, and wherein reporting on one or more identified notable events comprises indicating that the suspicious network activity has occurred.

15. A system for identifying notable events in a set of events to facilitate identification of computer or network security-related events, the set of events including a plurality of subsets of events, an event in the set of events comprising a portion of raw machine data representing activity involving at least one host in a plurality of hosts distributed across an enterprise'"'"'s network, the system comprising:

processing resources and memory coupled to the processing resources, the memory storing instructions that, when executed by the processing resources, cause the system to;

access a plurality of threat location lists from third-party sources;

receive, from a user;

(i) criteria for at least one correlation search for notable events that occur on the enterprise'"'"'s network, and

(ii) designation of a set of threat location lists from the accessed plurality of threat location lists;

generate an aggregated threat location list by merging and deduplicating the designated set of threat location lists from the plurality of threat location lists, the aggregated threat list stored in an index;

in response to receiving the criteria for the at least one correlation search, for a subset of events in the set of events;

extracting a network location and values for one or more fields identified in the criteria from at least one event in the subset of events, at analysis time, by using an extraction rule or regular expression that is associated with an identified field and with the subset of events, the extraction rule or regular expression defining how to extract the network location from the portion of raw machine data, and

determining notable events based on the criteria by determining that;

(i) the extracted network location for the at least one event in the subset of events matches a threat location in the aggregated threat location list stored in the index, and

(ii) the extracted values for the one or more identified fields for the at least one event in the subset of events match user-specified values in the criteria; and

generating, for each of a plurality of threat locations from the aggregated threat location list, a count of events from the identified notable events that include a network location matching the threat location; and

causing display of a graphical user interface (GUI) including at least one interface element indicating can amount of activity associated with threat locations from the aggregated threat location list, the amount of activity based on the generated count of events from the determined notable events, for each of the plurality of threat locations from the aggregated threat location list, the amount of activity associated with threat locations used to facilitate operation performance or security associated with at least one component in an IT environment.

16. The system of claim 15, wherein the aggregated threat location list identifies the threat locations by URL.

17. The system of claim 15, wherein the aggregated threat location list identifies the threat locations by IP address.

18. The system of claim 15, wherein the raw machine data includes log data.

19. The system of claim 15, wherein the raw machine data includes network packet data.

20. The system of claim 15, wherein the raw machine data is included in time-stamped searchable events.

21. The system of claim 15, wherein the raw machine data is included in a set of time-stamped searchable events, and wherein the GUI displays information associated with each event in the set that satisfies the criteria.

22. The system of claim 15, wherein the instructions, when executed by the processing resources, further cause the system to:

cause display of data on the GUI that identifies the plurality of threat location lists accessed from third-party source; and

receiving, via the GUI, input indicating the designation of the set of threat location lists from the accessed plurality of threat location lists.

23. The system of claim 15, wherein the instructions, when executed by the processing resources, further cause the system to:

receive a user-generated list of threat locations; and

include the threat locations in the user-generated list in the aggregated threat location list.

24. The system of claim 15, wherein each event

is associated with a time stamp indicating a time of occurrence of activity reflected in raw data in the event.

25. The system of claim 15, wherein causing display of the GUI includes displaying a threat location corresponding to one of the identified notable events.

26. The system of claim 15, wherein causing display of the GUI includes displaying information associated with the event from which at least one of the matching network locations was extracted.

27. The system of claim 15, wherein the one or more identified notable events comprise a suspicious network location access pattern.

28. The system of claim 15, further comprising determining that the one or more identified notable events involve suspicious network activity, and wherein reporting on one or more identified notable events comprises indicating that the suspicious network activity has occurred.

29. A computer readable storage medium, comprising:

instructions that, when executed by processing resources, cause the processing resources to;

access a plurality of threat location lists from third-party sources;

receive, from a user;

(i) criteria from at least one correlation search for notable events that occur on the enterprise'"'"'s network, and

(ii) designation of a set of threat location lists from the accessed plurality of threat location lists;

generate an aggregated threat location list by merging and deduplicating the designated set of threat location lists from the plurality of threat location lists, the aggregated threat list stored in an index;

in response to receiving the criteria for the at least one correlation search, for a subset of events in a set of events, wherein the set of events includes a plurality of subsets of events and an event in the set of events comprises a portion of raw machine data representing activity involving at least one host in a plurality of hosts distributed across an enterprise'"'"'s network;

extract a network location and values for one or more fields identified in the criteria from at least one event in the subset of events, at analysis time, by using an extraction rule or regular expression that is associated with an identified field and with the subset of events, the extraction rule or regular expression defining how to extract the network location from the portion of raw machine data, and

determine notable events based on the criteria by determining that;

(i) the extracted network location for the at least one event in the subset of events matches a threat location in the aggregated threat location list stored in the index, and

(ii) the extracted values for the one or more identified fields for the at least one event in the subset of events match user-specified values in the criteria; and

generating, for each of a plurality of threat locations from the aggregated threat location list, a count of events from the identified notable events that include a network location matching the threat location; and

causing display of a graphical user interface (GUI) including at least one interface element indicating can amount of activity associated with threat locations from the aggregated threat location list, the amount of activity based on the generated count of events, from the determined notable events, for each of the plurality of threat locations from the aggregated threat location list, the amount of activity associated with threat locations used to facilitate operation performance or security associated with at least one component in an IT environment.

30. The computer readable storage medium of claim 29, wherein the aggregated threat location list identifies the threat locations by URL.

31. The computer readable storage medium of claim 29, wherein the aggregated threat location list identifies the threat locations by IP address.

32. The computer readable storage medium of claim 29, wherein the raw machine data includes log data.

33. The computer readable storage medium of claim 29, wherein the raw machine data includes network packet data.

34. The computer readable storage medium of claim 29, wherein the raw machine data is included in time-stamped searchable events.

35. The computer readable storage medium of claim 29, wherein the raw machine data is included in a set of time-stamped searchable events, and wherein the GUI displays information associated with each event in the set that satisfies the criteria.

36. The computer readable storage medium of claim 29, wherein the instructions, when executed by the processing resources, further cause the processing resources to:

cause display of data on the GUI that identifies the plurality of threat location lists accessed from third-party source; and

receive, via the GUI, input indicating the designation of the set of threat location lists from the accessed plurality of threat location lists.

37. The computer readable storage medium of claim 29, wherein the instructions, when executed by the processing resources, further cause the processing resources to:

receive a user-generated list of threat locations; and

include the threat locations in the user-generated list in the aggregated threat location list.

38. The computer readable storage medium of claim 29, wherein each event is associated with a time stamp indicating a time of occurrence of activity reflected in raw data in the event.

39. The computer readable storage medium of claim 29, wherein causing display of the GUI includes displaying a threat location corresponding to one of the identified notable events.

40. The computer readable storage medium of claim 29, wherein causing display of the GUI includes displaying information associated with the event from which at least one of the matching network locations was extracted.

41. The computer readable storage medium of claim 29, wherein the one or more identified notable events comprise a suspicious network location access pattern.

42. The computer readable storage medium of claim 29, further comprising determining that the one or more identified notable events involve suspicious network activity, and wherein reporting on one or more identified notable events comprises indicating that the suspicious network activity has occurred.