USING NETWORK LOCATIONS OBTAINED FROM MULTIPLE THREAT LISTS TO EVALUATE NETWORK DATA OR MACHINE DATA

• US 20150180891A1
• Filed: 12/19/2013
• Published: 06/25/2015
• Est. Priority Date: 12/19/2013
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: SEWARD, Mark, COATES, John Robert
• Granted Patent: US 10,367,827 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/212 with details for data modelling support
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection

Abstract

Systems and methods are provided for identifying network addresses and/or IDs of a deduplicated list among network data, machine data, and/or events derived from network data and/or machine data, and for identifying notable events by searching for the presence of network addresses and/or network IDs that are deduplicated across lists received from multiple external sources. One method includes receiving a plurality of lists of network locations, wherein each list is received from over a network, wherein each of the network locations includes a domain name or an IP address, and wherein at least two of the plurality of lists each include a same network location; aggregating the plurality of lists of network locations into a deduplicated list of unique network locations; and searching network data or machine data for a network location included in the deduplicated list of unique network locations.

Claims

1-30. -30. (canceled)

31. A computer-implemented method, comprising:

accessing a set of events, wherein each event in the set comprises a portion of raw data representing activity data from at least one host in a plurality of hosts distributed across an enterprise'"'"'s network;

applying an aggregated threat location list to analyze, using a processor, values extracted from the raw data at analysis time, wherein preparing the aggregated threat location list includes merging and deduplicating a plurality of threat location lists received from separate third-party sources; and

searching the activity data from the hosts to identify and report on one or more suspicious activity patterns involving activity tied to one or more threat locations listed in the aggregated threat location list.

32. The computer-implemented method of claim 31, wherein the aggregated threat location list identifies the threat locations by URL.

33. The computer-implemented method of claim 31, wherein the aggregated threat location list identifies the threat locations by IP address.

34. The computer-implemented method of claim 31, wherein the activity data includes log data.

35. The computer-implemented method of claim 31, wherein the activity data includes network packet data.

36. The method of claim 31, wherein the activity data is included in time-stamped searchable events.

37. The method of claim 31, wherein the activity data is included in a set of time-stamped searchable events, and identifying suspicious activity patterns comprises using the aggregated threat location list to determine for each event in the set whether the event includes any threat location in the aggregated threat location list.

38. The method of claim 31, further comprising:

transmitting for display data that identifies a set of threat location lists of threat locations; and

receiving data indicative of a user selection of threat location lists to be used in the aggregated threat location list;

wherein the aggregated threat location list used to identify suspicious activity patterns applies the selected threat location lists.

39. The method of claim 31, further comprising:

receiving a user-generated list of threat locations; and

including the threat locations in the user-generated list in identifying the suspicious activity patterns.

40. The method of claim 31, further comprising:

determining that a set of one or more events derived from the activity includes a threat location in the aggregated threat location list and meets other criteria specified by a query or correlation search; and

based on the determination, generating a notable event or an alert or visualization.

61. The method of claim 31, wherein the values are extracted from the activity data at analysis time using an extraction rule or regular expression that defines where to find a field within the portion of raw data in an event and how to extract the value of the field.

62. The method of claim 31, wherein each event is associated with a time stamp indicating a time of occurrence of activity reflected in raw data in the event.

41. A computer-implemented system, comprising:

processing resources and memory coupled to the processing resources, the memory holding instructions that, when executed by the processing resources, cause the system to;

access a set of events, wherein each event in the set comprises a portion of raw data representing activity data from at least one host in a plurality of hosts distributed across an enterprise'"'"'s network;

apply an aggregated threat location list to analyze, using a processor, values extracted from the raw data at analysis time, wherein preparing the aggregated threat location list includes merging and deduplicating a plurality of threat location lists received from separate third-party sources; and

search the activity data from the hosts to identify and report on one or more suspicious activity patterns involving activity tied to one or more threat locations listed in the aggregated threat location list.

42. The computer-implemented system of claim 41, wherein the aggregated threat location list identifies the threat locations by URL.

43. The computer-implemented system of claim 41, wherein the aggregated threat location list identifies the threat locations by IP address.

44. The computer-implemented system of claim 41, wherein the activity data includes log data.

45. The computer-implemented system of claim 41, wherein the activity data includes network packet data.

46. The system of claim 41, wherein the activity data is included in time-stamped searchable events.

47. The system of claim 41, wherein the activity data is included in a set of time-stamped searchable events, and identifying suspicious activity patterns comprises using the aggregated threat location list to determine for each event in the set whether the event includes any threat location in the aggregated threat location list.

48. The system of claim 41, wherein the instructions, when executed by the processing resources, further cause the system to:

transmit for display data that identifies a set of threat location lists of threat locations; and

receive data indicative of a user selection of threat location lists to be used in the aggregated threat location list;

wherein the aggregated threat location list used to identify suspicious activity patterns applies the selected threat location lists.

49. The system of claim 41, wherein the instructions, when executed by the processing resources, further cause the system to:

receive a user-generated list of threat locations; and

include the threat locations in the user-generated list in identifying the suspicious activity patterns.

50. The system of claim 41, wherein the instructions, when executed by the processing resources, further cause the system to:

determine that a set of one or more events derived from the activity includes a threat location in the aggregated threat location list and meets other criteria specified by a query or correlation search; and

based on the determination, generate a notable event or an alert or visualization.

63. The computer-implemented system of claim 41, wherein the values are extracted from the activity data at analysis time using an extraction rule or regular expression that defines where to find a field within the portion of raw data in an event and how to extract the value of the field.

64. The computer-implemented system of claim 41, wherein each event is associated with a time stamp indicating a time of occurrence of activity reflected in raw data in the event.

51. A computer readable non-transitory storage medium, comprising:

instructions that, when executed by processing resources, cause the processing resources to;

access a set of events, wherein each event in the set comprises a portion of raw data representing activity data from at least one host in a plurality of hosts distributed across an enterprise'"'"'s network;

apply an aggregated threat location list to analyze, using a processor, values extracted from the raw data at analysis time, wherein preparing the aggregated threat location list includes merging and deduplicating a plurality of threat location lists received from separate third-party sources; and

search the activity data from the hosts to identify and report on one or more suspicious activity patterns involving activity tied to one or more threat locations listed in the aggregated threat location list.

52. The computer readable storage medium of claim 51, wherein the aggregated threat location list identifies the threat locations by URL.

53. The computer readable storage medium of claim 51, wherein the aggregated threat location list identifies the threat locations by IP address.

54. The computer readable storage medium of claim 51, wherein the activity data includes log data.

55. The computer readable storage medium of claim 51, wherein the activity data includes network packet data.

56. The computer readable storage medium of claim 51, wherein the activity data is included in time-stamped searchable events.

57. The computer readable storage medium of claim 51, wherein the activity data is included in a set of time-stamped searchable events, and identifying suspicious activity patterns comprises using the aggregated threat location list to determine for each event in the set whether the event includes any threat location in the aggregated threat location list.

58. The computer readable storage medium of claim 51, wherein the instructions, when executed by the processing resources, further cause the processing resources to:

transmit for display data that identifies a set of threat location lists of threat locations; and

receive data indicative of a user selection of threat location lists to be used in the aggregated threat location list;

wherein the aggregated threat location list used to identify suspicious activity patterns applies the selected threat location lists.

59. The computer readable storage medium of claim 51, wherein the instructions, when executed by the processing resources, further cause the processing resources to:

receive a user-generated list of threat locations; and

include the threat locations in the user-generated list in identifying the suspicious activity patterns.

60. The computer readable storage medium of claim 51, wherein the instructions, when executed by the processing resources, further cause the processing resources to:

determine that a set of one or more events derived from the activity includes a threat location in the aggregated threat location list and meets other criteria specified by a query or correlation search; and

based on the determination, generate a notable event or an alert or visualization.

65. The computer readable non-transitory storage medium of claim 51, wherein the values are extracted from the activity data at analysis time using an extraction rule or regular expression that defines where to find a field within the portion of raw data in an event and how to extract the value of the field.

66. The computer readable non-transitory storage medium of claim 51, wherein each event is associated with a time stamp indicating a time of occurrence of activity reflected in raw data in the event.