Investigative and dynamic detection of potential security-threat indicators from events in big data

• US 9,215,240 B2
• Filed: 07/31/2013
• Issued: 12/15/2015
• Est. Priority Date: 07/25/2013
• Status: Active Grant

• Current Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Original Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Inventors: Merza, Munawar Monzy, Coates, John, Hansen, James, Murphey, Lucas, Hazekamp, David, Kinsley, Michael, Raitz, Alexander
• Primary Examiner(s): Edwards, Linglan
• Assistant Examiner(s): BECHTEL, KEVIN M
• Application Number:
  US13/956,252
• Publication Number:
  US 20130326620A1
• Time in Patent Office: 867 Days
• Field of Search: 726 22- 25
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06F 2221/2151 Time stamp
• CPC Class Codes: H04L 63/1408 by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection

Abstract

A metric value is determined for each event in a set of events that characterizes a computational communication or object. For example, a metric value could include a length of a URL or agent string in the event. A subset criterion is generated, such that metric values within the subset are relatively separated from a population'"'"'s center (e.g., within a distribution tail). Application of the criterion to metric values produces a subset. A representation of the subset is presented in an interactive dashboard. The representation can include unique values in the subset and counts of corresponding event occurrences. Clients can select particular elements in the representation to cause more detail to be presented with respect to individual events corresponding to specific values in the subset. Thus, clients can use their knowledge system operations and observance of value frequencies and underlying events to identify anomalous metric values and potential security threats.

Claims

1. A method performed by one or more computing devices, comprising:

organizing raw machine data collected from one or more remote hardware devices, wherein the collected raw machine data relates to operations or activities in an information technology environment, wherein the one or more computing devices is configured to collect the raw machine data as a plurality of data types from a plurality of remote hardware devices into a set of searchable, time-stamped events;

storing the set of events in a data store stored in computer memory, wherein each event in the set of events is searchable based on its associated time stamp;

executing a computer-implemented search to identify a subset of the set of events satisfying search criteria that includes having a time stamp occurring within a specified time period;

while or after identifying the subset of the set of events, applying a schema to the raw machine data included in each event in the subset of the set of events in order to impose structure on the raw machine data and to extract a set of semantically related values;

calculating a population statistic based on the set of values;

receiving a criterion for determining whether each value in the set of values is sufficiently different than the population statistic;

determining a subset of values in the set of values, wherein each value in the subset of values is identified as an outlier in relation to the population statistic based on application of the criterion to each value in the set of values; and

causing graphical display of information relating to the subset of values.

2. The method of claim 1, further comprising identifying, for each value in the subset of values, an event that includes the machine data from which that value was extracted.

3. The method of claim 1, further comprising:

identifying, for one or more values in the subset of values, one or more events that include the machine data from which that value was extracted; and

displaying information relating to the identified one or more events.

4. The method of claim 1, further comprising:

receiving input reflecting an instruction to display underlying information from which the graphically displayed information was derived;

identifying, for one or more values in the subset of values, one or more events that include the machine data from which that value was extracted; and

displaying, based on the received input, information relating to the identified one or more events.

5. The method of claim 1, wherein the criterion includes a standard deviation from the population statistic.

6. The method of claim 1, wherein the criterion includes a Z-value relative to the population statistic or includes a modified Z-value relative to the population statistic.

7. The method of claim 1, where the population statistic includes a mean, median, mode, or range.

8. The method of claim 1, further comprising displaying the population statistic.

9. The method of claim 1, further comprising:

determining a set of unique values of the metric from the subset of values; and

determining, for each unique value in the set of unique values, a count of a number of times that value appeared in the subset of values.

10. The method of claim 1, further comprising:

determining a set of unique values of the metric from the subset of values; and

determining, for each unique value in the set of unique values, a count of a number of times that value appeared in the subset of values,

wherein graphically displaying information relating to the subset of values comprises plotting unique values in the set of unique values relative to the counts of the number of times the unique values appeared in the subset of values.

11. The method of claim 1, wherein the graphically displayed information relating to the subset of values is not displayed concurrently with information relating to values in the set of values that are not in the subset of values.

12. The method of claim 1, wherein each value in the set of values corresponds to a length of a hypertext transfer protocol (HTTP) user agent string.

13. The method of claim 1, wherein each value in the set of values corresponds to a length of a uniform resource locator (URL).

14. The method of claim 1, wherein each value in the set of values corresponds to a size of network traffic.

15. The method of claim 1, wherein an event in the set of events corresponds to a request for a webpage or a post of a webpage.

16. The method of claim 1, further comprising repeating all the steps in the method of claim 1, wherein at least one of the repetitions includes in the set of events new events generated since the previous repetition.

17. The method of claim 1, wherein the set of events is grouped in a bucket among a plurality of buckets located in a plurality of searchable data stores, each bucket among the plurality of buckets is located in at least one searchable data store among the plurality of searchable data stores.

18. The method of claim 1, wherein the computer-implemented search relates to performance aspects of the information technology environment.

19. The method of claim 1, wherein the computer-implemented search relates to security aspects of the information technology environment.

20. The method of claim 1, wherein the set of semantically related values are included in a same field.

21. A system, comprising:

a subsystem, implemented at least partially in hardware, that organizes raw machine data collected from one or more remote hardware devices, wherein the collected raw machine data relates to operations or activities in an information technology environment, wherein the one or more computing devices is configured to collect the raw machine data as a plurality of data types from a plurality of remote hardware devices into a set of searchable, time-stamped events;

a subsystem, implemented at least partially in hardware, that stores the set of events in a data store stored in computer memory, wherein each event in the set of events is searchable based on its associated time stamp;

a subsystem, implemented at least partially in hardware, that executes a search to identify a subset of the set of events satisfying search criteria that includes having a time stamp occurring within a specified time period;

a subsystem, implemented at least partially in hardware, that, while or after identifying the subset of the set of events, applies a schema to the raw machine data included in each event in the subset of the set of events in order to impose structure on the raw machine data and to extract a set of semantically related values

a subsystem, implemented at least partially in hardware, that calculates a population statistic based on the set of values;

a subsystem, implemented at least partially in hardware, that receives a criterion for determining whether each value in the set of values is sufficiently different than the population statistic;

a subsystem, implemented at least partially in hardware, that determines a subset of values in the set of values, wherein each value in the subset of values is identified as an outlier in relation to the population statistic based on application of the criterion to each value in the set of values; and

a subsystem, implemented at least partially in hardware, that causes graphical display of information relating to the subset of values.

22. The system of claim 21, wherein the criterion includes a standard deviation from the population statistic.

23. The system of claim 21, wherein the criterion includes a Z-value relative to the population statistic or includes a modified Z-value relative to the population statistic.

24. The system of claim 21, where the population statistic includes a mean, median, mode, or range.

25. The system of claim 21, wherein one or more of the processors is further configured for displaying the population statistic.

26. The system of claim 21, further comprising:

a subsystem, implemented at least partially in hardware, that determines a set of unique values of the metric from the subset of values; and

a subsystem, implemented at least partially in hardware, that determines, for each unique value in the set of unique values, a count of a number of times that value appeared in the subset of values.

27. The system of claim 21, wherein the hardware-implemented search relates to performance aspects of the information technology environment.

28. The system of claim 21, wherein the hardware-implemented search relates to security aspects of the information technology environment.

29. The system of claim 21, wherein the set of semantically related values are included in a same field.

30. A computer-program product comprising a non-transitory machine-readable storage medium with instructions stored thereon, the instructions configured to cause one or more microprocessors to perform:

organizing raw machine data collected from one or more remote hardware devices, wherein the collected raw machine data relates to operations or activities in an information technology environment, wherein the one or more computing devices is configured to collect the raw machine data as a plurality of data types from a plurality of remote hardware devices into a set of searchable, time-stamped events;

storing the set of events in a data store stored in computer memory, wherein each event in the set of events is searchable based on its associated time stamp;

executing a computer-implemented search to identify a subset of the set of events satisfying search criteria that includes having a time stamp occurring within a specified time period;

while or after identifying the subset of the set of events, applying a schema to the raw machine data included in each event in the subset of the set of events in order to impose structure on the raw machine data and to extract a set of semantically related values;

calculating a population statistic based on the set of values;

receiving a criterion for determining whether each value in the set of values is sufficiently different than the population statistic;

determining a subset of values in the set of values, wherein each value in the subset of values is identified as an outlier in relation to the population statistic based on application of the criterion to each value in the set of values; and

causing graphical display of information relating to the subset of values.

31. The computer-program product of claim 30, wherein the criterion includes a standard deviation from the population statistic.

32. The computer-program product of claim 30, wherein the criterion includes a Z-value relative to the population statistic or includes a modified Z-value relative to the population statistic.

33. The computer-program product of claim 30, where the population statistic includes a mean, median, mode, or range.

34. The computer-program product of claim 30, further comprising displaying the population statistic.

35. The computer-program product of claim 30, further comprising:

determining a set of unique values of the metric from the subset of values; and

determining, for each unique value in the set of unique values, a count of a number of times that value appeared in the subset of values.

36. The computer-program product of claim 30, the computer-implemented search relates to performance aspects of the information technology environment.

37. The computer-program product of claim 30, wherein the computer-implemented search relates to security aspects of the information technology environment.

38. The computer-program product of claim 30, wherein the set of semantically related values are included in a same field.