Analyzing a group of values extracted from events of machine data relative to a population statistic for those values

• US 9,516,046 B2
• Filed: 10/31/2015
• Issued: 12/06/2016
• Est. Priority Date: 07/25/2013
• Status: Active Grant

• Current Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Original Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Inventors: Merza, Munawar Monzy, Coates, John, Hansen, James M, Murphey, Lucas, Hazekamp, David, Kinsley, Michael, Raitz, Alexander
• Primary Examiner(s): Korsak, Oleg
• Application Number:
  US14/929,321
• Publication Number:
  US 20160057162A1
• Time in Patent Office: 402 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06F 2221/2151 Time stamp
• CPC Class Codes: H04L 63/1408 by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection

Abstract

A metric value is determined for each event in a set of events that characterizes a computational communication or object. For example, a metric value could include a length of a URL or agent string in the event. A subset criterion is generated, such that metric values within the subset are relatively separated from a population'"'"'s center (e.g., within a distribution tail). Application of the criterion to metric values produces a subset. A representation of the subset is presented in an interactive dashboard. The representation can include unique values in the subset and counts of corresponding event occurrences. Clients can select particular elements in the representation to cause more detail to be presented with respect to individual events corresponding to specific values in the subset. Thus, clients can use their knowledge system operations and observance of value frequencies and underlying events to identify anomalous metric values and potential security threats.

Claims

1. A method, comprising:

organizing raw machine data collected from one or more remote hardware devices, into a set of searchable, time-stamped events, wherein the collected raw machine data is produced by one or more components in an information technology environment and relates to operations or activities in that information technology environment, wherein each event in the set of searchable, time-stamped events is searchable based on its associated time stamp;

executing a computer-implemented search to identify a subset of the set of searchable, time-stamped events satisfying search criteria that includes having a time stamp occurring within a specified time period;

while or after identifying the subset of the set of searchable, time-stamped events, applying a schema to the raw machine data included in each event in the subset of the set of searchable, time-stamped events in order to impose structure on the raw machine data and to extract a set of values that relate to a same category;

calculating a population statistic based on the set of values;

receiving a criterion for determining whether a value in the set of values is within a range of the population statistic;

applying the criterion to each value in the set of values to determine;

(i) values in the set of values that are within the range of the population statistic, and (ii) values in the set of values that are outside of the range of the population statistic;

creating a subset of values in the set of values using one of;

(i) the values in the set of values determined to be within the range of the population statistic, or (ii) the values in the set of values determined to be outside of the range of the population statistic; and

causing graphical display of information relating to the subset of values.

2. The method of claim 1, wherein the creating a subset of values in the set of values identifies each value in the subset of values as being included within the population statistic based on application of the criterion to each value in the set of values.

3. The method of claim 1, wherein the creating a subset of values in the set of values identifies each value in the subset of values as an outlier in relation to the population statistic based on application of the criterion to each value in the set of values.

4. The method of claim 1, wherein the schema applied to the raw machine data includes an extraction rule defining how to extract a value in the set of values from an event.

5. The method of claim 1, wherein the schema applied to the raw machine data includes a regular expression defining how to extract a value in the set of values from an event.

6. The method of claim 1, wherein applying the schema to the raw machine data included in each event in the subset of the set of searchable, time-stamped events in order to impose structure on the raw machine data includes applying an extraction rule to the raw machine data included in each event to extract a value in the set of values from that event.

7. The method of claim 1, wherein applying the schema to the raw machine data included in each event in the subset of the set of searchable, time-stamped events in order to impose structure on the raw machine data includes applying a regular expression to the raw machine data included in each event to extract a value in the set of values from that event.

8. The method of claim 1, wherein the category to which the extracted set of values relate is associated with a field.

9. The method of claim 1, wherein the category to which the extracted set of values relate is associated with a field that can be referenced in a search query by an associated field name.

10. The method of claim 1, wherein the category to which the extracted set of values relate is a particular performance metric for measuring performance of a component in an information technology environment.

11. The method of claim 1, further comprising:

identifying, for each value in the subset of values, an event that includes the machine data from which that value was extracted.

12. The method of claim 1, further comprising:

identifying, for one or more values in the subset of values, one or more events that include the machine data from which that value was extracted; and

displaying information relating to the identified one or more events.

13. The method of claim 1, further comprising:

receiving input reflecting an instruction to display underlying information from which the graphically displayed information was derived;

identifying, for one or more values in the subset of values, one or more events that include the machine data from which that value was extracted; and

displaying, based on the received input, information relating to the identified one or more events.

14. The method of claim 1, wherein the criterion includes a standard deviation from the population statistic.

15. The method of claim 1, wherein the criterion is selectable via a pull-down menu and includes a selection for specifying a standard deviation from the population statistic.

16. The method of claim 1, wherein the criterion includes a Z-value relative to the population statistic or includes a modified Z-value relative to the population statistic.

17. The method of claim 1, where the population statistic includes a mean, median, mode, or range.

18. The method of claim 1, further comprising:

displaying the population statistic.

19. The method of claim 1, further comprising:

determining a set of unique values of the metric from the subset of values; and

determining, for each unique value in the set of unique values, a count of a number of times that value appeared in the subset of values.

20. The method of claim 1, further comprising:

determining a set of unique values of the metric from the subset of values; and

determining, for each unique value in the set of unique values, a count of a number of times that value appeared in the subset of values,

wherein graphically displaying information relating to the subset of values comprises plotting unique values in the set of unique values relative to the counts of the number of times the unique values appeared in the subset of values.

21. The method of claim 1, wherein the graphically displayed information relating to the subset of values is not displayed concurrently with information relating to values in the set of values that are not in the subset of values.

22. The method of claim 1, wherein each value in the set of values corresponds to a length of a hypertext transfer protocol (HTTP) user agent string.

23. The method of claim 1, wherein each value in the set of values corresponds to a length of a uniform resource locator (URL).

24. The method of claim 1, wherein each value in the set of values corresponds to a size of network traffic.

25. The method of claim 1, wherein an event in the set of events corresponds to a request for a webpage or a post of a webpage.

26. The method of claim 1, wherein the operations or activities in the information technology environment are related to a security aspect of the information technology environment.

27. The method of claim 1, wherein the set of values that relate to the same category are included in a same field.

28. A non-transitory computer-readable medium storing one or more sequences of instructions, wherein execution of the one or more sequences of instructions by one or more processors causes the one or more processors to perform:

organizing raw machine data collected from one or more remote hardware devices, into a set of searchable, time-stamped events, wherein the collected raw machine data is produced by one or more components in an information technology environment and relates to operations or activities in that information technology environment, wherein each event in the set of searchable, time-stamped events is searchable based on its associated time stamp;

executing a computer-implemented search to identify a subset of the set of searchable, time-stamped events satisfying search criteria that includes having a time stamp occurring within a specified time period;

while or after identifying the subset of the set of searchable, time-stamped events, applying a schema to the raw machine data included in each event in the subset of the set of searchable, time-stamped events in order to impose structure on the raw machine data and to extract a set of values that relate to a same category;

calculating a population statistic based on the set of values;

receiving a criterion for determining whether a value in the set of values is within a range of the population statistic;

applying the criterion to each value in the set of values to determine;

(i) values in the set of values that are within the range of the population statistic, and (ii) values in the set of values that are outside of the range of the population statistic;

creating a subset of values in the set of values using one of;

(i) the values in the set of values determined to be within the range of the population statistic, or (ii) the values in the set of values determined to be outside of the range of the population statistic; and

causing graphical display of information relating to the subset of values.

29. The non-transitory computer-readable medium of claim 28, wherein the creating a subset of values in the set of values identifies each value in the subset of values as being included within the population statistic based on application of the criterion to each value in the set of values.

30. An apparatus, comprising:

a subsystem, implemented at least partially in hardware, that organizes raw machine data collected from one or more remote hardware devices, into a set of searchable, time-stamped events, wherein the collected raw machine data is produced by one or more components in an information technology environment and relates to operations or activities in that information technology environment, wherein each event in the set of searchable, time-stamped events is searchable based on its associated time stamp;

a subsystem, implemented at least partially in hardware, that executes a computer-implemented search to identify a subset of the set of searchable, time-stamped events satisfying search criteria that includes having a time stamp occurring within a specified time period;

a subsystem, implemented at least partially in hardware, that, while or after identifying the subset of the set of searchable, time-stamped events, applies a schema to the raw machine data included in each event in the subset of the set of searchable, time-stamped events in order to impose structure on the raw machine data and to extract a set of values that relate to a same category;

a subsystem, implemented at least partially in hardware, that calculates a population statistic based on the set of values;

a subsystem, implemented at least partially in hardware, that receives a criterion for determining whether a value in the set of values is within a range of the population statistic;

a subsystem, implemented at least partially in hardware, that applies the criterion to each value in the set of values to determine;

(i) values in the set of values that are within the range of the population statistic, and (ii) values in the set of values that are outside of the range of the population statistic;

a subsystem, implemented at least partially in hardware, that creates a subset of values in the set of values using one of;

(i) the values in the set of values determined to be within the range of the population statistic, or (ii) the values in the set of values determined to be outside of the range of the population statistic; and

a subsystem, implemented at least partially in hardware, that causes graphical display of information relating to the subset of values.