INVESTIGATIVE AND DYNAMIC DETECTION OF POTENTIAL SECURITY-THREAT INDICATORS FROM EVENTS IN BIG DATA

• US 20130326620A1
• Filed: 07/31/2013
• Published: 12/05/2013
• Est. Priority Date: 07/25/2013
• Status: Active Grant

• Current Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Original Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Inventors: Merza, Munawar Monzy, Coates, John, Hansen, James, Murphey, Lucas, Hazekamp, David, Kinsley, Michael, Raitz, Alexander
• Granted Patent: US 9,215,240 B2
• Time in Patent Office: Days
• US Class Current: 726/22
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06F 2221/2151 Time stamp
• CPC Class Codes: H04L 63/1408 by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection

Abstract

A metric value is determined for each event in a set of events that characterizes a computational communication or object. For example, a metric value could include a length of a URL or agent string in the event. A subset criterion is generated, such that metric values within the subset are relatively separated from a population'"'"'s center (e.g., within a distribution tail). Application of the criterion to metric values produces a subset. A representation of the subset is presented in an interactive dashboard. The representation can include unique values in the subset and counts of corresponding event occurrences. Clients can select particular elements in the representation to cause more detail to be presented with respect to individual events corresponding to specific values in the subset. Thus, clients can use their knowledge system operations and observance of value frequencies and underlying events to identify anomalous metric values and potential security threats.

Claims

1. A computer-implemented method, comprising:

accessing a set of events, wherein each event in the set of events characterizes a computational action or computational communication;

determining a set of metrics, wherein each metric in the set of metrics is determined based on a corresponding event in the set of events;

generating a population characterization based on the extracted metrics;

receiving an input corresponding to an identification of a criterion to use to identify metrics to include in a subset of the set of metrics;

analyzing, for each event in the set of events, the extracted metric with respect to the population characterization;

determining, for each event in the set of events, whether the criterion is satisfied based on the analysis;

generating a subset of metrics that consists of all metrics for which the criterion is satisfied; and

generating a visual object that represents the subset of metrics.

2. The method of claim 1, further comprising:

identifying a set of unique metrics within the subset of metrics; and

identifying, for each unique metric in the set of unique metrics, a count that identifies how many of the metrics corresponded to the unique metric, wherein each metric in the set of unique metrics is included in the set of metrics, and wherein the set of unique metrics does not include any value more than once,

wherein the object represents the set of unique metrics and the counts.

3. The method of claim 1, further comprising:

receiving a second input corresponding to the identification a metric, the metric being one in the subset of metrics;

identifying each event in the set of events for which a metric determined based on the event was equal to the identified metric; and

generating a second visual object that represents additional detail for the identified events, wherein the second visual object does not represent a same type of additional detail for events from the set of events for which the extracted metric is not equal to the metric identified in the input.

4. The method of claim 1, wherein the population characterization includes a mean, median, standard deviation, median of absolute deviation, range, or property of a distribution.

5. The method of claim 1, wherein the metric includes a length of a hypertext transfer protocol (HTTP) user agent string, a length of a uniform resource locator (URL), a presence of a specific HTTP category, or a size of traffic.

6. The method of claim 1, wherein the input corresponds to an identification of a Z-score or modified Z-score.

7. The method of claim 1, wherein the metrics that do not satisfy the criterion are not represented in the object.

8. The method of claim 1, wherein an event in the set of events includes unstructured data.

9. The method of claim 1, wherein an event in the set of events characterizes a request for a webpage or a post of a webpage.

10. The method of claim 1, further comprising:

determining a summary statistic based on the metrics determined based on the set of metrics; and

concurrently presenting the summary statistic and the object.

11. The method of claim 1, further comprising:

receiving a new event;

determining a new metric based on the new event, wherein the new metric is determined in real-time;

updating the population characterization based on the new metric;

analyzing the new metric with respect to the updated population characterization;

determining that the criterion is satisfied based on the analysis;

adding the new metric to the subset of metrics;

updating, in real-time, the object to represent that the subset of metrics includes the new metric.

12. The method of claim 1, further comprising:

receiving a plurality of events, the plurality of events including the set of events and a set of other events;

extracting a timestamp from each event in the plurality of events;

indexing each event in the plurality of events based on the extracted timestamp;

receiving an input corresponding to an identification of a time period; and

identifying events within the plurality of events for which the extracted timestamp is within the time period, wherein the identified events within the plurality of events includes the set of events but not the set of other events.

13. A system, comprising:

one or more data processors; and

a non-transitory computer-readable storage medium containing instructions which when executed on the one or more data processors, cause the one or more processors to perform operations including;

accessing a set of events, wherein each event in the set of events characterizes a computational action or computational communication;

determining a set of metrics, wherein each metric in the set of metrics is determined based on a corresponding event in the set of events;

generating a population characterization based on the extracted metrics;

receiving an input corresponding to an identification of a criterion to use to identify metrics to include in a subset of the set of metrics;

analyzing, for each event in the set of events, the extracted metric with respect to the population characterization;

determining, for each event in the set of events, whether the criterion is satisfied based on the analysis;

generating a subset of metrics that consists of all metrics for which the criterion is satisfied; and

generating a visual object that represents the subset of metrics.

14. The system of claim 13, wherein the operations further comprise:

identifying a set of unique metrics within the subset of metrics; and

identifying, for each unique metric in the set of unique metrics, a count that identifies how many of the metrics corresponded to the unique metric, wherein each metric in the set of unique metrics is included in the set of metrics, and wherein the set of unique metrics does not include any value more than once,

wherein the object represents the set of unique metrics and the counts.

15. The system of claim 13, wherein the operations further comprise:

receiving a second input corresponding to the identification a metric, the metric being one in the subset of metrics;

identifying each event in the set of events for which a metric determined based on the event was equal to the identified metric; and

generating a second visual object that represents additional detail for the identified events, wherein the second visual object does not represent a same type of additional detail for events from the set of events for which the extracted metric is not equal to the metric identified in the input.

16. The system of claim 13, wherein the metric includes a length of a HTTP user agent string, a length of a URL, a presence of a specific HTTP category, or a size of traffic.

17. The system of claim 13, wherein the metrics that do not satisfy the criterion are not represented in the object.

18. The system of claim 13, wherein an event in the set of events includes unstructured data.

19. The system of claim 13, wherein an event in the set of events characterizes a request for a webpage or a post of a webpage.

20. The system of claim 13, wherein the operations further comprise:

determining a summary statistic based on the metrics determined based on the set of metrics; and

concurrently presenting the summary statistic and the object.

21. The system of claim 13, wherein the operations further comprise:

receiving a plurality of events, the plurality of events including the set of events and a set of other events;

extracting a timestamp from each event in the plurality of events;

indexing each event in the plurality of events based on the extracted timestamp;

receiving an input corresponding to an identification of a time period; and

identifying events within the plurality of events for which the extracted timestamp is within the time period, wherein the identified events within the plurality of events includes the set of events but not the set of other events.

22. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause one or more data processors to:

access a set of events, wherein each event in the set of events characterizes a computational action or computational communication;

determine a set of metrics, wherein each metric in the set of metrics is determined based on a corresponding event in the set of events;

generate a population characterization based on the extracted metrics;

receive an input corresponding to an identification of a criterion to use to identify metrics to include in a subset of the set of metrics;

analyze, for each event in the set of events, the extracted metric with respect to the population characterization;

determine, for each event in the set of events, whether the criterion is satisfied based on the analysis;

generate a subset of metrics that consists of all metrics for which the criterion is satisfied; and

generating a visual object that represents the subset of metrics.

23. The computer-program product of claim 22, wherein the instructions further cause the one or more data processors to:

identify a set of unique metrics within the subset of metrics; and

identify, for each unique metric in the set of unique metrics, a count that identifies how many of the metrics corresponded to the unique metric, wherein each metric in the set of unique metrics is included in the set of metrics, and wherein the set of unique metrics does not include any value more than once,

wherein the object represents the set of unique metrics and the counts.

24. The computer-program product of claim 22, wherein the instructions further cause the one or more data processors to:

receive a second input corresponding to the identification a metric, the metric being one in the subset of metrics;

identify each event in the set of events for which a metric determined based on the event was equal to the identified metric; and

generating a second visual object that represents additional detail for the identified events, wherein the second visual object does not represent a same type of additional detail for events from the set of events for which the extracted metric is not equal to the metric identified in the input.

25. The computer-program product of claim 22, wherein the metric includes a length of a hypertext transfer protocol (HTTP) user agent string, a length of a uniform resource locator (URL), a presence of a specific HTTP category, or a size of traffic.

26. The computer-program product of claim 22, wherein the metrics that do not satisfy the criterion are not represented in the object.

27. The computer-program product of claim 22, wherein an event in the set of events includes unstructured data.

28. The computer-program product of claim 22, wherein an event in the set of events characterizes a request for a webpage or a post of a webpage.

29. The computer-program product of claim 22, wherein the instructions further cause the one or more data processors to:

determine a summary statistic based on the metrics determined based on the set of metrics; and

concurrently present the summary statistic and the object.

30. The computer-program product of claim 22, wherein the instructions further cause the one or more data processors to:

receive a plurality of events, the plurality of events including the set of events and a set of other events;

extract a timestamp from each event in the plurality of events;

index each event in the plurality of events based on the extracted timestamp;

receive an input corresponding to an identification of a time period; and

identify events within the plurality of events for which the extracted timestamp is within the time period, wherein the identified events within the plurality of events includes the set of events but not the set of other events.