Detection of potential security threats based on categorical patterns

• US 10,091,227 B2
• Filed: 11/01/2016
• Issued: 10/02/2018
• Est. Priority Date: 07/25/2013
• Status: Active Grant

• Current Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Original Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Inventors: Merza, Munawar Monzy, Coates, John, Hansen, James M, Murphey, Lucas, Hazekamp, David, Kinsley, Michael, Raitz, Alexander
• Primary Examiner(s): Korsak, Oleg
• Application Number:
  US15/339,955
• Publication Number:
  US 20170048265A1
• Time in Patent Office: 700 Days
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06F 2221/2151 Time stamp
• CPC Class Codes: H04L 63/1408 by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection

Abstract

A metric value is determined for each event in a set of events that characterizes a computational communication or object. For example, a metric value could include a length of a URL or agent string in the event. A subset criterion is generated, such that metric values within the subset are relatively separated from a population'"'"'s center (e.g., within a distribution tail). Application of the criterion to metric values produces a subset. A representation of the subset is presented in an interactive dashboard. The representation can include unique values in the subset and counts of corresponding event occurrences. Clients can select particular elements in the representation to cause more detail to be presented with respect to individual events corresponding to specific values in the subset. Thus, clients can use their knowledge system operations and observance of value frequencies and underlying events to identify anomalous metric values and potential security threats.

Claims

1. A method, comprising:

organizing, by a computer system, raw machine data collected from one or more remote hardware devices, into a set of searchable time-stamped events, wherein the collected raw machine data is produced by one or more components in an information technology environment and reflects activity in the information technology environment, wherein each event in the set of searchable time-stamped events is searchable based on its associated time stamp;

executing, by the computer system, a search to identify a subset of the set of searchable time-stamped events satisfying search criteria that include a time stamp and that identify security-related events derived from raw machine data collected from remote hardware devices, and that fall more than a threshold distance from a defined statistical measure of a security-related data population;

while or after identifying the subset of the set of searchable time-stamped events, applying a schema, by the computer system, to the raw machine data included in each event in the subset of the set of time-stamped searchable events to impose structure on the raw machine data and to extract a set of values that relate to a particular category;

identifying, by the computer system, based on the particular category, one or more patterns among the set of values;

generating a multi-dimensional data object based on the one or more patterns;

determining, by the computer system, that a pattern of the one or more patterns occurs outside of a normal occurrence and indicates that a security threat exists, by analyzing data corresponding to the multi-dimensional data object to determine that the pattern occurs outside of the normal occurrence; and

causing, by the computer system, graphical display of information relating to the one or more patterns that occur outside of the normal occurrence, including display of the multi-dimensional data object.

2. The method of claim 1, wherein the schema applied to the raw machine data includes an extraction rule defining how to extract a value in the set of values from an event.

3. The method of claim 1, wherein the schema applied to the raw machine data includes a regular expression defining how to extract a value in the set of values from an event.

4. The method of claim 1, wherein applying the schema to the raw machine data included in each event in the subset of the set of searchable time-stamped events in order to impose structure on the raw machine data includes applying an extraction rule to the raw machine data included in each event to extract a value in the set of values from that event.

5. The method of claim 1, wherein applying the schema to the raw machine data included in each event in the subset of the set of searchable time-stamped events in order to impose structure on the raw machine data includes applying a regular expression to the raw machine data included in each event to extract a value in the set of values from that event.

6. The method of claim 1, wherein the category to which the extracted set of values relate is associated with a field.

7. The method of claim 1, wherein the category to which the extracted set of values relate is associated with a field that can be referenced in a search query by an associated field name.

8. The method of claim 1, wherein the category to which the extracted set of values relate is a particular performance metric for measuring performance of a component in an information technology environment.

9. The method of claim 1, further comprising:

identifying, for each value in the subset of values, an event that includes the raw machine data from which that value was extracted.

10. The method of claim 1, further comprising:

identifying, for one or more values in the subset of values, one or more events that include the raw machine data from which that value was extracted; and

displaying information relating to the identified one or more events.

11. The method of claim 1, further comprising:

receiving input reflecting an instruction to display underlying information from which the graphically displayed information was derived;

identifying, for one or more values in the subset of values, one or more events that include the machine data from which that value was extracted; and

displaying, based on the received input, information relating to the identified one or more events.

12. The method of claim 1, wherein the set of values that relate to the same category are included in a same field.

13. The method of claim 1, wherein the multi-dimensional data object has at least three dimensions.

23. The method of claim 1, wherein the search criterion that identifies security-related events includes identifying events that correspond to a predetermined z-score.

24. The method of claim 1, wherein the events that fall more than a threshold distance from a defined statistical measure include events with associated data that is over a threshold distance from a center of distribution determined based on the associated data, or that is over a threshold distance from a mean or median determined based on the associated data.

14. One or more non-transitory computer-readable storage media, storing instructions, which when executed by one or more processors cause performance of:

organizing raw machine data collected from one or more remote hardware devices, into a set of searchable time-stamped events, wherein the collected raw machine data is produced by one or more components in an information technology environment and reflects activity in the information technology environment, wherein each event in the set of searchable time-stamped events is searchable based on its associated time stamp;

executing a computer-implemented search to identify a subset of the set of searchable time-stamped events satisfying search criteria that include a time stamp and that identify security-related events derived from raw machine data collected from remote hardware devices, and that fall more than a threshold distance from a defined statistical measure of a security-related data population;

while or after identifying the subset of the set of searchable time-stamped events, applying a schema to the raw machine data included in each event in the subset of the set of searchable time-stamped events to impose structure on the raw machine data and to extract a set of values that relate to a particular category;

identifying, based on the particular category, one or more patterns among the set of values;

generating a multi-dimensional data object based on the one or more patterns;

determining that a pattern of the one or more patterns occurs outside of a normal occurrence and indicates that a security threat exists, by analyzing data corresponding to the multi-dimensional data object to determine that the pattern occurs outside of the normal occurrence; and

causing graphical display of information relating to the one or more patterns that occur outside of the normal occurrence, including display of the multi-dimensional data object.

15. The one or more non-transitory storage media of claim 14, wherein the category to which the extracted set of values relate is associated with a field.

16. The one or more non-transitory storage media of claim 14, wherein the category to which the extracted set of values relate is associated with a field that can be referenced in a search query by an associated field name.

17. The one or more non-transitory storage media of claim 14, wherein the category to which the extracted set of values relate is a particular performance metric for measuring performance of a component in an information technology environment.

18. The one or more non-transitory storage media of claim 14, wherein the multi-dimensional data object has at least three dimensions.

19. An apparatus, comprising:

a raw data processor, implemented at least partially in hardware, configured to organize raw machine data collected from one or more remote hardware devices, into a set of searchable time-stamped events, wherein the collected raw machine data is produced by one or more components in an information technology environment and reflects activity in the information technology environment, wherein each event in the set of searchable time-stamped events is searchable based on its associated time stamp;

a subsystem, implemented at least partially in hardware, configured to identify a subset of the set of searchable time-stamped events satisfying search criteria that include a time stamp and that identify security-related events derived from raw machine data collected from remote hardware devices, and that fall more than a threshold distance from a defined statistical measure of a security-related data population;

a schema device, implemented at least partially in hardware, configured to, while or after identifying the subset of the set of searchable time-stamped events, apply a schema to the raw machine data included in each event in the subset of the set of searchable time-stamped events to impose structure on the raw machine data and to extract a set of values that relate to a particular category;

a categorizer device, implemented at least partially in hardware, configured to identify, based on the particular category, one or more patterns among the set of values;

a generator device, implemented at least partially in hardware, configured to generate a multi-dimensional data object based on the one or more patterns;

an analyzer device, implemented at least partially in hardware, configured to determine that a pattern of the one or more patterns occurs outside of a normal occurrence and indicates that a security threat exists, by analyzing data corresponding to the multi-dimensional data object to determine that the pattern occurs outside of the normal occurrence; and

a display formatter, implemented at least partially in hardware, configured to cause graphical display of information relating to the one or more patterns that occur outside of the normal occurrence including display of the multi-dimensional data object.

20. The apparatus of claim 19, wherein the category to which the extracted set of values relate is associated with a field.

21. The apparatus of claim 19, wherein the category to which the extracted set of values relate is associated with a field that can be referenced in a search query by an associated field name.

22. The apparatus of claim 19, wherein the category to which the extracted set of values relate is a particular performance metric for measuring performance of a component in an information technology environment.

25. The apparatus of claim 19, wherein the multi-dimensional data object has at least three dimensions.