Universal link to extract and classify log data

• US 10,313,377 B2
• Filed: 10/19/2016
• Issued: 06/04/2019
• Est. Priority Date: 10/19/2016
• Status: Active Grant

• Current Assignee: Anomali, Inc.
• Original Assignee: Anomali, Inc.
• Inventors: Huang, Wei, Zhou, Yizheng, Njemanze, Hugh Seretse, Deng, Zhong
• Primary Examiner(s): Brown, Anthony D
• Application Number:
  US15/298,150
• Publication Number:
  US 20180109550A1
• Time in Patent Office: 958 Days
• Field of Search: 726 22- 25
• CPC Class Codes: G06F 16/2255 Hash tables
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/906 Clustering; Classification
• CPC Class Codes: H04L 2463/121 Timestamp
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection

Abstract

A universal link to extract and classify log data is disclosed. In various embodiments, a set of candidate data values that match a top level pattern that is common to two or more types of data value of interest is identified. The candidate data values are processed through a plurality of successive filtering stages, each stage of which includes determining which, if any, of said candidates match a more specific pattern associated more specifically with a specific data value type. Candidates, if any, which match the more specific pattern are classified as being of a corresponding specific data type and are removed from the set of candidate data values. A structured data record that associates each candidate data value determined to be of a corresponding one of said types of data value of interest with said corresponding one of said types of data value of interest is generated and stored.

Claims

1. A system, comprising:

a memory configured to store arbitrary log data; and

a processor coupled to the memory and configured to;

identify in said arbitrary log data a set of candidate data values that match a top level pattern that is common to two or more types of data value of interest;

process said candidate data values through a plurality of successive filtering stages, each stage of which includes determining which, if any, of said candidates match a more specific pattern associated more specifically with a specific one of said types of data value of interest;

classifying said candidates, if any, that match the more specific pattern as being of said corresponding specific one of said types of data value of interest; and

removing from the set of candidate data values any candidate data values so identified and classified; and

generate and store a structured data record that associates each candidate data value determined to be of a corresponding one of said types of data value of interest with said corresponding one of said types of data value of interest;

wherein the processor is further configured to apply one or more heuristics to more specifically classify and label one or more values determined to match a pattern associated with a specific one of said types of data value of interest; and

wherein said heuristics include heuristics based on one or more of presence in the arbitrary log data of a characteristic string;

placement within the log data of such a string relative to a given candidate data value;

location of a given candidate data value within the arbitrary log data; and

location within the arbitrary log data of a given candidate data value relative to one or more other candidate data values of the same type.

2. The system of claim 1, wherein said arbitrary log data comprises a discrete set of log data.

3. The system of claim 2, wherein said arbitrary log data comprises a line of log data.

4. The system of claim 1, wherein the processor is configured to identify said set of candidate data value at least in part by applying a regular expression to text comprising said arbitrary log data.

5. The system of claim 1, wherein said arbitrary log data may include log data expressed and represented in any format.

6. The system of claim 1, wherein said arbitrary log data may include zero, one, or more occurrences of each of said types of data value of interest.

7. The system of claim 1, wherein said types of data value of interest include one or more of the following:

an IP address type, a domain type, a URL type, and an email address type.

8. The system of claim 1, wherein the processor is further configured to discard candidate values comprising the set of candidate values that do not match any of said more specific patterns.

9. The system of claim 1, wherein the processor is further configured to detect that said arbitrary log data includes more than a prescribed threshold number of occurrences of data values of a given type, and to split the arbitrary log data into two or more sets each including the prescribed number or fewer occurrences of data values of said given type.

10. The system of claim 9, wherein the processor is configured to perform said recited steps to identify, process, and generate and store separately with respect to each set of log data generated by splitting the arbitrary log data into two or more sets.

11. The system of claim 1, further comprising a communication interface configured to receive said arbitrary log data.

12. The system of claim 1, wherein the processor is further configured to check a candidate data value determined to match a more specific pattern associated with a given type against a database of known values of that type.

13. The system of claim 12, wherein the processor is further configured to check said candidate data value determined to match a more specific pattern associated with said given type against said database of known values of that type at least in part using a data structure that has been marked to reflect at least a subset of values in said database.

14. The system of claim 13, wherein said data structure comprises a bloom filter.

15. A method, comprising:

using a processor to identify in an arbitrary log data a set of candidate data values that match a top level pattern that is common to two or more types of data value of interest;

using the processor to process said candidate data values through a plurality of successive filtering stages, each stage of which includes determining which, if any, of said candidates match a more specific pattern associated more specifically with a specific one of said types of data value of interest;

classifying said candidates, if any, that match the more specific pattern as being of said corresponding specific one of said types of data value of interest; and

removing from the set of candidate data values any candidate data values so identified and classified; and

using the processor to generate and store a structured data record that associates each candidate data value determined to be of a corresponding one of said types of data value of interest with said corresponding one of said types of data value of interest;

wherein the processor is further used to apply one or more heuristics to more specifically classify and label one or more values determined to match a pattern associated with a specific one of said types of data value of interest; and

wherein said heuristics include heuristics based on one or more of presence in the arbitrary log data of a characteristic string;

placement within the log data of such a string relative to a given candidate data value;

location of a given candidate data value within the arbitrary log data; and

location within the arbitrary log data of a given candidate data value relative to one or more other candidate data values of the same type.

16. The method of claim 15, wherein the processor is configured to identify said set of candidate data value at least in part by applying a regular expression to text comprising said arbitrary log data.

17. The method of claim 15, wherein said arbitrary log data may include log data expressed and represented in any format.

18. A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for:

identifying in an arbitrary log data a set of candidate data values that match a top level pattern that is common to two or more types of data value of interest;

processing said candidate data values through a plurality of successive filtering stages, each stage of which includes determining which, if any, of said candidates match a more specific pattern associated more specifically with a specific one of said types of data value of interest;

classifying said candidates, if any, that match the more specific pattern as being of said corresponding specific one of said types of data value of interest; and

removing from the set of candidate data values any candidate data values so identified and classified;

generating and storing a structured data record that associates each candidate data value determined to be of a corresponding one of said types of data value of interest with said corresponding one of said types of data value of interest; and

applying one or more heuristics to more specifically classify and label one or more values determined to match a pattern associated with a specific one of said types of data value of interest;

wherein said heuristics include heuristics based on one or more of presence in the arbitrary log data of a characteristic string;

placement within the log data of such a string relative to a given candidate data value;

location of a given candidate data value within the arbitrary log data; and

location within the arbitrary log data of a given candidate data value relative to one or more other candidate data values of the same type.

19. The computer program product of claim 18, further comprising computer instructions to identify said set of candidate data value at least in part by applying a regular expression to text comprising said arbitrary log data.

20. The computer program product of claim 18, wherein said arbitrary log data may include log data expressed and represented in any format.