Recommending and prioritizing computer log anomalies

• US 10,516,684 B1
• Filed: 04/20/2017
• Issued: 12/24/2019
• Est. Priority Date: 04/21/2016
• Status: Active Grant

• Current Assignee: Akamai Technologies, Inc.
• Original Assignee: Instart Logic Incorporated (Akamai Technologies, Inc.)
• Inventors: Jiang, Heju, Ahammad, Parvez
• Primary Examiner(s): Pyzocha, Michael
• Application Number:
  US15/492,965
• Time in Patent Office: 978 Days
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06F 21/577 Assessing vulnerabilities and evaluating computer system security
• CPC Class Codes: G06F 2221/034 Test or assess a computer or a system
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection

Abstract

Computer log entries are processed to determine a plurality of baseline rank values associated with a ranking dimension. An overall baseline rank indicator is computed using the determined baseline rank values. For each log data component value combination included in a group of log data component value combinations, a comparison rank value associated with the ranking dimension is determined. Each of the comparison rank values is compared with the overall baseline rank indicator. Based at least in part on the comparisons, one or more log data component value combinations included in the group of log data component value combinations are identified as more anomalous than other log data component value combinations included in the group of log data component value combinations.

Claims

1. A method, comprising:

processing computer log entries to determine a plurality of baseline rank values associated with a ranking dimension, wherein a ranking dimension identifies a specific log data component category, and wherein determining the plurality of baseline rank values comprises determining a count number of log entries for each unique value of the ranking dimension in the computer log entries and sorting the count numbers to rank each unique value of the ranking dimension by its count number;

using a processor to compute an overall baseline rank indicator using the determined baseline rank values;

determining for each log data component value combination included in a group of log data component value combinations, a comparison rank value associated with the ranking dimension;

comparing each of the comparison rank values with the overall baseline rank indicator; and

based at least in part on the comparisons, identifying one or more log data component value combinations included in the group of log data component value combinations as more anomalous than other log data component value combinations included in the group of log data component value combinations.

2. The method of claim 1, further comprising using the identification of the one or more log data component value combinations as more anomalous to detect a computer network security threat.

3. The method of claim 1, further comprising determining baseline log data component value combinations including by identifying a select number of top ranked values from each of a plurality of component data categories of the computer log entries and generating permutation combinations of each of the top ranked values across the plurality of component data categories of the computer log entries.

4. The method of claim 1, wherein determining the plurality of baseline rank values includes determining a baseline rank value for each of a plurality of different component data categories of the computer log entries.

5. The method of claim 1, wherein the plurality of baseline rank values includes reciprocal rank values.

6. The method of claim 1, wherein the plurality of baseline rank values is determined using a specified ranking metric based on number counts of total numbers of associated matching log entries.

7. The method of claim 1, wherein the baseline rank values and the comparison rank values are based on corresponding log entry count numbers ranked in descending order.

8. The method of claim 1, wherein each baseline rank value of the plurality of baseline rank values corresponds to a different baseline categorical value combination specified by a user.

9. The method of claim 1, wherein computing the overall baseline rank indicator includes calculating a mean reciprocal rank value of the baseline rank values.

10. The method of claim 1, wherein computing the overall baseline rank indicator includes calculating a weighted mean reciprocal rank value of the baseline rank values.

11. The method of claim 1, wherein the computer log entries processed to determine the baseline rank values are different from a second group of log entries utilized in determining the comparison rank values.

12. The method of claim 1, wherein the group of log data component value combinations includes log data component categorical value combinations that have been identified as being not included in a group of baseline log data component categorical value combinations.

13. The method of claim 1, wherein comparing each of the comparison rank values with the overall baseline rank indicator includes determining difference values between each of the comparison rank values with the overall baseline rank indicator.

14. The method of claim 13, wherein identifying the one or more log data component value combinations included in the group of log data component value combinations as more anomalous than the other log data component value combinations included in the group of log data component value combinations includes sorting the difference values and determining that the identified one or more log data component value combinations are associated with larger difference values than the other log data component value combinations included in the group of log data component value combinations.

15. The method of claim 1, further comprising finding one or more of the log entries that correspond to the one or more log data component value combinations identified as more anomalous.

16. The method of claim 1, wherein the computer log entries include computer network access log entries.

17. The method of claim 1, wherein the computer log entries are obtained from edge servers of a content delivery network.

18. The method of claim 1, further comprising processing the computer log entries to identify each unique value of each of one or more data categories of the computer log entries.

19. A system, comprising:

a storage configured to store a repository of computer log entries; and

a processor configured to;

process the computer log entries to determine a plurality of baseline rank values associated with a ranking dimension, wherein a ranking dimension identifies a specific log data component category, and wherein determining the plurality of baseline rank values comprises determining a count number of log entries for each unique value of the ranking dimension in the computer log entries and sorting the count numbers to rank each unique value of the ranking dimension by its count number;

compute an overall baseline rank indicator using the determined baseline rank values;

determine for each log data component value combination included in a group of log data component value combinations, a comparison rank value associated with the ranking dimension;

compare each of the comparison rank values with the overall baseline rank indicator; and

based at least in part on the comparisons, identify one or more log data component value combinations included in the group of log data component value combinations as more anomalous than other log data component value combinations included in the group of log data component value combinations.

20. A computer program product, the computer program product being embodied in a non-transitory computer-readable storage medium and comprising computer instructions for:

processing computer log entries to determine a plurality of baseline rank values associated with a ranking dimension, wherein a ranking dimension identifies a specific log data component category, and wherein determining the plurality of baseline rank values comprises determining a count number of log entries for each unique value of the ranking dimension in the computer log entries and sorting the count numbers to rank each unique value of the ranking dimension by its count number;

computing an overall baseline rank indicator using the determined baseline rank values;

determining for each log data component value combination included in a group of log data component value combinations, a comparison rank value associated with the ranking dimension;

comparing each of the comparison rank values with the overall baseline rank indicator; and

based at least in part on the comparisons, identifying one or more log data component value combinations included in the group of log data component value combinations as more anomalous than other log data component value combinations included in the group of log data component value combinations.