Automatic and scalable log pattern learning in security log analysis

• US 10,567,409 B2
• Filed: 02/06/2018
• Issued: 02/18/2020
• Est. Priority Date: 03/20/2017
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Zhang, Hui, Xu, Jianwu, Zong, Bo
• Primary Examiner(s): Idowu, Olugbenga O
• Application Number:
  US15/889,733
• Publication Number:
  US 20180270262A1
• Time in Patent Office: 742 Days
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/047 Pattern matching networks; Rete networks
• CPC Class Codes: H04L 63/02 for separating internal from external traffic, e.g. firewalls
• CPC Class Codes: H04L 63/0209 Architectural arrangements, e.g. perimeter networks or demilitarized zones
• CPC Class Codes: H04L 63/0236 Filtering by address, protocol, port number or service, e.g. IP-address or URL
• CPC Class Codes: H04L 63/0245 Filtering by information in the payload
• CPC Class Codes: H04L 63/0272 Virtual private networks
• CPC Class Codes: H04L 63/0281 Proxies
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1458 Denial of Service

Abstract

A method for implementing automatic and scalable log pattern learning in security log analysis is provided. The method includes collecting security logs generated by a computer system. An incremental learning process is implemented to generate a set of log patterns from the collected security logs. The collected security logs are parsed using the set of log patterns.

Claims

1. A computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method for implementing automatic and scalable log pattern learning in security log analysis, comprising:

collecting security logs generated by one or more management services of a computer system;

implementing an incremental learning process to generate a set of log patterns from the collected security logs; and

parsing the collected security logs using the set of log patterns;

wherein implementing the incremental learning process to generate the set of log patterns further comprises;

defining a first set as the training set, a second set as a set of log patterns that have been generated, and a third set as a set of logs of the training set that lack a matching pattern in the second set;

sampling the third set to generate a fourth set having a size corresponding to a parameter controlling a maximum resource requirement for the incremental learning process;

performing automatic log pattern recognition to generate a fifth set; and

performing a log filtering process based on the fifth set.

2. The computer program product as recited in claim 1, wherein performing the automatic log pattern recognition further comprises:

tokenizing logs of the fourth set to generate tokens from the logs of the fourth set;

applying a similarity measurement on the logs to capture similarities among the logs of the fourth set;

implementing a hierarchical clustering algorithm to generate a log cluster hierarchy for the logs of the fourth set;

aligning the logs of the fourth set within each cluster associated with a given level of the log cluster hierarchy;

conducting log motif discovery on the aligned logs to find log motifs; and

performing pattern recognition from the log motifs by recognizing one or more log fields.

3. The computer program product as recited in claim 1, wherein performing the log filtering process further comprises:

determining if the fifth set is empty;

in response to determining that the fifth set is empty;

updating the third set by subtracting the fourth set from the third set;

in response to determining that the updated third set is empty, outputting the second set; and

in response to determining that the updated third set includes at least one log, updating the fourth set by sampling the updated third set, and updating the fifth set by performing automatic pattern recognition using the updated fourth set; and

in response to determining that the fifth set includes at least one log pattern;

updating the second set by adding the fifth set to the second set;

updating the third set with logs identified in the third set that lack a matching pattern in the second set;

in response to determining that the updated third set is empty, outputting the updated second set; and

in response to determining that the updated third set includes at least one log, updating the fourth set by sampling the updated third set, and updating the fifth set by performing automatic pattern recognition using the updated fourth set.

4. The computer program product as recited in claim 3, further comprising using a log parser to identify the logs in the third set that lack a matching pattern in the second set.