Automatic log record segmentation

• US 9,626,414 B2
• Filed: 04/14/2014
• Issued: 04/18/2017
• Est. Priority Date: 04/14/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Kanjirathinkal, Rose Catherine, Raghavan, Sriram
• Primary Examiner(s): DANG, THANH HA T
• Application Number:
  US14/252,107
• Publication Number:
  US 20150293920A1
• Time in Patent Office: 1,100 Days
• Field of Search: 707748
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/278 Data partitioning, e.g. horizontal or vertical partitioning
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: H04L 63/14 for detecting or protecting against malicious traffic

Abstract

Methods and arrangements for segmenting log records. A log file is received. Candidates for a sequential pattern within the log file are automatically discerned, and, for each candidate, a likelihood is estimated that it represents a boundary within the log file. Other variants and embodiments are broadly contemplated herein.

Claims

1. A method of segmenting log files, said method comprising:

utilizing at least one processor to execute computer code configured to perform the steps of;

receiving a log file;

generating a plurality of tokens from the log file, wherein a token comprises a term contained within the log file;

generalizing the plurality of tokens, wherein the generalizing comprises replacing a token with a generic annotation corresponding to the token;

identifying a plurality of patterns, wherein each of the plurality of patterns contain a predetermined sequence of generalized tokens;

discerning, using a sequential pattern miner, at least one sequential pattern candidate, wherein each of the at least one sequential pattern candidates comprises a candidate for a boundary pattern;

determining at least one match to the sequential pattern candidate within the log file; and

identifying, based upon the at least one match, a boundary candidate within the log file.

2. The method according to claim 1, comprising ranking the at least one sequential pattern candidate based on an estimated likelihood of a match.

3. The method according to claim 2, comprising choosing the most likely of the at least one sequential pattern candidate via user verification of a match.

4. The method according to claim 3, wherein the user verification of a match is performed via a graphical user interface.

5. The method according to claim 4, comprising displaying on the graphical user interface at least one anomalous portion of at least one sequential pattern candidate.

6. The method according to claim 4, comprising determining the at least one anomalous portion associated with at least one sequential pattern candidate.

7. The method according to claim 1, comprising generating a sequence from the tokens, for review in said discerning of candidates for the at least one sequential pattern.

8. The method according to claim 1, wherein:

the log file comprises a plurality of log records; and

wherein the at least one sequential candidate represents a boundary between log records within the log file.

9. The method of claim 1, wherein the generic annotation comprises an annotation that increases a precision of a sequential pattern candidate.

10. An apparatus for segmenting log records, said apparatus comprising:

at least one processor; and

a computer readable storage medium having computer readable program code embodied therewith and executable by the at least one processor, the computer readable program code comprising;

computer readable program code configured to receive a log record;

computer readable code configured to generate a plurality of tokens from the log file, wherein a token comprises a term contained within the log file;

computer readable code configured to generalize the plurality of tokens, wherein the generalizing comprises replacing a token with a generic annotation corresponding to the token;

computer readable code configured to identify a plurality of patterns, wherein each of the plurality of patterns contain a predetermined sequence of generalized tokens;

computer readable program code configured to discern, using a sequential pattern miner, at least one sequential pattern candidate, wherein each of the at least one sequential pattern candidates comprises a candidate for a boundary pattern;

computer readable program code configured to determine at least one match to the sequential pattern candidate within the log file; and

computer readable program code configured to identify, based upon the at least one match, a boundary candidate within the log file.

11. A computer program product for segmenting log records, said computer program product comprising:

a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising;

computer readable program code configured to receive a log record;

computer readable code configured to generate a plurality of tokens from the log file, wherein a token comprises a term contained within the log file;

computer readable code configured to generalize the plurality of tokens, wherein the generalizing comprises replacing a token with a generic annotation corresponding to the token;

computer readable code configured to identify a plurality of patterns, wherein each of the plurality of patterns contain a predetermined sequence of generalized tokens;

computer readable program code configured to discern, using a sequential pattern miner, at least one sequential pattern candidate, wherein each of the at least one sequential pattern candidates comprises a candidate for a boundary pattern;

computer readable program code configured to determine at least one match to the sequential pattern candidate within the log file; and

computer readable program code configured to identify, based upon the at least one match, a boundary candidate within the log file.

12. The computer program product according to claim 11, comprising computer readable program code configured to rank the at least one sequential candidate based on estimated likelihood of a match.

13. The computer program product according to claim 12, comprising computer readable program code configured to facilitate choosing the most likely of the at least one sequential pattern candidate via user verification of a match.

14. The computer program product according to claim 13, wherein the user verification of a match is performed via a graphical user interface.

15. The computer program product according to claim 14, comprising computer readable program code configured to display on the graphical user interface at least one anomalous portion of at least one sequential pattern candidate.

16. The computer program product according to claim 14, computer readable program code configured to determine the at least one anomalous portion associated with at least one sequential candidate.

17. The computer program product according to claim 11, wherein:

the log file comprises a plurality of log records; and

wherein the at least one sequential candidate represents a boundary between log records within the log file.

18. A method comprising:

utilizing at least one processor to execute computer code configured to perform the steps of;

receiving a log file comprising a plurality of records;

tokenizing at least a portion of the log file to produce a plurality of tokens from the log file, wherein a token comprises a term contained within the log file;

generating a sequence from the tokens;

automatically discerning, using a sequential pattern miner, candidates for a sequential pattern within the log file by matching the generated sequence from the tokens within the log file; and

for each candidate, determining a likelihood that the candidate represents a boundary between log records within the log file, wherein the boundary is an ordered occurrence of the sequential pattern.