AUTOMATIC LOG RECORD SEGMENTATION

• US 20150293920A1
• Filed: 04/14/2014
• Published: 10/15/2015
• Est. Priority Date: 04/14/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Kanjirathinkal, Rose Catherine, Raghavan, Sriram
• Granted Patent: US 9,626,414 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/278 Data partitioning, e.g. horizontal or vertical partitioning
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: H04L 63/14 for detecting or protecting against malicious traffic

Abstract

Methods and arrangements for segmenting log records. A log file is received. Candidates for a sequential pattern within the log file are automatically discerned, and, for each candidate, a likelihood is estimated that it represents a boundary within the log file. Other variants and embodiments are broadly contemplated herein.

Claims

1. A method of segmenting log files, said method comprising:

utilizing at least one processor to execute computer code configured to perform the steps of;

receiving a log file;

automatically discerning candidates for a sequential pattern within the log file; and

for each candidate, estimating a likelihood that it represents a boundary within the log file.

2. The method according to claim 1, comprising ranking the candidates based on estimated likelihood.

3. The method according to claim 2, comprising choosing the most likely candidate via user verification.

4. The method according to claim 3, wherein the user verification is performed via a graphical user interface.

5. The method according to claim 4, comprising displaying on the graphical user interface at least one anomalous portion of a candidate.

6. The method according to claim 4, comprising determining at least one reason for an anomaly associated with a candidate.

7. The method according to claim 1, comprising tokenizing at least a portion of the log file to produce tokens.

8. The method according to claim 7, comprising generalizing the tokens.

9. The method according to claim 7, comprising generating a sequence from the tokens, for review in said discerning of candidates for a sequential pattern.

10. The method according to claim 1, wherein:

the log file comprises a plurality of log records; and

said estimating comprises estimating a likelihood that a candidate represents a boundary between log records within the log file.

11. An apparatus for segmenting log records, said apparatus comprising:

at least one processor; and

a computer readable storage medium having computer readable program code embodied therewith and executable by the at least one processor, the computer readable program code comprising;

computer readable program code configured to receive a log record;

computer readable program code configured to automatically discern candidates for a sequential pattern within the log file; and

computer readable program code configured to estimate, for each candidate, a likelihood that it represents a boundary within the log file.

12. A computer program product for segmenting log records, said computer program product comprising:

a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising;

computer readable program code configured to receive a log record;

computer readable program code configured to automatically discern candidates for a sequential pattern within the log file; and

computer readable program code configured to estimate, for each candidate, a likelihood that it represents a boundary within the log file.

13. The computer program product according to claim 12, comprising computer readable program code configured to rank the candidates based on estimated likelihood.

14. The computer program product according to claim 13, comprising computer readable program code configured to facilitate choosing the most likely candidate via user verification.

15. The computer program product according to claim 14, wherein the user verification is performed via a graphical user interface.

16. The computer program product according to claim 15, comprising computer readable program code configured to display on the graphical user interface at least one anomalous portion of a candidate.

17. The computer program product according to claim 15, computer readable program code configured to determine at least one reason for an anomaly associated with a candidate.

18. The computer program product according to claim 12, comprising computer readable program code configured to tokenize at least a portion of the log file to produce tokens.

19. The computer program product according to claim 12, wherein:

the log file comprises a plurality of log records; and

said computer readable program code is configured to estimate, for each candidate, a likelihood that it represents a boundary between log records within the log file.

20. A method comprising:

receiving a log file comprising a plurality of records;

tokenizing at least a portion of the log file to produce tokens;

generating a sequence from the tokens, for review in said discerning of candidates for a sequential pattern;

automatically discerning candidates for a sequential pattern within the log file; and

for each candidate, estimating a likelihood that it represents a boundary between log records within the log file.